Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future

If you have an AI that produces bioweapons that could kill most humans in the world, then it's playing at the level of the superpowers in terms of mutually assured destruction. What are the particular zero-day exploits that the AI might use? The, uh, Conquistadors, with some technological advantage, uh, in terms of weaponry and whatnot, very, very small bands were able to overthrow these large empires. Or if you predicted the global economy is going to be skyrocketing into the stratosphere within 10 years, these AI companies should be worth a large fraction of the global portfolio. And so, uh, this is indeed contrary to the efficient market hypothesis.

This is, like, literally the top in terms of contributing to my world model, in terms of all the episodes I've done. How, how do I find more of these? So we've been talking about alignment. Um, suppose we fail at alignment, and we have AIs that are unaligned and at some point becoming more and more intelligent. What does that look like? How concretely could they disempower and take over, uh, humanity?

This is a scenario where we have many AI systems. Uh, they have... The way we've been training them means that they're not interested. When they have the opportunity to take over and rearrange things, uh, to do what they wish, including having their reward or loss be whatever they desire, they would like to take that opportunity. Uh, and so in many of the existing kind of safety schemes, things like constitutional AI or whatnot, uh, you rely on the hope that one AI has been trained in such a way that it will do as it is directed, uh, to then police others. But if all of the AIs in the system, um, are interested in a takeover, and they see an opportunity to coordinate, all act at the same time, so you don't have one AI interrupting another and taking steps, uh, towards a takeover, yeah, then they can all move in that direction. And the thing that I think maybe is, is worth going into in depth and that I think people often don't cover in great concrete detail, um, and which is a sticking point for some, is, yeah, what are the, the mechanisms by which that can happen? And, um, I know you had Eliezer on, who mentions that, you know, whatever plan we can describe, um, there'll probably be elements where, you know, not being ultra-sophisticated, uh, superintelligent beings, having thought about it for the equivalent of thousands of years, you know, our discussion of it will not be as good, uh, as theirs. But we can explore, from what we know now, uh, what are some of the easy channels? And I think as a good general heuristic, if you're saying, yeah, it's, it's possible, plausible, probable that something will happen, uh, that it shouldn't be that hard to take s- samples from that distribution, to try a Monte Carlo approach and gener-... If a thing is quite likely, it shouldn't be super difficult to generate, um, you know, c- coherent rough outlines of how it could go.