Roman Yampolskiy: Dangers of Superintelligent AI | Lex Fridman Podcast #431

If we create general superintelligences, I don't see a good outcome long term for humanity. So there is X-risk, existential risk, everyone's dead. There is S-risk, suffering risks, where everyone wishes they were dead. We have also idea for I-risk, ikigai risks, where we lost our meaning. The systems can be more creative. They can do all the jobs. It's not obvious what you have to contribute to a world where superintelligence exists. Of course, you can have all the variants you mentioned where we are safe, we are kept alive, but we are not in control. We are not deciding anything. We are like animals in a zoo. There is, again, possibilities we can come up with as very smart humans, and then possibilities something 1,000 times smarter can come up with for reasons we cannot comprehend.

The following is a conversation with Roman Yampolskiy, an AI safety and security researcher and author of a new book titled AI: Unexplainable, Unpredictable, Uncontrollable. He argues that there's almost 100% chance that AGI will eventually destroy human civilization. As an aside, let me say that I will have many often technical conversations on the topic of AI, often with engineers building the state-of-the-art AI systems. I would say those folks put the infamous P-doom or the probability of AGI killing all humans at around 1 to 20%. But it's also important to talk to folks who put that value at 70, 80, 90, and as in the case of Roman, at 99.99 and many more nines percent. I'm personally excited for the future, and believe it will be a good one, in part because of the amazing technological innovation we humans create. But we must absolutely not do so with blinders on, ignoring the possible risks, including existential risks of those technologies. That's what this conversation is about. This is the Lex Fridman podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Roman Yampolskiy. What to you is the probability that superintelligent AI will destroy all human civilization?

What's the timeframe?

Let's say 100 years, in the next 100 years.

So the problem with controlling AGI or superintelligence, in my opinion is like a problem of creating a perpetual safety machine. By analogy with perpetual motion machine, it's impossible. Yeah, we may succeed and do a good job with GPT-5, 6, 7, but they just keep improving, learning, eventually self-modifying, interacting with the environment, interacting with malevolent actors. The difference between cybersecurity, narrow AI safety, and safety for general AI for superintelligence is that we don't get a second chance. With cybersecurity, somebody hacks your account, what's the big deal? You get a new password, new credit card, you move on. Here, if we're talking about existential risks, you only get one chance. So you're really asking me, what are the chances that we'll create the most complex software ever on the first try with zero bugs, and it will continue have zero bugs for 100 years or more?