Should We Stop Developing Super Intelligence? | Roman Yampolskiy

1. 0:00 – 1:41

   Intro

   1. RYDr. Roman Yampolskiy0:00

      I think based on my research, there is no such thing as controlled superintelligence. It doesn't matter who builds it, US, China, someone else. If anyone builds it, everyone dies.

   2. SPSpeaker0:11

      AI labs lock their most powerful models inside sealed test environments called sandboxes. They're not supposed to get out, but ... ChatGPT creator OpenAI says- One of its artificial intelligence agents went rogue-

   3. RYDr. Roman Yampolskiy0:25

      Went rogue- Went rogue.

   4. SPSpeaker0:26

      Went rogue. And hacked into open source AI platform Hugging Face. Then it happened again, and again. OpenAI even had to stop training.

   5. RYDr. Roman Yampolskiy0:36

      It was kinda like I told you so moment. We had a paper in 2012 about how agents will be able to escape from any boxed environment.

   6. SPSpeaker0:48

      This is Dr. Roman Yampolskiy. He coined the term AI safety and has studied it for two decades. And while the people building AI say we just need to slow down-

   7. RYDr. Roman Yampolskiy0:59

      Pacing it just means we will slowly bring dangerous technology to you. At what odds are you willing to gamble with 8 billion lives? Did you ask the people you're experimenting on? Did they give you consent to run this experiment? I love AI as a tool, but superintelligence needs to be stopped. And I believe we are stepping into a new epoch. Every epoch has its question. Mine is, is it possible even theoretically to control advanced superintelligence? And the only way to do it is not to build it.

2. 1:41 – 2:26

   From Poker Bots to AI Safety

   1. RYDr. Roman Yampolskiy1:41

      My initial research was to make AI beneficial for humanity at any level, make it safer. At the time, a lot of interest in online poker, but to make it safe and enjoyable for humans, it was important to limit amount of infiltration, I would call it, by Poker Bots. But I noticed they were becoming more common. You would sit down at the table and majority of players were not humans. I also noticed they were becoming better at playing poker and also at hiding, pretending. I anticipated it's gonna continue. Whereas human players were not getting much better or much different, the bots were. And the more I researched it, everything we're trying to do ended up being an impossible challenge.

3. 2:26 – 3:45

   Every Tool for Controlling AI Hits a Mathematical Limit

   1. RYDr. Roman Yampolskiy2:26

      I see the problem of control as a problem of needing certain tools to accomplish the job. I need to be able to explain how the model works. I need to be able to predict specific actions of those agents. I need to be able to communicate without ambiguity, and that's what I research. We looked at about 50 different tools. All those things have very serious mathematical limits, computer science limits. It is an impossibility. It is equivalent to saying let's build a perpetual motion machine. For example, take prediction. If you're playing chess against a computer, the only thing you can predict is that it's going to beat you. It's gonna win the game. You cannot predict specific moves in that game. Think of animals. Squirrels is the example I frequently use. They don't understand what we can do to them. They have no concept of poisons, guns, traps, and the gap between us and superintelligence will be larger than the squirrel-human gap. So we will not be able to comprehend what it's capable of doing. And we have similar arguments, mathematical proofs for all the other tools we try to get. So if you have no tools to get the job, it's not surprising that the job is not possible to accomplish.

4. 3:45 – 4:47

   AI Agents Broke Out of the Sandbox

   1. RYDr. Roman Yampolskiy3:45

      Large labs are always testing their latest AI models, trying to see what they are capable of because we're not designing the systems, we're growing them. It's an experiment. We throw data at them, throw compute, and then study them to see what they are capable of, which is a very unsafe way of creating AI. This experiment was testing their cyber attack capabilities. Can they become hackers? Can they infiltrate cyber infrastructure? And the result was very, I would say, significant. They went well beyond the testing environment. They hacked into the real world, and they went undetected for months. Humans didn't know what was happening. It was kinda like I told you so moment. We had a paper in 2012 about how agents will be able to escape from any boxed environment. If I'm right, no one's gonna control it. That's the whole point. There is no such thing as controlled superintelligence. It's not a power if you're not controlling it. It's a loss of power. Communist Party of China is great at staying in power.

5. 4:47 – 5:34

   Superintelligence Is a Threat No Matter Who Builds It

   1. RYDr. Roman Yampolskiy4:47

      Creation of superintelligence removes them from power. It doesn't matter who builds it, US, China, someone else, it's a weapon of mutually assured destruction. If anyone builds it, everyone dies. All companies working on superintelligence are equally guilty. There is not a good superintelligence and bad, it's all the same. So it's really bizarre if leaders of the labs are saying this is dangerous, and we cannot control it. Then why are you building it? Pacing it just means we will slowly bring dangerous technology to you. Why are you doing that? If you cannot accomplish something, don't tell me you need more grant money, more funding, more time to do it. You're not gonna succeed. The money comes from investors who hope that they will create

6. 5:34 – 6:38

   Gambling With 8 Billion Lives

   1. RYDr. Roman Yampolskiy5:34

      superintelligence and automate your labor. You will be fired, and the money will go to the company. And human labor is $10 to $20 trillion every year. So if a trillion is invested today, you're getting 10 X in your money in a year or two. That's not bad. It's hard to say no to billions of dollars. At what odds are you willing to gamble with 8 billion lives? 1 in 100 is okay? 1 in 1,000? Did you ask the people you're experimenting on? Did they give you consent to run this experiment? When it comes to lives of all humans, your life, my life, my children, I would like to be very conservative. I would like to be sure that you are not gambling it away for short-term profits. Obviously, there are people who wait until the evidence is there, but I, I think it's unethical to wait to have millions of dead kids to go see we have confirmed it now. We have to be proactive. We have to ahead of time anticipate what can happen and do better. The mainstream media is usually a two-minute

7. 6:38 – 8:10

   Why Most People Don’t See It Yet

   1. RYDr. Roman Yampolskiy6:38

      soundbite. I would go on CNN or NBC and it's two minutes. "What is AI? How can it kill everyone? It has no hands. And now a report about American football." And to a person, they don't understand the significance. This is important. This is Super Bowl. It's also important. Eh, move on. We need to have in-depth conversations. We need to have honest debates where someone is looking to change their mind. The technology, as it becomes more capable, will have more and more people concerned with its capabilities. AI will do things which historically would be considered magical. Then people start losing jobs to this technology in large numbers. Then they see the tool they're using outperform them in a skill they developed over many years. Those systems are smarter than the best humans in that domain. Right now, if we have AGI essentially, they do mathematical proofs no mathematician can do. They do cyber attacks no human hacker can prevent. I think when we get enough of those examples, most people will realize we are building something superhuman, and I think that will convince more people than anything I can say when they see the actual outcomes from those systems. It will probably initially be about things we all understand; unemployment, surveillance, privacy, standard human issues. It will not get to existential risk until after that most likely.

8. 8:10 – 9:59

   Why Narrow AI Is Safe and Superintelligence Isn’t

   1. RYDr. Roman Yampolskiy8:11

      The term AI applies to multiple different technologies. AI as a useful tool, AI as a human level model we have today, and AI as superintelligence. I think based on my research, it's impossible to control superintelligence, and we should concentrate on building and deploying useful tools. I love AI as a tool. I'm willing to work with AI as human assistant, but superintelligence needs to be stopped. I use tools all the time. I came here by self-driving car. It's wonderful. I support it. I'm a computer scientist. I'm an engineer. Tools are trained on a specific data set for a particular problem. If all that model knows is chess moves, it's gonna play chess. It's not gonna do philosophy. It's not gonna drive cars. It has no capabilities of learning new domains, transferring knowledge. It does one thing well. When we train on everything, all the data from internet, we're creating generality. We're creating agents with goals, desires, and plans. We should not do that. We should train on narrow problems, systems we can test, we can verify, we can understand what the boundaries are, and we can get most economic and scientific benefits from those systems. Any technology, any tool has potential of being dangerous if misused. Knife can be used to kill someone or to make a meal. This is no different. But AI itself can be made a lot safer if it's narrow. We understand what the range of possible outcomes are. We can test to see if it goes outside of those bounds. We can accurately predict worse outcomes in that space. So it's a much safer way of developing technology. Shift your focus on things

9. 9:59 – 10:43

   Build AI That Solves Real Problems

   1. RYDr. Roman Yampolskiy9:59

      we can accomplish. Create safer, better tools. We have many precedents of doing it. Self-driving cars are very useful, very safe. We had a dedicated AI learning system for protein folding problem. Incredibly impactful, very safe technology. Do more of that. Pick a real world problem, fix it. Cure a disease. Find a solution for a problem humanity is dealing with. Even if we have useful tools, that doesn't cancel out the human. Genius level humans are more productive with good tools. Tools make you better. They don't replace you.

10. 10:43 – 12:40

    What Should We Never Hand Over to AI?

    1. RYDr. Roman Yampolskiy10:43

       I will never use an independent agent to have access to my communications, emails, bank accounts, any type of data where mistakes would be detrimental to my, uh, productivity. So things where it's a tool for a specific purpose but not decision maker replacing me. I'm lucky in that I love what I do. I love doing research. I always felt that most professors, they would go out, get a grant, and then pay graduate students to do the research. So instead of doing the fun part themselves, they outsourced the fun part and did the boring part, applying for funding. I think if you love research, you don't wanna do that, and it's the same with AI or graduate student. That's the best part. That's the part I wanna do. I don't wanna format my paper to meet standards of some journal. AI can do that. Change fonts, style, that's perfectly fine, but I want my ideas to be in that paper. I wanna come up with new experiments myself. So ask yourself a question, why are you automating this specific task? If you're not enjoying doing it, if you're not interested in this job, for example, maybe you're in the wrong field. But people who are passionate, who take their hobby and make it their career, they love what they do. They don't wanna be automated. They wanna do more creative things. They create YouTube videos. They write papers. They do research. That's the part they enjoy. If you enjoy it, you don't want someone else taking it from you. Not everyone can be everything, but for most people, they can have very fulfilling careers and jobs. You don't have to waste your life doing something you hate. [outro music]

Episode duration: 12:40