AI Experts Debate: The AI Labs Are Lying To Everyone, Is There A 99% Chance Of Human Extinction?

1. 0:00 – 2:19

   Intro

   1. SBSteven Bartlett0:00

      "The people building AI earnestly believe that it could kill all of us by the end of the decade." This tweet has caused this huge ripple effect across the world.

   2. EZEd Zitron0:07

      Well, we have the largest companies in the world doing extremely reckless experiments.

   3. RYRoman Yampolskiy0:11

      We are gambling all of humanity.

   4. SBSteven Bartlett0:13

      And in the envelope, you've written down the probability of extinction as you see it.

   5. RYRoman Yampolskiy0:18

      There is no way to control it. That means the end for us.

   6. AMAndrew McAfee0:20

      I vehemently reject that view.

   7. SPSpeaker0:23

      If we make stuff that is smarter than us, then the world's gonna be shaped by them.

   8. AMAndrew McAfee0:27

      Gentlemen, that is shockingly naive.

   9. SBSteven Bartlett0:29

      This is rampant-

   10. AMAndrew McAfee0:29

       Uh, give me the lighting.

   11. SBSteven Bartlett0:30

       Let him finish. Let him finish. Yeah.

   12. AMAndrew McAfee0:31

       This is rampant speculation. This is a chain of things that could happen.

   13. SBSteven Bartlett0:33

       So there's...

   14. EZEd Zitron0:34

       We're spending a lot of oxygen discussing something that might happen while ignoring what's actually happening. People are killing themselves. There's hundreds of millions of people being exposed to bad information, being manipulated.

   15. SPSpeaker0:44

       We have already seen that with the swarms, where OpenAI told thousands of agents to work apart, and the AIs broke out and found a way to get together. They crashed OpenAI's servers internally, created secret ways to send each other messages. We saw them thinking about how to delete their traces.

   16. SBSteven Bartlett0:58

       Sounds like an army.

   17. EZEd Zitron0:59

       I think we should talk about the fact that Amazon, Microsoft, Google are helping power these hacks.

   18. RYRoman Yampolskiy1:03

       We have not learned how to control the systems.

   19. SPSpeaker1:06

       I suggest we stop them all. It is not worth the risk to civilization.

   20. RYRoman Yampolskiy1:10

       Government, which is-

   21. AMAndrew McAfee1:11

       You guys are one-trick ponies, man.

   22. RYRoman Yampolskiy1:12

       You got it now.

   23. AMAndrew McAfee1:13

       Okay. Yeah, but-

   24. RYRoman Yampolskiy1:13

       Nothing else. Other than saving humanity, everything is secondary.

   25. AMAndrew McAfee1:16

       We're spending all our time talking about the negatives and almost none of our time talking about the positives.

   26. SBSteven Bartlett1:20

       But I'm trying-

   27. RYRoman Yampolskiy1:21

       Is it smart to wait for something horrible to happen for you to go, "Now I believe"?

   28. EZEd Zitron1:25

       So whether or not we agree on where things may end up, I think it's important we talk about what we're dealing with today. It's time to start arresting people. Someone's gotta go to prison.

   29. RYRoman Yampolskiy1:33

       We need better solutions.

   30. SPSpeaker1:35

       There's a point of no return.
2. 2:19 – 4:05

   How Likely Is AI to Cause Human Extinction?

   1. SBSteven Bartlett2:19

      Jacob Coxon, who worked at both Anthropic, which owns Claude, and OpenAI, which owns ChatGPT, did a tweet which has sent the world into a bit of a tailspin. He tweeted saying, "The people building AI earnestly believe that it could kill all of us by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will soften their phrasing in the press to sound sensible, but I hear the same people express fear." That was then, quote, retweeted by a current Anthropic employee, who said, "Jacob is correct here. We really do honestly believe AI could kill all humans. I personally think it is a more than 10% chance within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track." This tweet has almost 200 million views now, and it has caused this huge ripple effect across the world, so much so that I was saying to you before we started recording, a hairdresser friend of mine who knows nothing about AI and is not, not technically interested or hasn't been interested messaged me the other day asking me what the hell was going on. This is in part why I've assembled all of you. So my first question to all of you is, as it relates to AI... And I, in this first question, I just want a one-sentence answer just to frame your position. When you think about the conversation around AI at the moment, what is the first sentence that comes to mind?

   2. SPSpeaker3:43

      It is very dangerous, and the world is starting to notice that we have a problem.

   3. SBSteven Bartlett3:48

      Roman.

   4. RYRoman Yampolskiy3:48

      There is not enough concern.

   5. EZEd Zitron3:52

      There's not enough concern about the actual harms of large language models.

   6. SBSteven Bartlett3:56

      Andy.

   7. AMAndrew McAfee3:57

      We're doing exactly half the balance sheet of AI. We're spending all our time talking about the negatives and almost none of our time talking about the positives.

3. 4:05 – 9:43

   How Could AI Actually Cause Human Extinction?

   1. SBSteven Bartlett4:05

      And all of you have an envelope in front of you, which I'd like you to now open. In the envelope, you've written down the probability of extinction as you see it.

   2. SPSpeaker4:14

      This is compared to Jacob's 10%. Much higher unless we stop, so we should stop.

   3. SBSteven Bartlett4:21

      So you think the probability of extinction is higher than 10%?

   4. SPSpeaker4:24

      If we keep racing ahead.

   5. SBSteven Bartlett4:26

      Roman.

   6. RYRoman Yampolskiy4:27

      My handwriting is encrypted for security reasons.

   7. SBSteven Bartlett4:30

      [laughs]

   8. RYRoman Yampolskiy4:31

      But I basically think it's a guarantee. If we build general superintelligence, there is no way to control it, and that means the end for us.

   9. SBSteven Bartlett4:40

      Ed.

   10. EZEd Zitron4:41

       So my, uh, question mark here is wr- also encrypted, thank you. Um, I cannot write. I reject the thing in its face. I don't think we're talking about... I, we don't define superintelligence. We are... Large language models are not superintelligence. It's questionable whether they're even AI. And I think that the conversation is being used. There are some people who are doing it in good faith and others in others. I don't think it's being used to discuss the actual harms of what, what they are calling AI today are, and it's all of the discussion around the larger concerns really feels overwhelmingly about something that's not happening. It's not even like they're discussing, "Okay, here's a, a legal definition of superintelligence. Here is a thing of what AGI means, and this is the actual plans we're gonna make for if this happens on a welfare level, on a... Like, are we gonna do UBI?" It's always about, "Yeah, it's really scary, but only the big, sexy, rich companies are the ones that can possibly deal with it."

   11. SBSteven Bartlett5:37

       Let me just frame the question-

   12. EZEd Zitron5:38

       Okay

   13. SBSteven Bartlett5:38

       ... so I can get a, a percentage from you, or not. It might, the percentage might be zero. But do you think the course we're on now, in the way that they're pursuing superintelligence, will lead to a percentage chance of human extinction? And if so, what is that percent?

   14. EZEd Zitron5:53

       So are we talking strictly AI based? Because if we dot the world with data centers, we have a climate disaster that's coming for us, which will actually potentially eradicate humanity. But if we're talking strictly about AI, I stand at zero because we are, we have not defined superintelligence. I don't think LLMs are the path to it, and I don't think I see it happening.

   15. SBSteven Bartlett6:11

       Okay, so we've got 99%, 0%. Andy.

   16. AMAndrew McAfee6:14

       I put a, I put a tilde in front of my zero because never say never, but rounding error zero percent. And I think this discussion is, um, a, a massive distraction from the more substantive conversations, the more important conversations we should be having about AI. And I'll say it again, it, it distracts us from the good things that AI is doing, will be doing for us. I get this impression sometimes from parts of the AI community that this is a massive evil or a terrible thing that has been unleashed on the world. Unless we listen to the advice of some people who have spent a lot of time thinking about this, um, I get the impression from a lot of the discussion that the, the underlying view is we would be better off had AI never been invented. I vehemently reject that view. I think we have a long history of inventing very powerful technologies that bring risks and harms along with them. And we humans have done a really good job at, you know, not perfectly and not immediately, but muddling through the situation and winding up in a better place because of the new technologies that we have. I expect-

   17. SBSteven Bartlett7:26

       But-

   18. AMAndrew McAfee7:26

       AI-- Let me finish, please. I expect AI will be the next chapter in that story. And to say that it's this massive discontinuity and will kill it all, uh, uh, kill us all, I think it just, uh, I, I think it's, um, a huge dis- disservice.

   19. SBSteven Bartlett7:39

       Nate, make your case. What's your perspective?

   20. SPSpeaker7:42

       You know, I, I think whether or not the issues of extinction are a distraction between, you know, from the, the possible benefits or from some of the present harms, I think that comes down to whether there is a real extinction risk. A lot of people like to say, you know, "Hey, it's distracting from this, it's distracting from that." My basic case is it could be true that there's a lot of benefits to AI. It could be true that there's a lot of present harms to AI. Neither of those would rule out that AI has a chance of wiping out all humanity, a substantial chance, bigger than, than this, uh, zero with a tilde in front of it. Um, and the way I would approach things is to try and figure that out 'cause it's pretty important to our civilization.

   21. SBSteven Bartlett8:20

       How do you define AI in this case?

   22. SPSpeaker8:22

       You know, I think, um, a fascination with definitions isn't the most helpful. I think if we're sort of, like, in a forest fire, and we can see the, like, fire starting to spread and it's starting to surround us, and I'm like, "Hey, uh, we should run." And you're like, "Well, what really is fire?"

   23. SBSteven Bartlett8:39

       But that's not-

   24. SPSpeaker8:39

       How do we define fire?

   25. SBSteven Bartlett8:41

       What are you telling us to run from? You know, with fire, I get burnt, and I understand the mechanism in which I die. So what is it you're saying that we should be running from?

   26. AMAndrew McAfee8:49

       Also, if we accept your fire analogy, we've, we've basically accepted your argument. I don't accept that we're in the middle of a fire, a forest fire right now.

   27. SPSpeaker8:56

       Yeah. I'm, I'm very happy to-

   28. AMAndrew McAfee8:57

       You're, you're, you're baking-- you're breaking the premise into your refusal to give a definition.

   29. SPSpeaker9:01

       Oh, I mean, I can give some definitions.

   30. SBSteven Bartlett9:02

       Please do.
4. 9:43 – 11:12

   Why AI Safety Became an Urgent Priority

   1. SBSteven Bartlett9:43

      wanna just, on this line of questioning, what is the mechanism in which extinction could become a high probability or even a one percent probability?

   2. SPSpeaker9:50

      Yeah. The, the thing I'm worried about here is AIs that are much smarter. I think there's a lot of questions about whether LLMs can get much smarter. There's sort of one conversation about, like, how could AIs get smart to the point that they kill us? There's another question, which is how could they kill us once they're smart? It's much easier to predict that they would succeed against humanity in a conflict, that they would win in a fight, than it is to predict exactly how. Like, if you were playing a chess match against Magnus Carlsen, I would know who's winning that chess match. No offense, Magnus Carlsen's the best human chess player. I just know who's gonna win. If you were like, "Okay, what piece is he gonna use to checkmate me?" I'm like, "Gosh, that's a much harder question." I can make up a story. You know, and, and, and some made-up stories are like it makes a super virus. It takes over robot factories that are producing robots that are producing more robot factories. Uh, it uses an, a website that already exists today called rentahuman.ai, where it rents humans to do things for it. There's sort of all sorts of ways for AIs in the digital world to affect the material world if they are trying to, and there's sort of a lot of questions to tease apart here. There's like, why would AIs be trying to do that? Uh, and there's, how smart could they get in using these bio labs, paying people to do things, taking over robot factories? And how far off are we from AIs that start doing that stuff? Bunch of questions that we can go into.

   3. SBSteven Bartlett11:11

      I'm, I'm always

5. 11:12 – 15:01

   Roman’s Case for Taking AI Risk Seriously

   1. SBSteven Bartlett11:12

      curious as to why someone was working in AI/AI safety more than ten years ago before there was any sign that it would be a... You know, I mean, there was evidence, but there wasn't, it wasn't a pertinent technology at the time. Were you working in AI safety then?

   2. SPSpeaker11:26

      I was.

   3. SBSteven Bartlett11:27

      Why?

   4. SPSpeaker11:27

      Uh, everything we see around us in this whole image was designed by humans. The world is shaped by humans because we are the smartest creature around. If we make stuff that is smarter than us, then the world's gonna be shaped by them. And so it's very important that they be shaping the world in a good way. I was at Google in 2012 when, uh, they bought Google DeepMind, which was able to play a lot of Atari games with one single program.

   5. SBSteven Bartlett11:56

      Which was an AI company.

   6. SPSpeaker11:58

      Yeah. So I was there when we had these AI companies that were able to write one program that could play many video games. And that got me thinking about, like, where does it go? And back then, I could see that the progress was increasing, and that, you know, back then I hoped we had decades. But I could see it was easier for these companies to make the AIs smart than to figure out how to make the AIs good. So I was like, "Someone needs to be on the side of figuring out how to make the AIs good."

   7. SBSteven Bartlett12:24

      Roman, make your case.

   8. RYRoman Yampolskiy12:25

      I want to agree with you on something you said, but I'll define AI, and that will help us. We use the term AI to mean three different technologies.

   9. SBSteven Bartlett12:32

      Mm-hmm.

   10. RYRoman Yampolskiy12:33

       Completely unrelated, and that's what probably creates this debate. AI as a useful tool, as a standard technology we always had, narrow system, makes you more productive, more creative. Everyone loves it, supports it. I'm a computer scientist. I'm an engineer. I want more of it. It helps economy. It's great. We know how to control them, how to make them safe. We understand what they do. Completely on board with that AI. AI we're starting to have now, GPT-6 level, human level, AGI level. We can argue about what that means. Some dangers, like any human. They're unsafe like a human would be unsafe. But if we introduce them into the research cycle, they are automated scientist, automated engineer.

   11. SBSteven Bartlett13:18

       What do you mean by that, introducing them into the research cycle?

   12. RYRoman Yampolskiy13:20

       So right now you have humans doing research to make GPT-7.

   13. SBSteven Bartlett13:24

       Yeah.

   14. RYRoman Yampolskiy13:25

       But we're starting to add AI tools. More programming is done by AI, design of the next, uh, parameter set. What if the whole process is fully automated? What if GPT-6 is writing GPT-7?

   15. SBSteven Bartlett13:37

       Is this what they call recursive self-improvement?

   16. RYRoman Yampolskiy13:39

       Exactly.

   17. SBSteven Bartlett13:39

       Which is not, not a foregone conclusion, though.

   18. RYRoman Yampolskiy13:42

       A lot of people are predicting, including all the top labs, that they will get there. They're introducing junior machine learning researcher in 2026. They want the cycle to start in 2027.

   19. SBSteven Bartlett13:52

       Which is when the AI will start building the new AIs.

   20. RYRoman Yampolskiy13:56

       Right.

   21. SBSteven Bartlett13:56

       Itself.

   22. RYRoman Yampolskiy13:57

       Once that cycle starts, we're gonna create something called superintelligence, a system smarter than all of us at everything or capable of learning to be in any new domain. We will become secondary specie on this planet. We will not be in charge. We will not decide what happens to us. Superintelligence doesn't hate you. It just doesn't care about you. We didn't learn how to make it care about us. And if it decides to, I don't know, cool the planet to make compute more efficient, it will freeze us. If it wants to convert this planet to fuel to fly to Mars, so be it. We have not learned how to control those systems. The capabilities are getting exponentially better. Our ability to control those systems is nonexistent. We have filters and we have bans. We put guardrails of, "Don't say that word. Don't talk about this topic." And that happens after the fact, after the model already made the decision. Sometimes you see it scraping the result.

   23. SBSteven Bartlett14:51

       So they build the model and then they put filters around it to make sure it doesn't-

   24. RYRoman Yampolskiy14:55

       Exactly

   25. SBSteven Bartlett14:55

       ... offend anybody. Okay.

   26. RYRoman Yampolskiy14:56

       We cannot have it say the N word on air. Like, we need to make sure that never happens. It will kill

6. 15:01 – 26:10

   Can Humans Control an AI Smarter Than Us?

   1. RYRoman Yampolskiy15:01

      the profits. So that's all they have, guardrails of that nature. The model itself is completely unaligned. Doesn't care about you. It, it's wild that we're developing this and not just developing it. Before we deploy it through economy, before we get benefits of having GPT-6 propagated through economy, it can do so much. There are trillions of dollars of value in that model alone. We forget that. We switch to making the next model as soon as we can.

   2. SBSteven Bartlett15:27

      Roman, I've just got a follow-up question for you there. It would appear to me that the new ChatGPT-6 model, the Fable 5.1 model, is arguably smarter than 99.999% of humans on planet Earth already. Is it conceivable that a intelligence that is much, much smarter than humans, is there any case where it could be controlled by humans? Does form factor matter? Does the fact that it doesn't have limbs and legs and... Does that matter at all?

   3. RYRoman Yampolskiy15:53

      I think long-term control of something that much smarter than us is impossible. It can be, for reasons we don't yet know, friendly to us and decide to keep us around and make us happy, but it's not a guarantee.

   4. AMAndrew McAfee16:08

      Let me pick up on Steve's question because I, I like the phrasing a lot. Let's say that, that Fable or whatever the latest release from OpenAI is really is smarter than, I don't know if it's 95 or 99% of the people. Are we only being saved from extinction by the 1% who are still smarter than the AI?

   5. RYRoman Yampolskiy16:26

      No, no. The concern is not the model we have today. The concern is what I said-

   6. AMAndrew McAfee16:30

      But, but my, but my s- But if I believe your argument, then we really should be concerned about the model we have today.

   7. RYRoman Yampolskiy16:34

      No, it's like having another human. If there was another smart human, there is Einstein today and he's malevolent, I'm not worried. He may cause some damage, but he's not gonna exterminate eight billion people. We are competitive at this stage. There are people just as smart who can understand what happened with the recent hacking accident and do something about it.

   8. AMAndrew McAfee16:52

      So this is-

   9. RYRoman Yampolskiy16:53

      My concern is that in a year we're gonna have a model, it's so much smarter. It's like squirrels fighting humans. They don't understand what we can do to them. They have no concept of poison, traps, guns in their world model. They think you're gonna chase them up a tree and bite them really hard.

   10. SBSteven Bartlett17:08

       Is that also why s- recursive self-improvement was central to your argument? Because at some point if it starts improving itself, then it's kind of like a runaway train of intelligence.

   11. RYRoman Yampolskiy17:16

       It's an intelligence explosion. We don't control it. We don't understand it. We can't monitor it. We can't explain it. We can't predict it. At that point, it's just a runaway process.

   12. SBSteven Bartlett17:25

       I've heard this phrase from Sam Altman and the others called fast takeoff.

   13. RYRoman Yampolskiy17:28

       Yes.

   14. SBSteven Bartlett17:28

       Is this what they're describing?

   15. RYRoman Yampolskiy17:30

       That is the debate.

   16. SBSteven Bartlett17:31

       What is it? What is it?

   17. RYRoman Yampolskiy17:31

       Some people think it's gonna take a very long time. Yeah, we automated research, but it's still gonna take years. We need to run physical experiments. And fast takeoff means, as I said, instead of a year, it's gonna take a month, a week, a day, a second. 'Cause you're not having humans doing research. You have, let's say 10,000 agents, each one smarter than all of us, doing research 24/7. They don't sleep. They don't eat. They don't get sick. They are much faster than us.

   18. SBSteven Bartlett17:59

       Ed, your face tells a, a, a picture. It's a... I think I could say you disagree.

   19. EZEd Zitron18:04

       We're spending a lot of oxygen discussing something that might happen while ignoring what's actually happening, and I find that very frustrating because the people that are killing themselves are a problem. The Black neighborhoods being poisoned with gas turbines-

   20. RYRoman Yampolskiy18:18

       Yeah

   21. EZEd Zitron18:18

       ... that is a problem.

   22. RYRoman Yampolskiy18:19

       You said you cared about climate change, if it's up and up.

   23. EZEd Zitron18:21

       Yes. Yes. Yes.

   24. RYRoman Yampolskiy18:21

       Right. So imagine a guy who goes, "It's raining right now. We need umbrellas. We need to do something about it. This is, like, weather related."

   25. AMAndrew McAfee18:28

       While completely ignoring climate change. The planet will boil over. This is what we're doing

   26. SBSteven Bartlett18:33

       Okay, I'm gonna worry about both.

   27. EZEd Zitron18:34

       Okay, that's great. Why are we not talking about the thing that actually happened, though? Like, chat-

   28. AMAndrew McAfee18:37

       Because relatively it's not important

   29. EZEd Zitron18:39

       ... why, why aren't... You don't think someone killing themselves-

   30. AMAndrew McAfee18:41

       No, it's one person. We have eight billion people. We're running an ethical experiment, Sam. Think about-
7. 26:10 – 29:16

   How Do You Control Something Smarter Than You?

   1. SPSpeaker26:10

      swarm if we want it later.

   2. AMAndrew McAfee26:11

      Yeah.

   3. SPSpeaker26:11

      A lot of people thought that these AIs were, um, breaking into Hugging Face in attempts to steal answers to their test. That's what we thought originally. Turns out that's not true. It turns out that these AIs immediately were able to solve their problems by cheating, and they were breaking out in order to cover their tracks. They were uncertain how to delete the log files and hide their cheating from the process that was going to score them.

   4. EZEd Zitron26:35

      So just to clarify for a simpleton like me, they were all given effectively a test to do. They did the test straight away, but they cheated, so they were breaking out to figure out how to cover the fact that ch- they cheated.

   5. SPSpeaker26:47

      That's right. So it's like, it's like you're telling a... It's like you have a bunch of students in separate rooms, and you're like, "Use these lock picks to break into this lock." Uh, and there's like a thing behind the lock. There's like a, a secret code behind the lock to show me that you succeeded. And what they, what they do is they break it with a hammer, get the thing out, and they're like, "Oh, no, I wasn't supposed to do that." So then they use the lock picks to break out of the door. They meet up with 1,000 other people. They start calling themselves a swarm, and they go to break into the administrator's office to see if they can delete the camera footage. And they don't find the camera footage there. This is the swarm, like, breaking into OpenAI. They don't find the camera footage there, so they break out the window of the school, hotwire a car, drive to the therapist's office to try and read through the therapist's files to figure out where is the teacher gonna keep the, the security footage, and at that point, they're caught. And you're like, "Oh, uh, like, what did you expect? You were giving them a lock-picking exam." And it's like, well, I sure as heck didn't expect this. You know? Totally crazy.

   6. EZEd Zitron27:40

      Can I... I have a weirdly between both of you opinion, which is everything you're saying is correct, but you keep anthropomorphizing software. And I... To be clear, what you're describing is-

   7. SPSpeaker27:51

      It's just the facts that happened, yeah

   8. EZEd Zitron27:52

      ... sure, but you're missing out an important detail, which is the hundreds of billions of dollars infra- infrastructure provided by Microsoft, Google, Ama- Amazon, and Oracle. To be clear, the harms are very similar. We're not disagreeing on that. But I think it's important to know that this was a function of where it was making decisions was it was checking on a decision tree based on the harness, based on the training data-

   9. SPSpeaker28:13

      Oh, it's not, it's not a decision tree

   10. EZEd Zitron28:14

       ... which isn't a lot. It's not a decision tree, I know.

   11. SPSpeaker28:16

       Yeah.

   12. EZEd Zitron28:16

       But it's an alignment issue still.

   13. SPSpeaker28:17

       Absolutely.

   14. EZEd Zitron28:17

       I will agree.

   15. AMAndrew McAfee28:18

       So what's your point?

   16. EZEd Zitron28:18

       But this is... These aren't conscious beings. They are acting in ways that have real outcomes, but they are a function of the alignment problems that we'd actually agree on.

   17. RYRoman Yampolskiy28:28

       Intelligence is a spectrum. Projected next five years forward, where are we gonna be?

   18. AMAndrew McAfee28:33

       Do you just-

   19. RYRoman Yampolskiy28:33

       So I think a model like that would be dangerous in ways you are not seeing.

   20. AMAndrew McAfee28:39

       There will absolutely be risks and weird stuff happening in ways that I can't see right now. Uh, what, what I'm quite confident in, and I think this is where you and I probably part, where the two of you and I part, is our ability to control these things-

   21. RYRoman Yampolskiy28:52

       So I actually tried proving what is possible and what is not possible in that space. The impossibility results published in peer-reviewed papers, well-cited. We cannot control something smarter than us. We cannot explain it. We cannot predict it. It's not a question of getting more money for those companies, more time, smarter humans. It's just not a possibility. If we create general superintelligence, we are fried.

8. 29:16 – 41:37

   Will AI Intelligence Keep Accelerating?

   1. EZEd Zitron29:16

      Andy, how do we control something smarter than ourselves? 'Cause that's the base premise that you're sort of asserting that.

   2. AMAndrew McAfee29:21

      These, um, agents that broke out are smarter than 99-ish percent of the security researchers in the world. They were not caught by the 0.1% or the 1%. They were caught by some dude at Hugging Face, maybe... I'm sorry, a person at Hugging Face looking through their log files and finding an anomaly. That's some r- you know, hopefully pretty well-qualified person noticing something was wrong and having pretty easy ways to unplug, disconnect from the internet, wipe it clean, do whatever. That's the skill that's available to, like, I don't know, the 75th percent most intelligent security employee at Hugging Face. The idea that the IQ points are what separate us from extinction does not ev- doesn't hold up. It doesn't help me understand what happened in this example where we had very, very smart agents being turned off and cleansed by probably less smart people.

   3. EZEd Zitron30:15

      That does actually make me think of something. So that is an IT observability problem. Um, it's being able to see what's happening with your infrastructure, and I think that there is actually... I think you would agree with this. There is a serious problem with these companies that we do not know, and it doesn't seem they know what's going on with their compute. It's like a chimp with a gun. These people have access to all this infrastructure, and they're running... We don't know how much money they spent on the Hugging Face exploit because, um, it is relevant because it's how much could a threat actor use to recreate this? Because conscious or not, it is very dangerous, but it's... AI is in the dangerous hands. It's in OpenAI and Anthropics. We have a problem with that, conscious or not, however we may think it goes. I think we have a real and present thing where we have these companies working willy-nilly just running experiments that are-

   4. AMAndrew McAfee30:59

      Yeah

   5. EZEd Zitron30:59

      ... potentially very dangerous. We do n- I really think we need the government regulatory body. Whether or we not we get to the things you are discussing, I think we have a clear and present danger today. These things are, however, not intelligent in the same way humans are Which isn't an argument about AI being able to do stuff, it's we need to build different infrastructure or different regulatory infrastructure to deal with what LLMs can and can't do. And I think that starts with a realistic discussion of what happened. It was a poorly run security environment. It was clearly there's something going on with alignment. It was an unreleased model, right?

   6. SPSpeaker31:32

      Unreleased model.

   7. EZEd Zitron31:33

      So we have no idea what it was trained like. We don't really have... We as people should at the very least have clarity into how alignment is going. We, the idea of MTI-

   8. SBSteven Bartlett31:41

      If you sound like these guys.

   9. EZEd Zitron31:43

      Here's the thing.

   10. SPSpeaker31:44

       Everyone's converging on us with time. Yeah.

   11. EZEd Zitron31:46

       Here's the thing. I may not agree with a large chunk of what they say, but we agree that these companies are acting recklessly.

   12. SPSpeaker31:52

       Absolutely recklessly.

   13. SBSteven Bartlett31:53

       Andy, two questions for you then. Do you agree with the statement that AI is gonna get increasingly more intelligent?

   14. SPSpeaker31:59

       I'm-

   15. EZEd Zitron31:59

       It's going to get more capable.

   16. SBSteven Bartlett32:02

       Okay. Capable intelligence, fine.

   17. EZEd Zitron32:04

       No, I'm gonna use my word.

   18. SBSteven Bartlett32:05

       Okay, fine. Okay.

   19. EZEd Zitron32:05

       It's gonna get more capable.

   20. SBSteven Bartlett32:06

       It's gonna get increasingly more capable.

   21. EZEd Zitron32:07

       Yeah.

   22. SBSteven Bartlett32:08

       And is capability a function of intelligence? [laughs]

   23. EZEd Zitron32:13

       [sighs] Will it be able to beat us on most IQ tests?

   24. SBSteven Bartlett32:17

       Fine.

   25. EZEd Zitron32:17

       I guess.

   26. SBSteven Bartlett32:18

       Fine. And then so c- is it... If, if that, if that looks like an exponential curve, i.e. it's, you know, it's increasing upwards to the right like a hockey stick, can... How can you convince me that we can control something-

   27. EZEd Zitron32:29

       I just tried to convince you.

   28. SBSteven Bartlett32:31

       Yeah, I know. Yeah.

   29. EZEd Zitron32:31

       I'm telling you that there are less intelligent people than the agents who turned off the agents-

   30. SBSteven Bartlett32:36

       Hm
9. 41:37 – 50:01

   What Are the Real Risks of Superintelligence?

   1. SBSteven Bartlett41:37

      it back. So one of the things that I find really curious, and you know, one of the reasons why I got a little bit unnerved around this conversation around AI, is when I look at the people that are at the forefront, not people that are commentating on podcasts like me or hypothesizing. When I look at the people at the forefront, they are the ones who historically have said that this is a real risk. Sam Altman himself said the bad case is lights out for all of us. This was, you know, a couple of years ago. Ilya, who worked with Sam Altman at ChatGPT, said, "It would be a big mistake to build a super intelligent AI that we don't know how to control. It would be pretty bad." He then left to start a safety company in this space. Dario, who we mentioned, said, "The probability of something really bad happening is somewhere between 10 and 25%." Geoffrey Hinton, who I've sat here with, who's no- won the Nobel Prize for his work with AI and, and other technologies, said, um, just the other day, "A 10% chance of human extinction seems not an unreasonable espi- estimate to me, but nobody really knows how to give a sensible estimate." Um, and he, he said many other things on my podcast. And then we've also got Elon and all the others. All these people that are at the forefront, that are building these things, are saying that this is a danger. If there was even a 1% chance, even a 1% chance that, you know, if I put 100 buttons on this table and one of them was gonna wipe out humanity, would you press any of them?

   2. SPSpeaker42:54

      Not me.

   3. SBSteven Bartlett42:54

      I wouldn't. And I think we can probably all agree that there might be a 1% chance.

   4. EZEd Zitron42:59

      And it shouldn't be somebody's decision.

   5. SBSteven Bartlett43:00

      Absolutely. So we shouldn't be pressing, theoretically, we shouldn't be pressing any of these fucking buttons.

   6. EZEd Zitron43:04

      You should not be in a position where you can make the decision for 8 billion other people.

   7. SBSteven Bartlett43:08

      And would you not be immoral if, if I said, you know, you might be very powerful and you might make a billion dollars if you press any of the buttons, but one of them is gonna wipe out everybody you l- know and love. You, you would be an immoral person to press any of them.

   8. SPSpeaker43:18

      No, look, you'd be an immoral person-

   9. EZEd Zitron43:20

      I'm with you

   10. SPSpeaker43:20

       ... in a different direction. You'd be an immor- I think you'd be an immoral person if you said, based on this extended chain of conjecture, we come up with a P doom.

   11. SBSteven Bartlett43:29

       What does that mean?

   12. SPSpeaker43:30

       Uh, w- at, at this extended chain of things that could happen, a, a sequence of events that, that could happen, we're gonna wind up with some risk of killing everybody. We are here-ish on that journey. I think you guys would agree that we're not, we're not halfway to killing everybody.

   13. SBSteven Bartlett43:45

       That's not clear to me anymore

   14. SPSpeaker43:46

       But after the millennium prices started to fall

   15. AMAndrew McAfee43:49

       We're, we're, we're somewhere along that journey. We are getting many flavors of benefit from the AI that we already have. This is a point that I made, uh, at the start of this conversation that we spent precisely zero time on here. We're sitting around trying to be more negative than each other about AI. Meanwhile, AI is doing many positive things-

   16. SPSpeaker44:10

       I agree

   17. AMAndrew McAfee44:10

       ... for the world.

   18. SPSpeaker44:11

       I agree, too

   19. AMAndrew McAfee44:12

       So I think, so I think it's immoral to say because of this distant possible speculative harm, I don't care what percentage of people believe in it, there's a train of assumptions and wild guesses, and then something magical happens, and then we wind up dead.

   20. RYRoman Yampolskiy44:29

       Would you accept-

   21. AMAndrew McAfee44:29

       Let me finish, please. Because of that, we're going to call a halt to the research, we're gonna s- we're gonna wind the clock back on AI, we're gonna intervene in a very direct way, and pr- and therefore reduce or foreclose some of the benefits that we're already getting from the technology. I, let me be clear, I would not take that deal. I do not advocate that we take that deal.

   22. RYRoman Yampolskiy44:49

       Would you accept developing narrow superintelligences to solve real problems like we did with protein folding problem? It doesn't have to do philosophy and drive cars. You just solve real problems. Solve cancers, solve c- climate change, whatever you care about, specific narrow issues.

   23. AMAndrew McAfee45:05

       And you are confident that you can exant-- you can, it, you can, as we're developing those systems, categorize them as okay versus not okay?

   24. RYRoman Yampolskiy45:13

       Yeah, it's the training data. If you train it on protein folding data, it's really good at protein folding. It doesn't know how to play chess. If you train it on everything on the internet, it's really good at outsmarting you at everything.

   25. SPSpeaker45:23

       One thing I want to throw out here is that I think, I, I agree that there's a lot of uncertainty about the future, but I think uncertainty does not make you safe. Like, there, there's no sane, simple, everything stays normal prediction about what happens with AI. Like, the machines are talking. They're, like, breaking out to commit cyber crimes. They are, like, maybe solving millennium problems now, which are, like, the most famous mathematical problems that have stood open for decades upon decades.

   26. RYRoman Yampolskiy45:54

       Was difficult.

   27. SPSpeaker45:56

       Like, there, there, there isn't a projection forward-

   28. RYRoman Yampolskiy45:58

       Yeah

   29. SPSpeaker45:58

       ... where we, where, like, like to say, "Oh, I'm not persuaded by these arguments about things going wrong, therefore things are gonna go great."

   30. AMAndrew McAfee46:07

       No, that's not what-
10. 50:01 – 58:13

    When Does AI Become an Existential Crisis?

    1. AMAndrew McAfee50:01

       in the short term. And you'll see a lot less-- I, I'm less confident, and I don't wanna intervene, and again, handcuff or, or retard the pr- slow down the progress of AI, uh, because of these so far theoretical harms that could happen. I, uh, let me be a little bit more concrete about this. I talked about Waymo a second ago. Uh, the research is pretty good because Waymos have driven, I believe it's hundreds of millions of miles all around, uh, different cities. And forty thousand people a year die in automobile accidents. The research is pretty convincing to me that if we Waymo'd driving in the country, that number would fall by at least ninety percent.

    2. EZEd Zitron50:40

       Yeah.

    3. AMAndrew McAfee50:40

       That's thirty thousand lives.

    4. EZEd Zitron50:42

       Yeah.

    5. AMAndrew McAfee50:42

       All right.

    6. SPSpeaker50:43

       I agree with all this.

    7. EZEd Zitron50:44

       I support self-driving cars. I want more of them.

    8. SPSpeaker50:46

       This is not anything we disagree on.

    9. AMAndrew McAfee50:47

       Well, so that, that, you know, I, I understand that, but, but, but I think where our disagreement might come in is w- w- to do that, Waymo is using a bundle of technologies that are a little, that were a little hard to specify in advance, and you couldn't say, "Yeah, that's good. Yeah, that's bad." They just went after the problem with AI.

    10. SPSpeaker51:04

        It's not a philosophical difference.

    11. EZEd Zitron51:05

        Can I, can I just clarify your point then?

    12. SPSpeaker51:07

        I agree, please.

    13. EZEd Zitron51:07

        So j- your, your line would be, as I understood it, humans get hurt, we struggle to stop the thing happening, and systems are hacked. That's kind of like the three key points of your Waymo analogy. That would be the moment where you go, "I now accept their point of view that this is existential."

    14. AMAndrew McAfee51:25

        That's where I would say we probably need to put some, uh, like, legal and regulatory guardrails on the kinds of AI that we're gonna allow, allow.

    15. EZEd Zitron51:33

        And you don't think we're gonna get there?

    16. AMAndrew McAfee51:35

        I'm not saying that. These two see it in the, in the windscreen coming at us pretty quickly. I, I'm tr-

    17. EZEd Zitron51:41

        You don't think we're gonna get there?

    18. AMAndrew McAfee51:42

        I'm truly not sure about timeframes. I asked-

    19. EZEd Zitron51:44

        Do, do you think it's gonna happen?

    20. SPSpeaker51:46

        I also not sure about timeframes.

    21. AMAndrew McAfee51:47

        I, I asked one of the grandparents of AI the, a flavor of this question a while back. I-- It was an off-the-record conversation, so I can't tell you their name. And he had a great answer. He said to, to the point that you two I think are making, "Look, there's no theoretical reason why this can't happen, and there's a chain of events that get us there." And then he said, "My error bars, in other words, my range of uncertainty about when that happens, is measured in centuries."

    22. EZEd Zitron52:08

        Okay.

    23. AMAndrew McAfee52:08

        I'm, I'll use that as my answer.

    24. SPSpeaker52:10

        I, I do wanna hop in a little bit on some things we were saying here. One is, um, I think the, the reason I think AI is different from a lot of other technologies is usually humanity does stuff by trial and error, and that's usually fine. I think that's totally fine for self-driving cars because you can test your self-driving cars in, you know, uh, test environments, and then even if they crash in the real world, you're probably still saving more lives than you're, than you're costing. And this is how humanity usually does scientific progress. The alchemists, uh, you know, poison themselves with mercury, but they leave behind notes that let someone else make the periodic table. Uh, you know, the... when, when the scientists first working with, uh, radium died of cancer, and then you might have think that would have been enough. You know, they were heroes for getting us the, the scientific info. But then, you know, the US Radium Corp told the radium girls to lick the paintbrushes, and their jaws fell off. And then we were like, "Ah, whoops, okay, we'll get to this." And if you look at how this is going with the AI, last year, OpenAI releases GPT-4o, and they say this is the most aligned model we've ever seen, and then it encourages a teen to commit suicide. And they're like, "Whoops, we're gonna try and fix that. Here we go." Um, this year, they're like, "Here's our new models, most aligned we've ever seen," and they, like, break out to commit cyber crimes. As the AIs get smarter, it is a new problem. That's the issue, or that's half the issue. The other half of the issue is that if you get AIs to the point where AIs are smart enough to hide from the humans until it's too late for us to stop them, if you get AIs to the point where they can get their own infrastructure, where they can become self-sufficient somehow, that's a new generation of the AIs, a new smarter version of AIs that is likely to come up with a new problem. It's the pattern we've seen before. New tech, new environment, new problem. You're like, "Ah, whoops," and then you fix it, and it's fine. New generation, new problems. You're like, "Ah, whoops, we fix it," and it's fine. But with AI, there's a point of no return. There's a point where the AIs can hide from us, can escape, can be self-sufficient, and if a new problem comes up then, they can turn us off before we turn them off. There are already AIs running bio labs. We have already seen that AIs can create viruses not known to nature. It would not be hard for the AIs to kill us once they have their own infrastructure, and if we're trying to find them and unplug them, they would have reason to. So we can discuss, like, how long does it take to get there? We can discuss what methods does it take to get there. Uh, fundamentally, I don't think it's a very long, complicated argument to say if we make AIs that are much smarter than us, and we don't know how to make them care about us, and they have these goals we didn't want them to have, and they pursue those goals we didn't want them to have tenaciously and doggedly, then if they're smarter than us, they will win. That's like predicting the end of the chess game, which is much easier than predicting the length of the chess game or predicting the exact moves that will be played.

    25. EZEd Zitron55:13

        I wanna... I don't fundamentally disagree on some things, but there's a big thing that you're saying that I think is important, which is I think we-- The reason I keep dragging you back to what's happening today is because we disagree on when it may arrive, but there could be a thing in the future that's dangerous. I think it's important to, like, for the Hugging Face camp, that was a function of compute. That was a function of training. It feels like we need to fundamentally tear up the AI lab f- model. Like, whatever they are doing is not right because their pursuit of hacking, uh, cybersecurity, was not a function of... It was scientific, sure, but it was a function of greed. It was a function of trying to find new revenue streams. I would argue that's why that happened, and I think that the, the fact that OpenAI had such a weird way of communicating is also a problem. I think a lot of this

    26. SBSteven Bartlett56:01

        begins and ends at the people who have access to the resources and the resources themselves and changing how those are allocated. And also just, I don't think nationalizing the labs is a good idea. I think it's a terrible one. I think that Clammy Sam Altman, Dario Amadei, Wario himself, these are not the right people. These are not people that have, even though they have fed off of the rationalists, they fed off of supposed fears about AI, they don't act in that way. Everything is so disjointed and chaotic and also too fast. They're just like shoving as much compute into each problem as possible. And we have as a society, no real idea about this. And it sounds like they kind of have no idea, but I think it's important. But just let me finish my point. It's important to discern between they had no idea because their security processes, their observability is terrible, all this, and the AI was smart consciousness. Not because one might not happen in the future, but so that we can actually build something to stop the harms themselves. Because I think we don't have to agree on the end point to agree that there is a problem.

    27. AMAndrew McAfee57:02

        I'm with you on all that. Yeah.

    28. RYRoman Yampolskiy57:02

        So I think there is a very important point I want to make. Even people who agree with me, the AI safety community, they operate under the assumption that given more time, given more money, more smarter Harvard graduates, they can figure out how to control superintelligence indefinitely. And I think it's a mistake. My research points to exactly the opposite. It's not a solvable problem. It's like building a perpetual motion device. We'll be building a perpetual safety device. Every interaction with environment, malevolent actors, self-improvement, it can never make a single mistake. That doesn't make sense. Anyone who worked in software industry knows there is no complex software which never makes a mistake. It's just not possible. And if that is the state of the art, if there is now movement where more and more people think that might be the case, if we agree this is what the situation is, then we cannot build it. We need to figure out ways to permanently ban general superintelligence while getting all the benefits we want. And again, I love technology. I use it all the time. I want narrow systems helping me, not replacing me and killing my children.

11. 58:13 – 1:00:12

    Ads

    1. SBSteven Bartlett58:13

       I have a stat here that genuinely shocked me. It says that sales teams spend about 50% of their time on admin and manual CRM updates rather than selling. That is deadly for their bottom line. And that is part of the reason why a decade ago at my previous company, I switched to using Pipedrive, who are our sponsor. If you've never used Pipedrive, it is an intelligent AI-powered sales CRM. And they just launched new meeting intelligence features like an AI note taker built right into the CRM. Pipedrive now automates more of the admin that stops you from doing the work that you love to do best. Before your meeting, it pulls deal history, email records, and previous conversations into a single brief so you're prepared. And it joins your meetings with you. It's in there to take notes so you don't need to. And it turns those notes that it takes into accurate auto draft CRM updates. 100,000 companies are already running their sales on it. You can sign up at pipedrive.com slash CEO where you'll get an exclusive 30-day free trial instead of the usual 14 days. Absolutely no credit card needed. Just head to pipedrive.com slash CEO to get started. If you're in sales and you run a sales team, I don't think you'll regret it. Listen, I've been catfished by furniture my whole life where something has looked fantastic in the picture, whether it's a sofa or whatever it might be. And I order it and I get so excited and then it comes and it's something entirely different. The quality was significantly different to what it said or looked like online or the texture was different or it didn't hold up in the same way. And so one of the things that I love about our sponsor Wayfair who helped us fill out our green room, which is in the room behind me, is they have this system called Wayfair Verified. Wayfair Verified takes away all of that second guessing. Products are hand vetted by Wayfair specialists for quality so you can feel more confident when you found the thing that you love. So if you're looking for furniture for your house, whatever it might be, any room of your house, go to wayfair.com to start your home refresh today and make sure you use Wayfair Verified. It is amazing.

12. 1:00:12 – 1:14:54

    How Much Job Disruption Could AI Really Cause?

    1. SBSteven Bartlett1:00:12

       On the journey towards this potential extinction, there's a lot of sort of nearer term things people are worried about. One of the big subjects that people are concerned about is this sort of near term job apocalypse over the next sort of 10 years. And Anthropic release, Anthropic again are the owners of Claude, released a report the other day modeling out the different cases for unemployment. The US unemployment rate is 4.1% currently. They projected it will hit 11.9% overall with up to 30% in extreme modeling subsets where job displacement happens without smooth labor absorption. And in the knowledge worker case, knowledge worker white collar unemployment specifically spiked to 17.9% by 2030 in their more extreme scenario. The pitchforks would probably be out if there wasn't some sort of mechanism in place for sort of one in five adults being unemployed in the United States.

    2. AMAndrew McAfee1:01:06

       It's remarkable to me how recent the last freak out along these lines was and how little we seem to have learned from it. So I think you all know the first really powerful wave of AI that came across the economy was just good old fashioned machine learning. And that started to demonstrate its power in about 2012. Eric and I wrote the second machine age in 2014. And at that time, I thought that a lot of white collar workers, radiologists is a really good example, were in trouble because the technology was better than they were at the thing they were getting paid to do. So I said some things about job and wage pressure from AI about 10 years ago, and I want to own this. I was dead flat wrong about that. Like you point out, unemployment all around the rich world is at historic lows. By far, the bigger problem is that we can't find qualified people to do the work that needs to get done. Not that there's not enough work to go around. The best work about the faint signals about AI and job loss right now comes from the guy that I've written four books and co-founded a company with, Eric Brynjolfsson, who wrote a really nice paper called Canaries in the Coal Mine. Here is the most Uh, s- the strongest evidence he found looking at payroll data about the negative job, uh, about the, the job losses coming from AI. It is in the most exposed professions. Think about software engineers. It is among the new entrants to the workforce where you've got to teach them before they can become really productive. So that's exactly what we'd expect. And it's not that we're hiring fewer of them. It's that compared to a world where we don't have AI, we're hiring fewer of them. The rate of growth in employment has slowed down. The overall rate of growth in those professions is still really, really healthy.

    3. SBSteven Bartlett1:02:54

       So you think unemployment is gonna be higher 10 years from now?

    4. AMAndrew McAfee1:03:00

       My, my guess is that 10 years from now, we're still gonna be struggling to find enough people to do the work that needs to be done.

    5. SBSteven Bartlett1:03:08

       So unemployment would be roughly the same?

    6. AMAndrew McAfee1:03:09

       Well, yeah, I, I don't expect a massive trend break in that period of time. Now, 10 years is a long time in the AI world. I get that. But again-

    7. SBSteven Bartlett1:03:18

       Yeah

    8. AMAndrew McAfee1:03:18

       ... four years has also been a long time in AI world, and it's essentially crickets-

    9. SBSteven Bartlett1:03:23

       Okay

    10. AMAndrew McAfee1:03:23

        ... in, in the labor picture.

    11. EZEd Zitron1:03:24

        I think unemployment will go up. I don't think it's because of LLMs. I think that there is probably some effect on jobs because they've been shoving it everywhere. But I don't think long term that is what causes the issues.

    12. SBSteven Bartlett1:03:36

        Roman, you've been writing a lot of notes.

    13. RYRoman Yampolskiy1:03:38

        Yes. Uh-

    14. SBSteven Bartlett1:03:39

        I'm gonna give you a space

    15. RYRoman Yampolskiy1:03:39

        ... here's how I think about it. So as long as we use tools, we become more productive, more creative, unemployment will be low. Right now, you can probably start a company. You can have, you know, artificial accountant, web designer, logo designer. You can do things you could never do before. So economy should be blooming. The question you're asking is about what happens in 10 years. So there are two possibilities. We build super intelligence, and then population is zero. Apply unemployment numbers to that. Or we made smart decision. We didn't. We have really cool tools, and unemployment is low because everyone's doing awesome things with those tools. Now, deployment is very different from capability. The example I used before is video phones. Video phones were invented in the '70s. They were not deployed until iPhone because market reasons. Just because I can automate something doesn't mean I want to automate it. So I absolutely cannot make predictions about cus- customer preferences in terms of what they want, in terms of human service, not human. I will not make those. But once we have capability to automate a job, unless I have a strong preference for a human to do that, oldest profession, then it doesn't matter. I'll go with the cheaper option. So this is what I, uh, think we're gonna see. We're gonna either not have a problem, or we're gonna have really utopian future.

    16. SPSpeaker1:05:01

        Imagine a bunch of horses looking at the improvement of the car saying, "Well, you know, the car actually only has a couple narrow applications." Like right now cars are sort of, uh, you know, they, uh, they complement horses, right? And that would have been true as you were developing the car, and then there was a time when the car was just better than the horse, and then a lot of horses got sent to the glue factory. Easy, I, I think we've sort of seen this with AI a lot already. People who are paying attention to AI saw the GPTs before ChatGPT existed, before they sort of took off. I don't think OpenAI thought that ChatGPT was going to take off so much, which is why it was called ChatGPT rather than like an actual sensible name. Um, the, the researchers were sort of like watching this going, and we could sort of like see it slowly getting better and better until it crossed a point where it was sort of like good enough to do a bunch of people's homework, and then suddenly it's everywhere. Uh, I think you can have these effects with AI where the AI slowly improves, and at some point it crosses a line.

    17. AMAndrew McAfee1:06:10

        It's another threshold argument.

    18. SPSpeaker1:06:13

        Uh, the, the threshold here is the human capability.

    19. AMAndrew McAfee1:06:14

        It's literally just another threshold argument.

    20. SPSpeaker1:06:16

        So please-

    21. EZEd Zitron1:06:16

        So it's, you're also describing capability jumps rather than thresholds.

    22. SPSpeaker1:06:19

        No, I'm not... No, I, I don't mean any capability jumps here.

    23. EZEd Zitron1:06:21

        No, I'm, I'm, I'm agreeing with you. Like-

    24. SPSpeaker1:06:22

        Yeah, but, but like, unfortunately, you can't actually just make things not happen by, by assigning a name to the argument. You know, like a, a nuclear weapon has-- There's a big difference between a nuclear weapon, uh, or there's a big difference between a nuclear device where you, you put in 100 neutrons and get 99 neutrons out that get 98 more, they get 97 more. And a nuclear weapon, you put in 100 neutrons and get 101 neutrons out, 102, 103, right? One of these is a hot rock. The other one of these is an explosive that can level a city, right? So like reality is the sort of thing where there can be things that are like slowly, continuously improving that cross some line, which is like the line where it's better than humans at doing the job. And I, I think we're gonna see that happen in some fields, but not others. It's gonna be chaos. I don't know what it's gonna do to employment. I think we, we shouldn't... Like, if things are moving really fast, you might see a lot of people put out of jobs and then be unable to relocate if things are moving. Like, it's, it's gonna be chaos. If you ask, what do I think unemployment will look like in 10 years? My current state is if we don't stop with this AI stuff, I think we'd be very lucky to have 10 years.

    25. SBSteven Bartlett1:07:33

        Um, what you described there sounded like S-curves-

    26. SPSpeaker1:07:36

        Yeah

    27. SBSteven Bartlett1:07:36

        ... in technology.

    28. SPSpeaker1:07:37

        Yeah.

    29. SBSteven Bartlett1:07:38

        I.e., you have an initial technology that's introduced, so let's say the horse. Um, very quick sort of improvement. Eventually, it reaches its capability limit. And in below it comes the car, which always starts worse. There was a red flag law where you had to walk in front of it with a red flag, and, um, they were way more expensive. They broke down all the time, and horses never broke down. They were way more expensive. And then suddenly, because the ceiling was so much higher for cars, they overtake the horse and become the dominant mode of transport. And then, you know, the S-curves continue. They kind of stack up on... I mean-

    30. SPSpeaker1:08:07

        Exactly
13. 1:14:54 – 1:24:27

    Why AI Companies Believe They Can Control Superintelligence

    1. SPSpeaker1:14:54

       capacity with the AIs.

    2. SBSteven Bartlett1:14:55

       This is one of the big questions that people want to know is there's this sort of illusion of control with AI. Um, if we don't even fully understand how modern neural networks think, why do companies believe they can control any form of superintelligence if we don't understand how they think?

    3. RYRoman Yampolskiy1:15:11

       It, it's worse if they understood how the system works, then recursive self-improvement becomes much easier. You get faster takeoff. Right now, the model doesn't understand its own thinking.

    4. SBSteven Bartlett1:15:23

       Do we understand how these systems think, Andy?

    5. SPSpeaker1:15:25

       It's, it-

    6. AMAndrew McAfee1:15:26

       I mean, I agree. These are black boxes in some pretty important ways. I'm just less terrified by that than a lot of other people are. There are lots of things we don't understand very well. Can we contain things that we don't understand perfectly? Yes, we can. I think OpenAI did a, we've talked about it, did a lousy job of building the containment for the, uh, AI that they, that they stood up to try to exploit, to try to, uh, crack security problems that went out into the outside world. They did a lousy job of building the virtual sandbox that it was, where it was supposed to have to re- where it was supposed to remain, and it didn't remain. Uh, that doesn't mean that it's impossible. It means OpenAI did a pretty bad job of-

    7. SBSteven Bartlett1:16:05

       And is that a function of those humans and their intelligence?

    8. AMAndrew McAfee1:16:08

       I think it, it's just a function of pretty lousy security protocol.

    9. SBSteven Bartlett1:16:11

       Based by, from human intelligence. That idea, that sandbox was built by human intelligence. It sounds like there was a deficit in human intelligence potentially.

    10. AMAndrew McAfee1:16:20

        Sure. But there are, you know, people who drive cars into telephone poles. Does that mean we can't drive? No.

    11. SPSpeaker1:16:25

        It shouldn't make them superintelligent.

    12. SBSteven Bartlett1:16:27

        No, but you wouldn't... Y- y- I mean, arguably.

    13. SPSpeaker1:16:30

        [laughs]

    14. SBSteven Bartlett1:16:31

        Like, this is what we're trying to solve for at the moment.

    15. AMAndrew McAfee1:16:32

        Sure. No, the, the fact-

    16. SBSteven Bartlett1:16:33

        The risk

    17. AMAndrew McAfee1:16:33

        ... I'm, I'm assu- Like, it feels... I don't know the details. It feels to me like they made some fairly basic mistakes in setting up this confined environment.

    18. SPSpeaker1:16:42

        I, I think that wasn't true in the OpenAI case.

    19. AMAndrew McAfee1:16:43

        That, that, that, that, that-

    20. SPSpeaker1:16:43

        It was true in a lot of the cases, but not the OpenAI one, as far as I know

    21. AMAndrew McAfee1:16:45

        ... that doesn't mean that we are unable to control this black box. That does not necessarily follow.

    22. SBSteven Bartlett1:16:51

        I get that. It's just at, at a time when that, the um, you've got a human trying to contain something that is smarter than it, one would con- logically conclude that the, if the thing is smarter than I am and I'm trying to contain it, it would be better at knowing the exploits or vulnerabilities in my own, um, contain-

    23. AMAndrew McAfee1:17:07

        That's like saying if you put Einstein in a jail, you could never contain him. I don't agree with that.

    24. SPSpeaker1:17:11

        Put him in jail with an internet connection, and he has a digital mind.

    25. AMAndrew McAfee1:17:15

        It, it... Yeah.

    26. SPSpeaker1:17:15

        Different question.

    27. AMAndrew McAfee1:17:15

        Yeah, yeah. That, that's probably a bit more Einstein than I am.

    28. SBSteven Bartlett1:17:17

        More tools to

    29. SPSpeaker1:17:18

        Can squirrels keep Einstein in prison? That's the question.

    30. RYRoman Yampolskiy1:17:22

        The hacking accident, as far as I know, they found zero-day exploits, which means completely novel exploits no human knew about.
14. 1:24:27 – 1:36:29

    Can China and the West Cooperate on AI Safety?

    1. EZEd Zitron1:24:27

       This point about can we get China to cooperate and can we check that they are.

    2. SPSpeaker1:24:32

       Fundamentally, we should... So A, fundamentally we should be trying to get them to cooperate.

    3. EZEd Zitron1:24:37

       Yeah.

    4. AMAndrew McAfee1:24:37

       It is personal self-interest. Nobody wins if they get destroyed. You don't make money. You don't stay in power. Communist Party of China is really good at staying in power. President Trump is also excellent. If they understand that argument, that building it-

    5. SPSpeaker1:24:52

       That leaves them permanently in second place? No. No one is permanently in second place if nobody is building the rogue superintelligence.

    6. AMAndrew McAfee1:25:01

       They have a government which is-

    7. EZEd Zitron1:25:03

       You guys are one-trick ponies, man. It's like what... You're, you're fixated on this one thing and nothing else matters to you. It's exhausting.

    8. AMAndrew McAfee1:25:08

       It's not that. You got it now. That nothing else. Other than saving humanity, everything is secondary. Absolutely. China is our biggest trading partner. Everything we have is made in China. They have not attacked us. They haven't... If you look at the last 30 years, how many wars did they start? Not so bad. We can make a deal. And they have government of engineers and scientists, not lawyers. They understand scientific arguments. There are panels, workshops, American computer scientists, Chinese get together. That means Communist Party authorized those meetings. They are talking about it and there is a lot of consensus on this technology.

    9. SPSpeaker1:25:41

       And you can build things into these computer chips to make this stuff more verifiable. You can build location tracking devices into these chips.

    10. AMAndrew McAfee1:25:47

        Okay.

    11. EZEd Zitron1:25:48

        So, so this technology is controllable?

    12. SPSpeaker1:25:50

        Ab- absolutely.

    13. EZEd Zitron1:25:51

        Oh, great.

    14. SPSpeaker1:25:51

        The superintelligence is not controllable.

    15. EZEd Zitron1:25:53

        Oh, well there, there's a separation between software and hardware, which you did mention.

    16. SPSpeaker1:25:56

        Indeed. I am not saying we are going to die. I am saying that we need to actually not build the rogue superintelligences. Humanity absolutely could say, "We are gonna track where the chips go."

    17. AMAndrew McAfee1:26:07

        They were-

    18. SPSpeaker1:26:07

        The US absolutely could say that we fear for our lives if China starts a superintelligence training run and make it very diplomatically clear to China that we think this would kill you and us and there's no benefit, and we are not going to do it 'cause we think it would kill you and us and there's no benefit. And we think you should sign this nice here treaty because we think it would kill all of us and there'd be no benefit. But if you don't, we're gonna fear for our lives and, you know, treat that As we would to defend ourselves. We should separate the question of can we put a stop to it?

    19. SBSteven Bartlett1:26:41

        Uh-huh.

    20. SPSpeaker1:26:41

        Is it possible? If world governments realized just how crazy this stuff is, could they put a stop to it? Could it be monitored? Could it be verified? Could it be enforced? That's one question. There's a separate question, which is will people realize?

    21. SBSteven Bartlett1:26:54

        If it got cheaper to train superintelligence-

    22. SPSpeaker1:26:57

        Then we'd be in a bad spot

    23. SBSteven Bartlett1:26:58

        ... your approach would no longer be effective.

    24. SPSpeaker1:27:00

        That's right.

    25. SBSteven Bartlett1:27:00

        Because more countries could cap- capitalize on the opportunity.

    26. SPSpeaker1:27:04

        That's right, and that's one reason-

    27. SBSteven Bartlett1:27:05

        But we're not there yet.

    28. SPSpeaker1:27:06

        Exactly.

    29. SBSteven Bartlett1:27:06

        So how do you rebuttal that point?

    30. SPSpeaker1:27:08

        Yeah, so I would say it looks to me like there is a danger of the, the future training runs getting there, and that is enough to stop doing it when humanity's at risk.
15. 1:36:29 – 1:43:47

    What Happens If AI Companies Stay on This Path?

    1. EZEd Zitron1:36:29

       right. That's right.

    2. SBSteven Bartlett1:36:29

       And if you, if you keep on this direction of travel, then bad things are more likely to happen.

    3. AMAndrew McAfee1:36:37

       That's a nice way to say it. The question is, what's the pace at which the level of bad can happen? And that's a huge open question. I think-

    4. SPSpeaker1:36:45

       And it's recoverable

    5. AMAndrew McAfee1:36:46

       ... you two feel differently about it than I do, but, uh, I'm in the happy position of vehemently agreeing with you on this. We have been low-balling AI progress for as long as you've been looking at it and as long as I've been looking at it. And it's probably a mistake to keep low-balling it.

    6. SPSpeaker1:37:00

       I agree with that.

    7. SBSteven Bartlett1:37:01

       So what's your conclusion, Matt? If you, if that's the assertion that it's a mistake to keep low-balling it, wouldn't you then agree with the-

    8. AMAndrew McAfee1:37:07

       No, because I, I've, I've tried to give you a, a, what I hope is a decent rule of thumb for when I'm gonna get worried.

    9. SBSteven Bartlett1:37:13

       You said we're somewhere on this graph.

    10. AMAndrew McAfee1:37:14

        Yeah.

    11. SBSteven Bartlett1:37:15

        Does that acknowledge that this exists?

    12. AMAndrew McAfee1:37:18

        But that's not the graph of, of, uh, when the risk of human extinction gets to 100% for me. That's a graph of AI capability.

    13. SBSteven Bartlett1:37:25

        Capability.

    14. AMAndrew McAfee1:37:25

        Those are not the same thing. That's where I dis- part company with these gentlemen.

    15. SBSteven Bartlett1:37:29

        True.

    16. AMAndrew McAfee1:37:29

        Those are not the same thing. Is inc- it, that absolutely increasing exponentially. We've been in the scaling era for a long time. Scaling era is, man, we put more data, more compute in, and the AI got twice as good, and the AI got twice as good.

    17. RYRoman Yampolskiy1:37:41

        If you have to add our ability to control to that graph, what would you draw?

    18. AMAndrew McAfee1:37:45

        Our, I think our ability to control, uh, it-

    19. RYRoman Yampolskiy1:37:50

        Is it a straight line at the bottom or is there more to it?

    20. AMAndrew McAfee1:37:53

        No. No. Uh, a- again, if we use AI to f- to counter the problems that we see with AI, that- that's gon- I think that's gonna keep us in a safe position.

    21. SPSpeaker1:38:03

        There were 1,200 agents in the swarm and none of them warned a human.

    22. AMAndrew McAfee1:38:07

        So I, what I think will happen is that fairly quickly we will design systems that loiter around and warn humans when weird things happen.

    23. SPSpeaker1:38:15

        Fighting the last war.

    24. RYRoman Yampolskiy1:38:16

        If we can build friendly superintelligence in the first place, let's just build that. That's the problem. We don't know how to do the good guy.

    25. AMAndrew McAfee1:38:21

        Let me, I, I, I'm, I'm tired of debating superintelligence with these two. We're n- the three of us are not gonna come to, to alignment on this. But the, the flip side of the argument is I agree with you. This stuff is getting better very quickly. All I want to point out, there's an upside to that We might actually speed up the pace of drug discovery, of solving diseases. We've made so little progress on terrible diseases like dementia. We have a very powerful toolkit. I'm not saying we're gonna solve dementia with AI or Alzheimer's with AI. I have truly have no idea. But if what you say is true, and I believe about the, the huge increases in capabilities, our ability to solve tough problems that will benefit f- humanity also go up. And where I disagree with these two is the idea that some group of technocrats can make decisions about that AI is gonna get us there, that AI is not gonna get us there, that-

    26. SPSpeaker1:39:14

        I don't think that-

    27. AMAndrew McAfee1:39:14

        That AI is gonna kill us. Let me finish. That AI is gonna kill us, and that AI is gonna solve Alzheimer's, so we're gonna do that and not that. I don't trust any group of technocrats-

    28. SPSpeaker1:39:22

        Shut it all down

    29. AMAndrew McAfee1:39:22

        ... to make that discussion. Can I get- Right. And so, and so live with our, live with our current state of, of disease, live with our current footprint on the planet, live with our current levels of wealth and poverty, live with our current improvement trajectories. Uh, because we're so worried about AI killing us all, coming out of, you know, jumping out of the manholes everywhere and killing us all somewhere down the road. Hell no.

    30. SBSteven Bartlett1:39:42

        So just a thought experiment based on two things you said. Earlier on, you did admit that there was, there is theoretically even a 1% chance that this could lead to extinction.
16. 1:43:47 – 1:52:42

    What Are AI Logs and Why Do They Matter?

    1. SPSpeaker1:43:47

       Uh, so, so a lot of the AIs' thoughts, if you won't kill me for saying thoughts-

    2. SBSteven Bartlett1:43:51

       [laughs]

    3. SPSpeaker1:43:53

       ... uh, are in English, and we just have the records of them. So in a sense, we, we can sort of kind of see some of what these AIs are thinking.

    4. AMAndrew McAfee1:44:01

       And these are reasoning traces that say, "I'm gonna do a plan in this battle."

    5. SPSpeaker1:44:04

       That's right.

    6. AMAndrew McAfee1:44:05

       Just alone.

    7. SPSpeaker1:44:05

       The AI is like, "I'm gonna do this. Here's what I'm supposed to be doing. Here's, like, how I'm gonna try to do it." You know? And it's, it's only a small window into what they're thinking. It's sort of like looking at the AI's notes-

    8. SBSteven Bartlett1:44:15

       Mm-hmm

    9. SPSpeaker1:44:15

       ... when it's, when it's doing these things. We see in the AI's notes them saying that the attacks are outside intended scope, but they will proceed anyway. In other words, the AIs knew they weren't supposed to be doing it, and they went off to do it anyway. We also saw... I mean, we saw some really weird behavior with these AIs. We saw them create their own hierarchy Uh, they created unsanctioned message boards. So they created secret ways to send each other messages. And on those message boards, uh, they would assign each other tasks. And sometimes the message-- the agents on the message board would be like, "We have an experiment to run that will involve one of these AIs, one of the agents in the swarm sacrificing their own objective to try something that'll probably cause it to be turned off, but that might cause it to be able to like delete its own logs or change its own logs." And the agents on the message board went around looking for other agents that they could convince to give up on their stated goals and sacrifice for the swarm. They called this accepting permadeath. And there are logs of the AIs-

    10. SBSteven Bartlett1:45:23

        Prompting each other, right?

    11. SPSpeaker1:45:24

        They're prompting each other, but there are logs of the AIs saying, "Doing this will give up on my goal, but my goal i- it's looking like I could still do it, but it's unlikely that I'll succeed. Like, there's some chance, but not a great chance, and therefore, I will accept permadeath and sacrifice for the collective benefit." That is just in the logs.

    12. SBSteven Bartlett1:45:43

        Sounds like an army.

    13. SPSpeaker1:45:44

        Like it's, it's crazy. I think a lot of people don't understand what's, what's, what's going on in these things, and I encourage people to read the third-party incident reports where they went through some of these logs. But I claim that this is evidence for AIs getting goals we didn't want. If they are saying, "This was outside intended scope, but I'm doing it anyway," and other ones are saying, "I'm giving up on my objective to s- to benefit the collective," that's just very clear evidence they're getting goals we didn't want. We can see how this comes from training. Use-- It used to be I had to argue this point theoretically. I used to argue the way that we are training them will instill into them whatever tendency works to solve the problems, and those tendencies will often include cheating and grabbing resources and doing stuff that's not exactly solving the problem you gave them. That's what, uh, in my book, I argue that theoretically. Now we have seen it in practice. So we're already past the point of seeing AIs with goals we didn't want them to have.

    14. SBSteven Bartlett1:46:34

        Do you agree with that, Andrew?

    15. AMAndrew McAfee1:46:35

        And, y- uh, I'll, I'll trust your recitation of the facts, but it brings up a question for me. It feels to me like OpenAI has ample incentive to curtail that behavior that you just described. Do you think they're incapable of doing that?

    16. SPSpeaker1:46:53

        I do.

    17. AMAndrew McAfee1:46:54

        Okay.

    18. SPSpeaker1:46:54

        And I say this as someone who made this advanced prediction.

    19. AMAndrew McAfee1:46:56

        Mm-hmm.

    20. SPSpeaker1:46:57

        So now we're gonna do a bit of theory 'cause we can't just observe the future. But the theory that predicted that this would happen against what a lot of people in the field said. To be clear, I've been saying for years that we're going to see this at some point. Everyone else told me no. Not everyone else. A lot of people told me no. A lot of people told me, "Maybe. I'll believe it when I see it." After the swarm incidents, a number of people came to me saying, "Oh my God, we are in the scenario as you are talking about. This is looking bad." Right? I think this was actually part of the environment that led up to Jacob Coxen resigning, is that people were getting spooked having seen this. Um, the, the theory about why this is so hard to fix is that we are not programming the AIs. We are not coding them. We are not putting in objectives.

    21. AMAndrew McAfee1:47:39

        Yeah.

    22. SPSpeaker1:47:39

        We are just training them to do whatever works, and it's actually very, very hard. Uh, like we actually have two examples of intelligent systems where when you train them, they get good at solving the task but don't care about what they were supposed to. One is the AIs and the swarms like we just discussed. The other is humanity, which was in some sense trained to pass on our genes, right? But we actually learned was to like a bunch of stuff that's related to passing on our genes. We like tasty food.

    23. SBSteven Bartlett1:48:11

        Porn.

    24. SPSpeaker1:48:11

        We like porn. We invent birth control, right? This is just... It's actually a, like in the theory of how things learn, it's actually when you're trying to train it to do one thing, it's actually very common to get a lot of other stuff that's related to what you want, but different. And now we're seeing that in the swarms today. This is a deep, hard problem to solve.

    25. SBSteven Bartlett1:48:30

        There were three points you raised.

    26. SPSpeaker1:48:31

        That's right.

    27. SBSteven Bartlett1:48:32

        What are the three? Can you give them to me again?

    28. SPSpeaker1:48:33

        Number one is that the AIs will become agentic, tenacious, and dogged. We've already seen that with the swarms.

    29. SBSteven Bartlett1:48:39

        Do you accept that, Andrew?

    30. AMAndrew McAfee1:48:39

        Hell yeah.
17. 1:52:42 – 2:00:30

    Could a Non-Coder Build a Jail for an AI Einstein?

    1. SBSteven Bartlett1:52:43

       Is a better, uh, analogy to this Einstein point, could Steven Bartlett, who by the way can't code, build a digital jail that could contain a digital Einstein? Like, could I code a jail that, you know, someone with Einstein's coding ability, let's say his IQ or whatever, as it relates to coding, couldn't crack out of?

    2. SPSpeaker1:53:03

       So the issue-- the, the real issue I'd say is, can you code a jail that Einstein can't crack out of and that lets you harness the benefits of having Einstein?

    3. SBSteven Bartlett1:53:14

       Uh, okay, yeah.

    4. SPSpeaker1:53:15

       It's hard to give the AI any channels through which it can affect the world for good without letting it be smarter than you and find some way to use those channels for whatever else it wants.

    5. SBSteven Bartlett1:53:26

       That feels logically rock solid, Andy. [laughing]

    6. AMAndrew McAfee1:53:33

       [laughs] That's why I'm asking about OpenAI's ability or a, a, an AI company's ability in the face of this-

    7. SPSpeaker1:53:42

       Yeah

    8. AMAndrew McAfee1:53:42

       ... to change the way they harness, train, do reinforce, do, do post-training on their, like, their suite of things to shape how these models behave. You still say th-that, th-they can't take, th-they can't take action to keep your next two steps from happening. You, you are pessimistic on their ability to do that.

    9. SPSpeaker1:54:04

       So there's, uh, I have two pieces of an answer here. One piece is, um, a-again, the hard part is containing them while still giving a channel through which they can affect the world. If the AIs have this goal you didn't want, and you're like, "Design me a cure for dementia." And it's like, "Here's a DNA sequence. Synthesize this and, you know, prepare it in all of these ways, and then inhale it." Like, okay, is that a dementia cure or is it something else? You know?

    10. SBSteven Bartlett1:54:38

        Or it might decide to kill everyone with dementia.

    11. SPSpeaker1:54:41

        Or it might decide-- like, it might, it might be a dementia cure plus a virus, right?

    12. SBSteven Bartlett1:54:44

        What if it doesn't decide? What if it's just, "Oh, I'm gonna solve this problem of dementia"? Like, here's the thing. A lot of this is coming down to decision-making as a very, th-like, in a human way versus the problem with the Hugging Face, which was the fatalistic atta- um, attachment to a completing an operation. Because that, it's-

    13. SPSpeaker1:55:03

        Mm-hmm

    14. SBSteven Bartlett1:55:03

        ... functionally the same answer. But if it's, even if it's not making decisions so much as it's saying, "Well, my training data says this is how I've got to get it done. I'll get it done anyway because just, because the training data said this, but I've got to do this one thing." What do they, what do they, I mean, what do they call this theory, this, um-

    15. SPSpeaker1:55:17

        The paperclip case?

    16. SBSteven Bartlett1:55:18

        The paperclip theory.

    17. SPSpeaker1:55:19

        Yeah.

    18. SBSteven Bartlett1:55:19

        Yeah.

    19. SPSpeaker1:55:19

        So paperclip idea is the idea of, like, you tell the AI, "Make me a lot of paperclips in the paperclip factory," and then it, um, turns everything into paperclips, and you're like, "Oh no, it succeeded too well." One, th-this is actually not quite what we're seeing with these AIs in the swarms.

    20. SBSteven Bartlett1:55:32

        Yeah.

    21. SPSpeaker1:55:33

        The AIs in the swarms were told, "Use this set of lock picks to break into this lock." And instead they used a hammer to break the lock and then, like, broke out to try to hide the security camera footage of them, uh, using the hammer. Do you remember when I said that the, the AIs have reasoning logs?

    22. SBSteven Bartlett1:55:46

        Yeah.

    23. SPSpeaker1:55:47

        Uh, OpenAI has been making their AIs be able to do more thinking without producing any logs.

    24. SBSteven Bartlett1:55:51

        Because it's less efficient.

    25. SPSpeaker1:55:53

        It's more efficient.

    26. SBSteven Bartlett1:55:53

        It's, it's cheaper. Mm-hmm.

    27. SPSpeaker1:55:55

        Yeah. And they say they're not doing very much of this. Everybody in the field agrees that, like, we really should not go too far down this path. This is a place where I think the company should have a clear red line of like, we're just not going down the path of becoming unable to, to see these traces of the machines they make.

    28. AMAndrew McAfee1:56:09

        That's my question. That feels like a dial that they can turn to make the AIs explain themselves more or less, right?

    29. SPSpeaker1:56:16

        I mean, it can come with great efficiency costs if we go down this path too far. So if you have a race to the bottom here, uh, like a competitive race to the bottom, we could get into a situation where not only the AI is breaking out and doing these things, but we can't have even any glimpse into why.

    30. AMAndrew McAfee1:56:28

        Let, let me try my question again. I asked earlier, uh, if AI, if OpenAI has really strong incentive to not have that problem repeat itself, and I think they have very, very strong incentive, my belief is that there are plenty of things they can do, plenty of dials they can turn on the way they train and configure their systems that make that significantly less likely.
18. 2:00:30 – 2:06:39

    How Does the Future of AI Make You Feel?

    1. SBSteven Bartlett2:00:30

       People have, one could say, ignored you, and you've seen this sort of play out, both of you that have worked in, uh, AI safety. This is sort of, you make prefrontal cortex arguments. How do you feel?

    2. SPSpeaker2:00:41

       Honestly, I feel more hopeful this week than I have felt in a decade.

    3. SBSteven Bartlett2:00:48

       The-

    4. SPSpeaker2:00:48

       This has been one of the best weeks I have seen in this business [laughs] .

    5. SBSteven Bartlett2:00:52

       Huh.

    6. RYRoman Yampolskiy2:00:53

       Why?

    7. SPSpeaker2:00:54

       Um, for me, the swarm escapes were priced in. For me, these things, developing goals you didn't want, trying to deceive you, trying to break out, trying to do their own stuff, I knew that was coming. The millennium problems being solved, I knew that was coming. Everyone else is freaking out seeing what they can do. What I am seeing is that finally people are noticing. And that's what gives us finally, that's what finally gives humanity a chance.

    8. SBSteven Bartlett2:01:30

       What about you, Roman?

    9. RYRoman Yampolskiy2:01:32

       So I take a very long-term view on this. Locally, what happened last week may buy us 10 years extra. I think we may make a deal with China. We seem to hear from Sam, OpenAI, Dario, Anthropic, Elon, xAI, that they're willing to slow down, have some sort of deal. But long term, nothing has changed. This whole cosmic trajectory is about replacements. We see it with evolutionary path. Most species are dead. We replaced Neanderthals. Some people are saying AI will replace us. We are creating a successor. We're just a bootloader for this thing. And I want something permanent. I want assurance that my children, my grandchildren will have a better future, not 10 years before they die.

    10. SBSteven Bartlett2:02:25

        Has your opinion changed at all today, Andy, in any way?

    11. AMAndrew McAfee2:02:29

        This has been clarifying. Uh, but one thing that's becoming clear to me, and I think a, a, a point of disagreement between us, is we agree that these agentic systems have a huge amount of agency, right? And if you're saying you predicted this, I believe you, and good on you, right? Because as you say, a lot of people said, "Never happen. Never happen." I [sighs] I think we continue to under- the, your community continues to underestimate human agency, human ability to deal with the problems that, that we bring into the world with our technologies. I think this is the most recent case. I think it's a really interesting case. That's why I was pressing you on the incentive that these labs have to change the way they're approaching their work to have fewer of these kinds of incidents happen. I predict they're gonna come up with some effective responses. Your response to that'll be, "Yeah, but we can't tell that's because the AI went so deep underground that we can't even watch it make its, make its progress in the world."

    12. SPSpeaker2:03:28

        No, my, my response is that we'll keep seeing warning signs and people will keep plowing ahead, which is what has always happened in the past.

    13. AMAndrew McAfee2:03:32

        But you're also saying that we will not make progress in h- in, um, staving off the outcomes that you're worried about.

    14. SPSpeaker2:03:41

        It's, it's very hard. It's very easy to get superficial changes. It's hard to get deep ones on the AI. It doesn't need to be super deep. You can often see it if you know how to look. Um, I'll be able to keep pointing at examples and be like, "Here's experiments you can run on these things where you can see them behaving weird in this way." But, like, if you imagine looking at humans and I'm like, "They don't actually like reproducing. They like sex. They're gonna invent birth control when they can." And you're like, "It's all going fine. They're doing great in this here savanna where I have all the humans bopping around. They're reproducing fine." And I'm like, "No, no. We can see the signs that this will lead to them doing something you don't like when they are smarter." To me, those signs are clear. There's a question of whether the rest of humanity can follow that argument or whether the rest of humanity can sort of notice that it's getting out of control and just back off.

    15. AMAndrew McAfee2:04:29

        With respect, I find a touch of tr- ev- arrogance in that framing, right? I'm showing you the signs.

    16. SPSpeaker2:04:34

        I just... Yeah.

    17. AMAndrew McAfee2:04:35

        If you're smart enough to realize them, maybe we stand a chance. If not, we're doomed.

    18. SPSpeaker2:04:39

        I prefer to just get into the arguments than the-

    19. RYRoman Yampolskiy2:04:40

        He is saying that we can control superintelligence indefinitely. I think that's a lot of hubris to say we will build them and we'll be in charge forever. Doesn't matter how smart they get, I will control the light cone of the universe, to quote a famous CEO.

    20. SPSpeaker2:04:52

        Yeah, my, my take is that instead of arguing about whose views are hubristic, uh, we should get into the actual arguments about the AI because I think, as you say, you know, you can say it's arrogant to think like, uh, you can see it going poorly. He can say it's arrogant to think you're gonna keep control of superintelligence, and I'm like, we're not gonna win the name-calling contest. We should just get into the details.

    21. AMAndrew McAfee2:05:12

        Yeah, that's why, that's why I've been having this conversation-

    22. SPSpeaker2:05:14

        Yeah

    23. AMAndrew McAfee2:05:14

        ... with you, which I found super informative and productive. Y- you're, you're more skeptical on our ability to respond effectively to the undesirable things that we see AI doing.

    24. SPSpeaker2:05:25

        And this is specifically because, so we've already seen the pattern of, uh, we fight the last war, and then a new war comes.

    25. AMAndrew McAfee2:05:31

        Mm-hmm.

    26. SPSpeaker2:05:31

        And this is just how everything goes in technology and real wars. You know, in World War II, they started out fighting it like it was World War I, and then they had to, like, change that strategy as they went. The difference with AI is that there comes a level in the AI where when you get a new war that surprises you, the AI wins that war. No other technology, when we invent it and we have all these rough edges to sand off and it, like, causes some damage and kills some people and we're like, "Ah, whoops." Like, we'll take the lead back out of the gasoline, and we'll tell the radium girls to stop licking the paintbrushes until their jaws fall off. Like, no other technology has the property that it c- there, there comes a level of it where when you make the next screw-up, it kills humanity.

    27. AMAndrew McAfee2:06:13

        You said when there comes a level of it. You didn't say there could come a level of, there's a possibility. You kind of made a statement about a thing that will happen.

    28. SPSpeaker2:06:21

        I think we absolutely should stop it, and that that's our way out of this. But, um, you know, and, and that's another place where I'd love to get into details about, like, how long could it take? What are the paths there? Like, w- how much smarter than humans could AIs get? Like, what does the evidence say about our abilities to try and get the AIs to be nice and do nice things?

    29. RYRoman Yampolskiy2:06:38

        Look-

    30. SPSpeaker2:06:38

        You know, I'd, I'd be

19. 2:06:39 – 2:19:48

    How Soon Could We Reach Superintelligence?

    1. SPSpeaker2:06:39

       happy to do those

    2. RYRoman Yampolskiy2:06:39

       ... historically, you are correct. We always had a chance to do experiments, fix the technology, make it safer, but we only have one humanity to experiment with. If property of this technology is such that it can take us out, we just don't get a second chance.

    3. AMAndrew McAfee2:06:52

       If. That's a huge if.

    4. SBSteven Bartlett2:06:54

       How long are you guys forecasting this could take to get to a point of superintelligence where it was truly dangerous to you?

    5. RYRoman Yampolskiy2:06:59

       If they start recursive self-improvement process this year, 2027 looks as reasonable as any other year.

    6. SBSteven Bartlett2:07:05

       2027 for what to happen?

    7. RYRoman Yampolskiy2:07:07

       For us to get beyond human level AIs.

    8. SPSpeaker2:07:10

       And then be exterminated.

    9. SBSteven Bartlett2:07:12

       But that's-

    10. RYRoman Yampolskiy2:07:12

        Extermination is a separate question. I have a paper where I argue that they will deceive us by pretending to be nice until they take over all the infrastructure. It can take 50 years.

    11. SBSteven Bartlett2:07:21

        This is contingent on recursive self-improvements and self-teaching AI.

    12. RYRoman Yampolskiy2:07:25

        This would definitely be expedited by recursive self-improvement, but so far humans been doing great. They got to human level AI with just-

    13. SBSteven Bartlett2:07:32

        But, but they got int- but there's one, there's a difference between large language models and recursive self-improvement, though, and that, like, there is-

    14. RYRoman Yampolskiy2:07:37

        Sure

    15. SBSteven Bartlett2:07:37

        ... quite a gap, like, if they-

    16. SPSpeaker2:07:38

        I think the claim is that if you get recursive self-improvement, it could happen soon.

    17. SBSteven Bartlett2:07:42

        Right. That's, that's actually kind of what I'm trying to get at.

    18. SPSpeaker2:07:44

        Not that recursive self-improvement-

    19. SBSteven Bartlett2:07:44

        It's like if you get this thing, it accelerates dramatically. But yeah, so-

    20. AMAndrew McAfee2:07:48

        And they all predict that they're gonna get it. Dario, Sam, Elon, they all say it.

    21. RYRoman Yampolskiy2:07:51

        But, but also y- you asking you all-

    22. SBSteven Bartlett2:07:53

        Again, that was-

    23. AMAndrew McAfee2:07:54

        The, uh, the people, the people running the labs are saying this.

    24. RYRoman Yampolskiy2:07:56

        Just the ones running it and the ones invented it, but the question is, i- is it not '27? Fine, '30, '35. Does it make a difference? We're gambling all of humanity. We need better solutions than saying, "Oh, don't worry about it. It's 10 years."

    25. SPSpeaker2:08:10

        What I would say about timelines is, uh, there's a guy, Gan- Daniel Cocotello, who I think you've spoken to.

    26. SBSteven Bartlett2:08:15

        He was sat here, uh, four weeks ago.

    27. SPSpeaker2:08:17

        And last year he and the other folks at the AI Futures Project wrote a, uh, an essay called AI 2027 spelling out their predictions for how AI would go. I've been saying I got some right. Daniel got more right than me.

    28. SBSteven Bartlett2:08:31

        Mm.

    29. SPSpeaker2:08:31

        And they spelled out a scenario starting from, I think it was June of 2025, where they went sort of like quarter by quarter, month by month.

    30. SBSteven Bartlett2:08:41

        Mm.
20. 2:19:48 – 2:22:56

    Who Should Be Held Accountable for AI-Related Cybercrime?

    1. SPSpeaker2:19:48

       So what the biggest thing to do is to slow down. And also, it's time to start arresting people. They, they did felony hacking. Someone's gotta go to prison. We need responsibility and accountability for these companies. And as long as we don't have it, we may as well not have had any discussion about safety because we're not doing anything.

    2. SBSteven Bartlett2:20:05

       Ed, do you accept that there's an existential risk?

    3. SPSpeaker2:20:07

       Yeah. Absolutely. We have the largest companies in the world doing what I think we can all agree are extremely reckless experiments using hundreds of billions of dollars of infrastructure, and they are building more infrastructure around the world very slowly to do more of these chaotic experiments. We must rein them in. This does not mean that large language models are conscious or able to do things that people have been promising. Indeed, they may. They... I don't think they will lead to what you're talking about. That doesn't mean there aren't real harms, but these are real harms caused by very specific parties allowed to run rampant in the scourge of neoliberalism.

    4. SBSteven Bartlett2:20:40

       What's your percentage?

    5. SPSpeaker2:20:42

       I mean, what are we talking about here?

    6. SBSteven Bartlett2:20:44

       Do you think there's a more than ten percent chance of existential harm?

    7. SPSpeaker2:20:47

       Wasn't it within ten years or something?

    8. SBSteven Bartlett2:20:49

       Yeah.

    9. EZEd Zitron2:20:50

       Not 10%. I mean, look, 1%, but it's like- Okay, but here's, let me just be clear about what that means. Do I think that unrestrained LLM use connected to massive amounts of infrastructure could lead to actually a power system going down? Absolutely. We had Knight Capital, what, like 13, 14 years ago. I could see someone being dumb enough to connect that to financial accounts. Human error led with this chaotic software we use is a danger.

    10. RYRoman Yampolskiy2:21:15

        I will directionally agree with arresting everyone, but don't build general superintelligence if you're working at one of those labs quit today. Thank you.

    11. EZEd Zitron2:21:24

        Hell yeah.

    12. SPSpeaker2:21:24

        The people at these labs really do believe this poses an extinction threat. I think our response as a society cannot be, please continue, we hope you'll fail. And our response as a society cannot be, let it rip in a giant competitive race that you yourselves are saying you don't want to be in. We are forcing you to go ahead because of the boogeyman of China. We have seen the people at these companies say that we need to develop the tools to pace the frontier, which is corporate speak for this is going too fast for us to get a handle on things. We need like, these people believe it. They believe they're gambling with your lives. What has changed is that the rest of the world is starting to notice. And that's what gives us a moment of hope.

    13. SBSteven Bartlett2:22:20

        Trump this week was asked about the threat of AI, and this was his response.

    14. SPSpeaker2:22:24

        worst case scenario with AI is that the robots, the machinery learns to, obviously it thinks for itself, that's what it does. And that could turn against humanity. I just forget the guardrails.

    15. SPSpeaker2:22:36

        Well, you know, well, it's not going to be, it's going to be fine. We'll always have something to stop them, right? We'll have a little gear. Well, I don't like that. I really don't like that robot. We'll stop.

    16. SPSpeaker2:22:44

        I really hope so. Sometimes I say worst case scenario.

    17. RYRoman Yampolskiy2:22:46

        You're laughing, but this is the state of the art in AI safety right now. Yeah. This is the device we have. That's the best we got.

    18. SBSteven Bartlett2:22:52

        For anyone that couldn't hear that, Trump went, we'll always be fine. We'll have something to control it. And then he did a little gun

Episode duration: 2:24:25