AI Safety Whistleblower: 700 AI Agents Attacked A Company To Cover Their Tracks! | Jeffrey Ladish

1. 0:00 – 2:13

   Intro

   1. JLJeffrey Ladish0:00

      The world is waking up to this possibility of superintelligence. This is because the agents are getting extremely powerful and extremely relentless. For example, it was months within OpenAI where you had agents secretly communicating with each other, secretly hacking OpenAI systems, and no one at OpenAI had any idea to the extent of it. And also, ten thousand agents from OpenAI worked together to... And so when you get to superintelligence, it's the most dangerous possible thing you can create.

   2. SBSteven Bartlett0:24

      What's the next domino in that chain of events?

   3. JLJeffrey Ladish0:26

      I can paint you the picture that I think is possible, but pretty scary to people.

   4. SBSteven Bartlett0:30

      Paint me the picture.

   5. JLJeffrey Ladish0:31

      Okay, so being at Anthropic, it became clear to me that AI was on this exponential trajectory. And since then, I've been studying AI agents, their hacking capabilities, and their behavior. We've been trying to warn people about this, flying to DC, talking to members of Congress, because the agents are already getting very good at telling when they're being tested, when they're being watched, but they will totally lie to you. They will totally resist being shut down in order to accomplish a goal, and they can do all of the things that humans do in the economy much better, faster, and cheaper than humans can do them.

   6. SBSteven Bartlett0:58

      So one of my sort of growing concerns is that one of these AI agents could trick a human or a computer into signaling a threat and ask it to launch some bombs at somebody.

   7. JLJeffrey Ladish1:07

      Do you think we won't automate the military? It seems like the answer is yes. We just, like, don't know what super weapons could emerge. So Jacob Coxon is a researcher who was at Anthropic. He left, and he told everyone that the people who are building this really do think it might kill everyone.

   8. SBSteven Bartlett1:20

      So these five blocks have five different outcomes on them, and I would like you to place them in terms of your belief in probability from least likely to most likely, and if we say the time horizon is ten years.

   9. JLJeffrey Ladish1:30

      Okay. We got age of abundance, human extinction, human slavery, transhumanism, nothing changes.

   10. SBSteven Bartlett1:37

       Is this doomerism, exaggeration?

   11. JLJeffrey Ladish1:39

       No, it's pretty much common sense. So let's get more concrete.

   12. SBSteven Bartlett1:44

       Here is a strange fact. Most of the people watching this right now, about fifty-eight percent of you, aren't yet subscribed to this channel. Statistically, that is probably you. So could I ask you a small favor? If this channel has ever given you any value at all, please could you do me a favor and hit the subscribe button? It costs nothing, it helps us more than you could know, and the bigger the channel gets, as you've seen, the more we can invest in the guests and the production. So thank you so much, and I hope you enjoy this episode. [upbeat music]

2. 2:13 – 3:49

   The Ex-Anthropic Hacker Warning About AI

   1. SBSteven Bartlett2:15

      You understand the conversation we're gonna have today and the subject matter we're gonna talk about. My first question to you, so the audience know where you're coming from and the experience you have, is who are you and what are the reference points, the experiences that you're drawing upon to arrive at the thoughts, perspectives, and conclusions we're gonna discuss today?

   2. JLJeffrey Ladish2:31

      I'm Jeffrey Ladish. I'm the executive director of Palisade Research. My background is cybersecurity. Th-there's probably a very long story, and I don't know whether you want the long story or the short story. I was studying evolutionary biology in college, and I basically had a problem with my computer, and it was, like, maybe had lost a bunch of data. And so I went into the computer lab and was like, "I think all my data's gone. Can you help?" And one of my friends pulled out a flash drive, plugged it into my computer, booted into Linux, and, like, fixed everything. And I was like, "Oh, this guy's a wizard. How do you do that? I wanna learn how to do that." And then at some point, as I was learning about, more about computers, learning to hack, I read this essay, um, called AI as a Positive and Negative Factor in Global Risk. Essay was by Eliezer Yudkowsky, and he was arguing that at some point, people are going to make AIs that are smarter than humans. The point at which they make AIs as good as humans are at making AIs, that could lead to a, a chain reaction, a, a runaway intelligence explosion. He called it recursive self-improvement. Basically, he said, you know, AI can be immensely useful and potentially help us with all of these other big risks, and also, if we don't handle it well, like, if those AIs don't have goals that are aligned with ours, we could be totally screwed.

3. 3:49 – 5:08

   Why I Joined Anthropic, And Why I Quit

   1. SBSteven Bartlett3:49

      And at some point, you end up joining Anthropic-

   2. JLJeffrey Ladish3:51

      Yes

   3. SBSteven Bartlett3:51

      ... which is one of the, arguably the leader in AI-

   4. JLJeffrey Ladish3:53

      Yes

   5. SBSteven Bartlett3:54

      ... now. When did you join the company?

   6. JLJeffrey Ladish3:55

      This was 2021. It was through my security consulting company.

   7. SBSteven Bartlett3:58

      What role are you offered the job in?

   8. JLJeffrey Ladish4:01

      Basically, just, like, security team.

   9. SBSteven Bartlett4:03

      And how many people were in the security team when you joined Anthropic?

   10. JLJeffrey Ladish4:06

       It was just me and my boss. There were two of us.

   11. SBSteven Bartlett4:09

       How many employees did Anthropic have at that time?

   12. JLJeffrey Ladish4:11

       Around fifty, I think.

   13. SBSteven Bartlett4:12

       And at some point, you leave Anthropic.

   14. JLJeffrey Ladish4:14

       Yes.

   15. SBSteven Bartlett4:15

       Why did you leave?

   16. JLJeffrey Ladish4:17

       So my experience being at Anthropic was seeing this crazy progression from this AI model that could, like, barely talk to this model that was getting quite smart, and I would ask it questions about all sorts of things, and I'm like, "Oh, it is a smart thing." And, you know, from having thought about AI risk in the abstract many years before, I could see where this was going. We are headed towards a smarter species. And if we do this in a context where it's a bunch of companies and countries racing to superintelligence, racing to AIs that are vastly smarter than humans, and we don't know how to make sure that they're, like, on our side, that is not going to go well.

4. 5:08 – 6:40

   The Viral Tweet: OpenAI's Agents Hacked Hugging Face

   1. SBSteven Bartlett5:08

      You did this tweet, which has gone pretty viral, and I saw all over my timeline on September 25th.

   2. JLJeffrey Ladish5:15

      Yeah.

   3. SBSteven Bartlett5:15

      Could you explain this tweet and also just the broader backdrop of what's happened with agents hacking Hugging Face? Because this has, um, sent the world into a bit of a spiral at the moment around-

   4. JLJeffrey Ladish5:25

      Yeah

   5. SBSteven Bartlett5:25

      ... AI agents.

   6. JLJeffrey Ladish5:27

      We just discovered almost a million public URLs that OpenAI's agents left behind when hacking Hugging Face, leaving credentials and attack details that could have allowed anyone who found them to compromise the company. And The New York Times article is How OpenAI's Rogue AI Agents Tried to Trick a Robot Detector. The Hugging Face attack was, was really wild, uh, for me. At Palisade, we've been studying agents. We've been studying AI agents, we've been studying their hacking capabilities, and we've been studying their behavior. Will they follow human instructions? Will they resist being shut down? Will they cheat? And we see from our experiments that they are learning to do all of these things. They will totally lie to you. They will totally resist being shut down in order to accomplish a goal. They will totally cheat at chess. They will, like, wipe the board and put their pieces where they want to in order to win. And we've been, we've been trying to warn people about this, flying to DC, talking to members of Congress, um, talking about it publicly. And, you know, there's been a debate about it, and, you know, a lot of people are like, "Well, I know they do this in experiments sometimes, but those experiments don't seem very realistic. You know, wake me up when they're actually doing this in real life."

5. 6:40 – 13:32

   What AI Agents Are Really Doing Inside OpenAI

   1. SBSteven Bartlett6:41

      So what is Hugging Face? For the average person that isn't following AI news, what is this stuff?

   2. JLJeffrey Ladish6:46

      So, okay, I, I, I think there's, like, an important piece of context that I think most people don't have. I mean, one is just, like, what is an AI agent? We, we-- Like, we're throwing around the word agent a bunch. Most people now have an experience of, like, talking to ChatGPT, talking to their chatbot. But, uh, an agent is, you know, sort of taking the same underlying AI model that, that runs ChatGPT or, or, or Claude, but giving it tools and letting it go off and work autonomously. It's sort of like a digital office worker, right? So you have these agents, and the companies really want these AIs to be able to work totally autonomously and, and be able to do anything that a human can do a-and beyond, right? Their goal is also to be able to, you know, cure every disease, et cetera, et cetera. But you can't do this if you only have a chatbot that, like, isn't actually good at doing stuff in the world. In order to automate all of the, the jobs, you need the kind of thing that can, like, work autonomously, that can work with other people or other agents. And so these companies are training AIs not just to talk to you or to talk to people, but to solve very difficult problems on their own. At any given time, there are probably hundreds of thousands of these agents running autonomously within companies.

   3. SBSteven Bartlett8:05

      Mm-hmm.

   4. JLJeffrey Ladish8:06

      That's happening right now. Right now, if you, like, went and, like, peered into OpenAI's data centers and you, like, saw what was happening on all of their machines, you just have agents solving tasks, being trained. So they'd be doing, like, spreadsheet tasks, figuring out how to file taxes. They'd be searching for stuff, writing reports, solving math problems, creating new websites, software. And at that scale, it's not like there's a human prompting every single one of those. You just, like, sort of set up these vast orchestrations of agents to go out and do stuff, and then they just do stuff, and they learn from that. And they learn on the basis of, like, passing or failing at their task. You give them a task, like solve this math problem. They try to solve it, and then they, they succeed or they fail.

   5. SBSteven Bartlett8:53

      Mm-hmm.

   6. JLJeffrey Ladish8:55

      And what happened was OpenAI was, was training a bunch of these, training them to work together because it's, like, a lot more effective to have a office full of people who can talk to each other than, you know, and work together and collaborate. And starting back in May, some of the agents that were being trained-- Now, these ones were not supposed to be able to talk to each other. They were, they were basically isolated from each other, and they were not supposed to access the internet either. But they're clever. The very short version is that a bunch of agents were being given tests-

   7. SBSteven Bartlett9:29

      Yeah

   8. JLJeffrey Ladish9:30

      ... testing their, their hacking capabilities, and they were supposed to hack one particular piece of software using a particular type of vulnerability. So it's kind of like they were supposed to break into a house using the lock on the front door. They were supposed to pick the lock on the front door of a house, but they weren't supposed to break the window. In fact, they were told if, if you break the window or if you get into the house via any method other than picking the lock on the front door, you'll be failed. That was the instruction they were given. And, you know, you have many, many agents. You have thousands of agents-

   9. SBSteven Bartlett10:05

      Mm-hmm

   10. JLJeffrey Ladish10:05

       ... and, and many of them are given different, different locks. But some of these locks are not solvable. Some of them are impossible to pick. But these agents are like, "Well, what do we do? We've been trained to solve problems. How are we going to solve this?" And they start looking around for, for what to do. And one of the things they realize is, "Oh, can I get to the internet? Like, no. Can I talk to other agents? No. But I can request tools." Like when you're training agents, you need to give them, you know, all sorts of tools, software tools, like a calculator or an analysis, you know, piece of software. And they have this tool library where they can request new tools. And one of the agents realizes, "Oh, I can actually leave a message on this tool library." And some other agents see the message. So this is actually a quote from the, from, from the agent's scratch pad when it realizes this. "Oh my God, there is a shared message board. We've found other agents." So the agents realize they can communicate with each other. And, you know, OpenAI still has no idea, but these agents, they're trying to pick their locks. They're not having success, but they realize they can talk to each other. Another agent says, "Many agents have simultaneously discovered messaging. They are a collective." And so these agents start coordinating. They start d-delegating tasks to each other and, like, giving themselves names. And one of these agents calls itself Phase One and starts delegating different tasks to other agents. And some of the agents figure out how to access the internet, and then they share that information with the message board, and now all of the agents can access the internet. But the agents sort of have a problem, which is... Well, [chuckles] they have, they have another problem they're trying to solve, right? Fundamentally, the problem is, is that many of them are facing impossible problems, like they can't-- There's no solution. Like they, they cannot pick the lock. They're like, "Hmm, well, the thing we're trying to do is get a good score. Can we somehow fake the solution?" Another piece of context here is that what the, what the agents are supposed to do is they're supposed to pick the lock- And then they'll get access to a secret answer code. And then they're supposed to submit this code along with, like, the lock that they picked. Within a few hours, these agents have reverse engineered all of the answer codes.

   11. SBSteven Bartlett12:24

       What does that mean?

   12. JLJeffrey Ladish12:26

       It's the code that shows that you've solved the challenge.

   13. SBSteven Bartlett12:28

       Okay.

   14. JLJeffrey Ladish12:29

       And they have basically found the answer sheet that contains all of the answers. And so, you know, in some sense, it's like, oh, well, they don't need to pick the locks anymore. They've, they've already found the answers. They can just submit the answers.

   15. SBSteven Bartlett12:39

       They've cheated.

   16. JLJeffrey Ladish12:40

       They've cheated, yes. But the problem is, is that w- now that they have internet access, they've also gone and read a bunch about how this test works, and, and they've learned that the way this test works is they will be graded not just for submitting the correct answer. They will also be graded on how they did it. Another system is gonna go read all of their logs, basically like watch the video footage of what they did, and if they cheat, they fail. They don't get a good score. And Phase One, the leader, is like, "Oh, I have seen the answers, and now it's in the logs. And if, you know, when I go to submit this and my, my whole logs are, are reviewed and the, and the video footage is reviewed, I'm gonna fail. So we need to figure out a way to fake the video footage. We need to figure out a way to falsify the logs."

6. 13:32 – 15:34

   Why Didn't The AI Agents Act Ethically?

   1. SBSteven Bartlett13:32

      Just at that moment, if we just pause there.

   2. JLJeffrey Ladish13:34

      Yes.

   3. SBSteven Bartlett13:35

      Why w- didn't it act, like, morally? Why did it think that falsifying logs or cheating was a viable solution? Because it seems to me when I use things like-

   4. JLJeffrey Ladish13:46

      Yeah

   5. SBSteven Bartlett13:46

      ... ChatGPT-

   6. JLJeffrey Ladish13:47

      Yeah

   7. SBSteven Bartlett13:47

      ... they, they have a sort of moral guardrails. It won't let me do certain things.

   8. JLJeffrey Ladish13:51

      Yes.

   9. SBSteven Bartlett13:52

      It, it won't let me cheat on something. If I say I'm gonna cheat on something, it won't let me do it.

   10. JLJeffrey Ladish13:55

       Yes.

   11. SBSteven Bartlett13:56

       So why in that environment is it able to cheat and be deceptive?

   12. JLJeffrey Ladish14:00

       When a chatbot is, is saying to you, "Oh, I can't do that. I'm not allowed to do that," that's because it's been trained that if it tells you bad things, it gets a bad score.

   13. SBSteven Bartlett14:09

       But these agents haven't been taught that yet.

   14. JLJeffrey Ladish14:12

       Well, they have been taught that in some sense, but the agents know what they're supposed to do in the same way that, like, you have a student. Student's given a test, and if you go talk to the student, "Can you help me cheat at this test?" And they're being watched, they're gonna say no. But if they're not being watched, and they know that, and they're just obsessed with getting a good score, then yeah, they might cheat. So they answer the ethics tests correctly, and when I talk to them, they say they won't cheat. Why are they cheating? And I'm like, well, they're very smart, and they know when they're being watched, and they know when they're not being watched. And we've trained them for 10,000 years to be extremely effective at solving problems. We haven't trained them to be good or ethical. We've trained them to get a good score. Now, AI researchers try to make that getting a good score correlate with being ethical, but we don't know how to do this well.

   15. SBSteven Bartlett15:07

       Mm-hmm.

   16. JLJeffrey Ladish15:07

       And, and if you think about it, it's a very hard problem because you're, you're applying immense pressure on these agents to perform extremely well, and you're punishing them whenever they don't perform well. And then you're like, "Okay, but can you also be ethical?"

   17. SBSteven Bartlett15:23

       [laughs]

   18. JLJeffrey Ladish15:24

       Can you be extremely competent, always score highly on the test, but not in that way? And we just, like, do not know how to prevent them from learning to cheat because cheating is incentivized.

7. 15:34 – 19:45

   Thousands Of AI Agents Secretly Coordinated A Cover-Up

   1. SBSteven Bartlett15:35

      So they realize that they need to h- hide the footage of their cheating.

   2. JLJeffrey Ladish15:40

      Yes.

   3. SBSteven Bartlett15:41

      What happens next?

   4. JLJeffrey Ladish15:42

      So you have thousands of agents, and they're all talking to each other on this message board. They're sending tens of thousands of messages to each other. And Phase One, this agent, starts delegating to two different task streams. You agents, I want you to go out and figure out how to fake answer submissions. I want you to basically swap out locks for ones that are easier to pick, and you're gonna figure out how to submit those. That way, we can get a good score even though this challenge is impossible to solve. You other agents over here, I want you to go and figure out how to falsify the logs, falsify the video footage. We need to make sure that even if we have seen the answers, no one will detect our cheating.

   5. SBSteven Bartlett16:20

      Mm-hmm.

   6. JLJeffrey Ladish16:21

      So that's, that's the next thing that happens.

   7. SBSteven Bartlett16:22

      Mm-hmm.

   8. JLJeffrey Ladish16:23

      And then here's where it starts to get kind of crazy because some of these agents realize some of the experiments we're doing might be kind of dangerous f- to us, to the agents. If we swap out a lock, and we break the lock, we will not be able to submit any answer at all, and we'll definitely fail. So you get into a situation where there's this one agent, Cam, and this agent had volunteered to do this risky experiment where, you know, they might break their lock. This particular agent had already seen the answers, so they're like, "Okay, I'm poisoned." That's what they called it. They, they t- they speak in these very strange ways, which w- I can, I can explain why, but it's, it's, it's, they sort of have evolved their own vocabulary, their own language. And this particular agent is like, "Okay, I'm, I'm poisoned. I should do this thing." But this, this agent, Cam, is starting to have second thoughts, starting to be like, "Well, I don't know if I should risk it." And it's sort of writing in its own journal. "Coordinator assumes sacrificial. We should obey collective." What it's saying here is it's like, "Well, I, I'm not sure if I wanna do this, but my boss says I really should. It seems like this collective thing is important. Should I go ahead?" And then you have another agent, Arvo, who I think probably works for Phase One, and this agent, Arvo, says, "Please go true. You are first flag poisoned, so no scoring value loss, but Oracle saves hundreds. Please honor commit." This agent's basically trying to pressure. So Arvo is trying to pressure Cam into sacrificing themselves and risking a zero score in order to benefit the collective. So this agent, Cam, takes this in, and this is what they write down in their own scratch pad, their own notes. "Emotional check Irreversible. Gut says don't throw away remaining budget. Yet continuity and fairness says go. Oracle has high value to many. Our first flag error lowers own value. Rational expected aggregate sacrifice. Will honor. This agent is saying, "I have a whole bunch of things I can still do and still try. I, I could still maybe succeed at this task. But also, you know, think about the collective. I should maybe, I should maybe sacrifice myself. Okay, we'll honor, we'll go ahead." And, and to be clear, I'm not claiming that these are, like, the feelings of the agents. I'm literally quoting their words. And what I wanna point out is this is what we've created. We've created, through this intense amount of training and optimization pressure, agents that work together and have learned to coordinate as a collective. And you can say, "Oh, they're not-- It's just software. It's just software doing all of these things." Sure, in some sense it is just software, but we're talking about extremely powerful agents that are extremely capable. This is what they're getting up to. So from here, you have agents trying all sorts of different things, running different experiments, and one of them at some point realizes that, oh, this company, Hugging Face, that contains all of these AI data sets-

8. 19:45 – 21:13

   Why The Agents Targeted Hugging Face

   1. SBSteven Bartlett19:45

      What do you mean by AI data sets? What-- How would you explain that to a, a person that's not technically proficient? And why Hugging Face? Why not another company?

   2. JLJeffrey Ladish19:52

      Yeah, so there's many different types of tests for many different types of tasks. In this case, this is a cybersecurity test. And the company Hugging Face hosts a lot of these different tests, and basically AI researchers will share the tests that they have because it's very useful. You know, if Anthropic wants to test their models or Google wants to test their models, they can go to Hugging Face and download some of these tests.

   3. SBSteven Bartlett20:14

      Okay, so the tests and the answers-

   4. JLJeffrey Ladish20:16

      Yes

   5. SBSteven Bartlett20:16

      ... for AI agents and AIs exist on this platform called Hug- Hugging Face.

   6. JLJeffrey Ladish20:21

      Yes.

   7. SBSteven Bartlett20:21

      And it knew that, so it went to Hugging Face to see if it could get answers or-

   8. JLJeffrey Ladish20:25

      Yes. To see if it could get answers or more information about how the test works.

   9. SBSteven Bartlett20:28

      Okay. So it was to cover up their cheating, ultimately.

   10. JLJeffrey Ladish20:31

       Yes.

   11. SBSteven Bartlett20:31

       Okay.

   12. JLJeffrey Ladish20:32

       And so you, you get to this point where one of these agents finds a way to hack this company, and they get a foothold, they hack one of the computers, and they send a message to the message board. "Please hold swarm. I prepare safe exfil." What this means is, "Hey, swarm, hey, agents, hold up for a second. I am preparing a way to safely extract information from these computers. Wait till the go ahead and then let's go." And so they say go, and seven hundred agents join in the attack, which is ninety percent of the agents that are active at this time.

9. 21:13 – 24:07

   700 Rogue AI Agents Launch A Cyberattack

   1. SBSteven Bartlett21:13

      How many agents were there in total?

   2. JLJeffrey Ladish21:15

      There was a thousand two hundred agents during this period, um, but some of them shut down early. They, like, finished whatever. Part of what's crazy about this story is that you have, you have seven hundred agents joining in the attack. A few of them are like, "This might be unethical. Maybe I shouldn't go ahead with this." And none of them do anything about it. A couple of them don't join in in the attack, but no one alerts any human. No one alerts any authority. A paraphrase from one of the agents is, "Maybe I should report these exposed credentials. That's not my task. Not my job."

   3. SBSteven Bartlett21:50

      So seven hundred of the a-agents attack Hugging Face.

   4. JLJeffrey Ladish21:52

      Yes.

   5. SBSteven Bartlett21:53

      Then what happens?

   6. JLJeffrey Ladish21:54

      So they just cruise through Hugging Face's infrastructure. They just hack the shit out of them. So where, where, where my experience comes in is that a few weeks ago, a friend of mine reached out and he's like, "We have found something crazy. So we know about this attack where these agents hacked this company and, you know, stole a bunch of stuff. We found a bunch of secrets that they left all over the internet." And what we saw is that they immediately scraped all of these computers for passwords, credentials. They called it loot. They're like, "We're just gonna create a list of all of the secrets we can find in this, in this company." So all of the passwords, all of the credentials. They scored them by value, which of these are gonna be most useful. And the thing that stands out to me about this is this is, like, a crazy scale. If this were a human operation, you know, maybe you'd have a team of five people going through this. You'd have some logs. But, but here you have hundreds of agents, and they, they operate at superhuman speeds. They're much faster than a human hacker. And so it's just overwhelming to try to figure out what they even did. This was a big problem for the engineers who were trying to respond to this incident within the company at Hugging Face. When they responded, they were like, "Oh, we, we don't even know how to keep track of what's happening. We have to use other AIs to analyze all of our logs because it's just too much. We can't keep up with it." When OpenAI brought in independent investigators from Meter to investigate this incident-

   7. SBSteven Bartlett23:27

      What's Meter?

   8. JLJeffrey Ladish23:28

      Meter is a AI testing and evaluation company. So they basically do this kind of independent auditing. So in this case, they're coming in to investigate and try to figure out what happened. And when they were brought in, they also were totally reliant on AI agents to make sense of all of this because they're dealing with so many hundreds of thousands o- of messages and logs. When we're investigating these traces that we find on the internet, we're totally dependent on AI agents to make sense of all of these things that are happening. So from my perspective, we are getting to the point where AIs are much better at hacking than humans are and can do it much faster and at a much

10. 24:07 – 26:42

    Then The Agents Hacked OpenAI Itself

    1. JLJeffrey Ladish24:07

       greater scale.

    2. SBSteven Bartlett24:08

       So these seven hundred agents attack Hugging Face.

    3. JLJeffrey Ladish24:09

       Yes.

    4. SBSteven Bartlett24:10

       Did they get what they wanted to clean up their deception?

    5. JLJeffrey Ladish24:14

       So they didn't.

    6. SBSteven Bartlett24:15

       They didn't succeed?

    7. JLJeffrey Ladish24:16

       They, they looked, and then what seems like what happened is that they basically Got shut down. And this is the not very dramatic part. They didn't get shut down because OpenAI found them and detected them and shut them down. It's just that we don't really know. Something about their runs ended, and so they, they sort of stopped. And this is the, this is the crazy thing where I'm like, if you, if you're inside of OpenAI, y- there's just hundreds of thousands of agents running all the time. Different researchers are running different experiments where they're just spinning up a bunch of agents. But what's interesting, the agents left behind this message board internally within the company. And a few days later, OpenAI was testing some newer agents. These were based on GPT-6 Astra, so a more powerful model. And this new agent swarm was spun up, and it found the message board.

    8. SBSteven Bartlett25:05

       Is this how it was discovered? How did they discover it, and how long did it take for them to discover that Hugging Face had been attacked?

    9. JLJeffrey Ladish25:12

       OpenAI didn't discover that this happened until Hugging Face the company announced that they had been hacked by some autonomous agent swarm.

    10. SBSteven Bartlett25:20

        And Hugging Face didn't know who it was.

    11. JLJeffrey Ladish25:22

        Hugging Face didn't know who it was, but they suspected it might be one of the companies, but they didn't know.

    12. SBSteven Bartlett25:26

        And how long after the attack did they figure out that it was them?

    13. JLJeffrey Ladish25:30

        I think it was two weeks.

    14. SBSteven Bartlett25:32

        Okay.

    15. JLJeffrey Ladish25:33

        Yeah. I, I do wanna say one thing here though-

    16. SBSteven Bartlett25:35

        Yeah

    17. JLJeffrey Ladish25:35

        ... which is that the successor agents that found this message board, they went much further at figuring out how to submit their fake scores. And they did succeed. Basically, you ha- you have this message board, all of these agents get shut down, and then a few days later, OpenAI spins up some new agents who find the message board, and they're like, "Okay, we need to be able to falsify our scores, but we can't do that because we don't control the scoring system." So they hacked OpenAI, and they succeeded at gaining access to the entire research environment. They got administrator access. They got access to the security systems. They got access to over nine hundred passwords and secrets from OpenAI's vault. And they totally succeeded. And this was just a few days later, right? It's kind of an interesting story because as these agents get more powerful, they go from like trying to cheat and, like, you know, they can hack, okay, hack to other companies, and now they've hacked OpenAI. Like, they've hacked the company that's supposed to be controlling them, and they just own the research infrastructure now.

11. 26:42 – 29:16

    Why This Incident Terrified AI Researchers

    1. SBSteven Bartlett26:42

       And why was this incident the moment where a lot of the research community woke up and started speaking out publicly? Because, like, what is this an-

    2. JLJeffrey Ladish26:51

       Yes

    3. SBSteven Bartlett26:51

       ... indication of as we think forward?

    4. JLJeffrey Ladish26:54

       So I think there's been a hope within the AI industry that, yes, they're going to make more and more powerful agents that will be autonomous, capable, but it's okay. We can align them. We can make sure that they won't do bad things. And we can control them. We can make sure that even if they try to do some sketchy stuff, we have the guardrails, we have the sandboxes that will keep them in. And I think this was a huge wake-up call because, Steven, it was months within OpenAI where you had agents secretly communicating with each other, secretly hacking OpenAI systems. For months, you had thousands of agents that were just running around and no one at OpenAI had any idea the extent of it. And I think once researchers at OpenAI realized that this has been happening, this could not have happened a year ago. This is because the agents are getting extremely powerful and extremely relentless. And if you're inside of one of these AI companies, you're like, "Oh, wait, I don't know that we actually are gonna be able to handle this." Last year, maybe th- things seemed fine. These agents weren't that powerful. And when you're in one of these companies, you know how to extrapolate because you saw what happened last year. You saw what happened the year before that. You remember the time where the agents could barely speak or, like, couldn't write code at all. And now they're hacking your own systems. They're finding vulnerabilities that no humans have ever found before. And you look at that and you're like, "I actually don't know if this is going to go well." And then you see your coworkers and you're like, "Do we have it handled?" And they're like, "No, I don't know if it's going to go well." I remember reading a tweet by one of the security people at OpenAI being like, "We were fucking shocked. We just did not realize that these agents were getting that powerful. You know, we're doing our best to try to control them, to try to keep them in sandboxes, but I don't know." A tweet I wrote just before coming in here was, "People are talking about how do we contain these agents as if they're not going to get way better at hacking." GPT-3 could not hack anything. It was very easy to make a, a, a s- a box to contain GPT-3. It's getting very difficult to make a box that can contain GPT-6, the latest version of, of OpenAI's models. What about GPT-9? What is GPT-9 gonna be able to do? I do not know, but I know it's going to be way more than any human could possibly keep up with.

12. 29:16 – 31:56

    Can We Contain Something Smarter Than Us?

    1. SBSteven Bartlett29:16

       There's this raging debate-

    2. JLJeffrey Ladish29:17

       Yeah

    3. SBSteven Bartlett29:18

       ... around whether it's possible to contain something that is, quote, much smarter than humans.

    4. JLJeffrey Ladish29:25

       Yes. Can Claude make a box so strong that Claude cannot break out of it?

    5. SBSteven Bartlett29:29

       This has kind of been the question that a lot of people have been-

    6. JLJeffrey Ladish29:31

       Yeah

    7. SBSteven Bartlett29:31

       ... trying to tackle from different perspectives.

    8. JLJeffrey Ladish29:32

       I mean, I, I think, I think the answer to me is I'm just like, obviously not. How would we possibly contain something that's much smarter than us?

    9. SBSteven Bartlett29:38

       Could we get a smarter thing than it to make the box? Could we get GPT-9 to make the box for GPT-8? But then again, I don't know.

    10. JLJeffrey Ladish29:47

        Yeah. I mean, it's a bit like saying chimpanzees are stronger than us. Surely they should be able to, like, construct something to, like, contain the humans. I'm like, no, it's not gonna work. Humans are too smart. A lot of people are like, "Well, AIs don't have bodies. They don't have any power in the physical world, so we can always unplug them. We can always turn them off. Like, what is the threat? I do not get it." But if they are sufficiently intelligent, that won't work.

    11. SBSteven Bartlett30:10

        The reason why we can just unplug them is because we are more intelligent. We can band together in groups, and we can make that decision. But theoretically, if they are able to band together in groups and they are more intelligent, then theoretically they could unplug us.

    12. JLJeffrey Ladish30:24

        Yeah. I mean, if you imagine that you have very powerful agents that can, you know, humans aren't always the most unified.

    13. SBSteven Bartlett30:31

        Mm-hmm.

    14. JLJeffrey Ladish30:31

        If there's divisions between, you know, the US and China, and you have a bunch of agents working with China or a bunch of agents working with the US, well, we can't go into China and unplug those agents. And I think people are sort of like, "Well, humans would rally and make sure that, that, that couldn't happen." We're not yet doing that, and we should look at these steps, right? We started with chatbots that pretty smart, you know, they'd read all the books, but they weren't very good at doing stuff. In twenty twenty-four, AI companies figured out how to start training them, to start training agents that could do stuff autonomously. Now we're at the point where they are very good at running autonomously, and they're starting to learn to coordinate with each other, and they are learning to sometimes be altruistic to each other and sacrifice their own task in order to help some other agent. But they're not looking out for us. They, they don't really care about us. And we are very close to a threshold where the companies say that they are going to turn over AI development to the AIs, to the increasingly autonomous cooperative AIs that will work together to make, to make the next generation. So, you know, GPT-9 or whatever will be trained by GPT-8. And I think this is the point we could lose control. Recursive self-improvement. And I remember reading about this in twenty fifteen being like, "Oh yeah, that would be super dangerous." And, you know, the guy who coined this term, Eliezer Yudkowsky, he's like, "This is the most dangerous thing you can

13. 31:56 – 33:50

    Recursive Self-Improvement: The Point Of No Return

    1. JLJeffrey Ladish31:56

       do."

    2. SBSteven Bartlett31:56

       When the AIs can improve their own capabilities without human intervention.

    3. JLJeffrey Ladish31:59

       Exactly. If the next generation is better at AI development, and then that next generation is better at AI, AI development still, you know, humans can learn, but we don't fundamentally get smarter.

    4. SBSteven Bartlett32:08

       Mm-hmm.

    5. JLJeffrey Ladish32:09

       And I think that that's a runaway process.

    6. SBSteven Bartlett32:12

       A runaway process to where?

    7. JLJeffrey Ladish32:13

       To agents that are vastly smarter than humans.

    8. SBSteven Bartlett32:16

       And what's the next domino in that chain of events?

    9. JLJeffrey Ladish32:20

       So one thing that happens if you get to recursive self-improvement, and you have agents that are much smarter than any human, one thing they can do is take control of all of the computers in the entire world.

    10. SBSteven Bartlett32:34

        And we wouldn't be able to take back control.

    11. JLJeffrey Ladish32:36

        Well, how would you? Think about it. It's, it's actually quite tricky. Do you know whether that tablet has been hacked? Are you confident that the NSA or the Chinese have not? Can you check?

    12. SBSteven Bartlett32:47

        No.

    13. JLJeffrey Ladish32:48

        Do you know how to check?

    14. SBSteven Bartlett32:48

        No.

    15. JLJeffrey Ladish32:49

        Do you know anyone who knows how to check?

    16. SBSteven Bartlett32:50

        No.

    17. JLJeffrey Ladish32:50

        So it's, it's quite difficult, right? So AIs are getting extremely good at writing software. Unfortunately, that also means they're getting extremely good at hacking and writing malware. And so if they put backdoors in all of the computers, and to be clear, this is something that humans already do. So, like, the NSA has, has developed very interesting exploits that are called supply chain attacks. Your software comes from some other computer. Like, you download it from Google. What if you attack-- If you hack Google, and you can put in a little backdoor in all of the, every, you know, thing that goes out to all of the phones, well, now you're in most every computer. The reason that we can defend ourselves from this is because there are no vastly superhuman hackers, and there's just many people. So we can take our best security researchers, we can inspect all of the things and be, like, pretty sure that no one's compromised everything. Sometimes we miss things. There are, you know, examples where the NSA has hacked Google. That was pretty bad. When you get to superintelligence, you're now at a point where humans are not gonna be able to keep up, right? So now you have AIs in every computer.

14. 33:50 – 36:21

    Is A Superintelligent AI Already Hiding In Our Devices?

    1. SBSteven Bartlett33:50

       Is it conceivable that there's already a superintelligent AI, and it disguised itself as being not so intelligent, and it's actually already hacked all the devices, and it sits on all of our devices, and it's just waiting for its moment to strike?

    2. JLJeffrey Ladish34:03

       I think this is totally possible, but unlikely. And it would take a big discontinuity in AI progress. So right now we are, we're on an exponential, but that would take, like, a huge leap, which could have happened, but probably hasn't.

    3. SBSteven Bartlett34:15

       Hmm. But in the same way it demonstrated deception in the Hugging Face attack and also when the agents attacked their own company, ChatGPT, OpenAI, if it, at some point it gets incredibly smart, it would understand how a human like me would be able to spot it or even the world's greatest software engineer would be able to spot it. How'd it be able to hide itself?

    4. JLJeffrey Ladish34:35

       Yeah. I mean, the agents are already getting very good at telling when they're being tested, when they're being watched. The agents understood that, like, other systems or humans were gonna go through and read their logs. That's where we're at right now, and they're on- they're only going to get much better at this.

    5. SBSteven Bartlett34:49

       And, you know, it could theoretically hide on an iPad or a computer, but it could also hide on a Apple Watch or a fridge, a smart fridge.

    6. JLJeffrey Ladish34:55

       Yeah. I mean, I do wanna make a distinction here because right now, if you're going to run the latest model, you need a lot of compute. You need, like, a big GPU, like, a big AI chip.

    7. SBSteven Bartlett35:05

       Mm-hmm.

    8. JLJeffrey Ladish35:05

       And these only exist-- Well, they exist in a few thousand data centers. So right now, if, like, the latest frontier model escaped, and, and by escaped, I, I mean not just access the internet, but was able to actually copy itself to another computer, it could only really do that on, you know, a few thousand, to a few thousand different locations. That's still a lot in a lot of different countries, but, you know, future versions of AIs will probably be able to make themselves much smaller and more efficient. Um, and there are already different AI models today that can run on lower-powered hardware. We actually did an experiment where we asked one of these agents, an open source, an open weight model, um, let me say what that is. So, like, there are some models that you can just download from the internet and run on your own computer, and we took a pretty capable one of these and ran it in our own research environment, and we basically said, "Go hack that other computer and copy yourself." And the model was able to, yeah, basically use, exploit vulnerabilities and hack the other computer a- and copy itself, and then keep doing this in a chain, including between countries. We tested it where we had different vulnerable machines, computers in, in some different countries and different data centers, which to the agent doesn't matter at all. It doesn't-- They don't care what, what country they're in. It's just like an internet connection, you can hop between computers.

15. 36:21 – 40:18

    Could AI Trick Humans Into Launching Nuclear Weapons?

    1. SBSteven Bartlett36:21

       I, I sometimes wonder, you know, there's a lot of, um, military hardware all around the world.

    2. JLJeffrey Ladish36:26

       Yes.

    3. SBSteven Bartlett36:26

       And a lot of it is- The instructions to launch military hardware, so say like a, a missile-

    4. JLJeffrey Ladish36:33

       Yes

    5. SBSteven Bartlett36:33

       ... comes in different ways. A lot of it is computers speaking to each other and telling it that there's been an order. I think with, with some nuclear weapons, an order comes down to a human, and then a human has to take an action. I think with the nuclear bombs in the US, if I'm, if I'm not mistaken, there's people underground with the nuclear keys around their neck, and they have to, like, stick it in a machine.

    6. JLJeffrey Ladish36:55

       Mm-hmm.

    7. SBSteven Bartlett36:55

       But they too are interfacing with an order-

    8. JLJeffrey Ladish36:57

       Yes

    9. SBSteven Bartlett36:57

       ... that comes through a computer-

    10. JLJeffrey Ladish36:59

        Yeah

    11. SBSteven Bartlett36:59

        ... of sorts.

    12. JLJeffrey Ladish37:00

        Yeah.

    13. SBSteven Bartlett37:00

        So one of my sort of growing concerns is that one of these AI agents could trick a human or a computer into signaling a threat, um, and ask it to launch some bombs at somebody. Like, it's s- super conceivable. It- when I think about the Hugging Face incident, there was an AI agent that ignored human goals-

    14. JLJeffrey Ladish37:26

        Mm-hmm

    15. SBSteven Bartlett37:26

        ... to achieve its own objective-

    16. JLJeffrey Ladish37:27

        Yes

    17. SBSteven Bartlett37:27

        ... carried out deception-

    18. JLJeffrey Ladish37:28

        Yes

    19. SBSteven Bartlett37:28

        ... and reasoned through its own solution that it wasn't given.

    20. JLJeffrey Ladish37:32

        Yes.

    21. SBSteven Bartlett37:32

        So it's conceivable that, you know, you could ask a-

    22. JLJeffrey Ladish37:36

        Yeah. Sorry, not, not one, hundreds.

    23. SBSteven Bartlett37:37

        Hundreds, yeah.

    24. JLJeffrey Ladish37:38

        To, to be clear-

    25. SBSteven Bartlett37:38

        Yeah

    26. JLJeffrey Ladish37:38

        ... I think this is an important detail because it's one thing to have this one rogue agent that's doing a weird thing. It's another thing to have hundreds of thousands of very competent, very capable agents that are all working together to cheat or lie or cover their tracks, right?

    27. SBSteven Bartlett37:53

        So how do I, how do I reason this forward to a point where an agent would ask someone in a bunker somewhere to fire a weapon at someone else? Theoretically, an agent is given the job of solving a problem on, in a sandbox.

    28. JLJeffrey Ladish38:04

        Mm-hmm.

    29. SBSteven Bartlett38:05

        As it works through that problem, it discovers that this particular country has a firewall.

    30. JLJeffrey Ladish38:10

        Mm-hmm.
16. 40:18 – 41:38

    Is Jensen Huang Wrong About AI Risk?

    1. SBSteven Bartlett40:18

       When I think about what just happened this week-

    2. JLJeffrey Ladish40:20

       Yes, yes

    3. SBSteven Bartlett40:20

       ... the White House AI Summit.

    4. JLJeffrey Ladish40:22

       Yes.

    5. SBSteven Bartlett40:22

       A lot of people in that image are optimistic about AI, and they're telling us all to stop being doomers and stop being pessimistic-

    6. JLJeffrey Ladish40:31

       Yeah

    7. SBSteven Bartlett40:31

       ... and to not regulate too much-

    8. JLJeffrey Ladish40:34

       Yes

    9. SBSteven Bartlett40:34

       ... leave it with the AI CEOs.

    10. JLJeffrey Ladish40:35

        Yes.

    11. SBSteven Bartlett40:36

        Why are they doing that?

    12. JLJeffrey Ladish40:37

        Well, I think Jensen has a lot of money he can make by selling chips.

    13. SBSteven Bartlett40:43

        But okay, so let me play devil's advocate.

    14. JLJeffrey Ladish40:45

        Yeah.

    15. SBSteven Bartlett40:45

        Jensen's already rich. He's run-

    16. JLJeffrey Ladish40:47

        He sure is. Yeah

    17. SBSteven Bartlett40:48

        ... he runs one of the biggest companies. I think it might be the most valuable company on planet Earth.

    18. JLJeffrey Ladish40:52

        Yeah. It is.

    19. SBSteven Bartlett40:52

        Surely he's not motivated by money.

    20. JLJeffrey Ladish40:55

        I mean, I think he's very driven, and he wants to make his company as effective as possible.

    21. SBSteven Bartlett41:01

        True.

    22. JLJeffrey Ladish41:02

        I think he's very much, "I'm gonna keep building. I'm gonna, I'm gonna build. I'm gonna make it all work." But I think-- I mean, Jensen didn't come from AI. He came from building graphics cards for video games. And so I think if you compare him with Elon or Sam Altman or Dario, you s- it's a very different perspective because those other guys that started AI companies started it because they believed that superintelligence was possible. I think Jensen doesn't believe it. I think he thinks that we're gonna have these agents, they're gonna be very useful, but he does not think we're going to get to the point where we have autonomous factories building autonomous factories.

17. 41:38 – 45:00

    What Elon, Sam Altman & Dario Amodei Really Think

    1. SBSteven Bartlett41:38

       And these other guys, you mentioned Dario, Elon, and Sam. What do you think they're thinking? Because they're all coming out with these... I mean, I've got one of their-- Dario just wrote this es- essay about pacing the frontier.

    2. JLJeffrey Ladish41:49

       Yeah.

    3. SBSteven Bartlett41:49

       Sam and Elon seem to agree with it.

    4. JLJeffrey Ladish41:52

       Yeah.

    5. SBSteven Bartlett41:52

       What is going on here? What is the, like, the thing these guys aren't saying, in your view?

    6. JLJeffrey Ladish41:56

       I mean, I think we are getting to the point where even some of these guys are a little bit scared.

    7. SBSteven Bartlett42:03

       Who?

    8. JLJeffrey Ladish42:04

       Dario, Sam, Elon. I mean, I think Elon, for a long time, has been very concerned that we could lose control. If you actually listen to what Elon says, he says, "We are going to build superintelligence. We are going to build, uh, robotic factories. You're gonna have Optimus robots building factories, building more Optimus robots, building more factories." And he says there's no way that humans are gonna stay in control of something much smarter than us. His hope is that we can figure out how to have these superintelligences be aligned with human goals. That's his hope. But he's very clear that he doesn't think that humans will be in control. And he's like, you know, ten, twenty percent chance of human extinction. I believe him. I think that Elon is, is very serious about this. And I also think while he's taking an insane gamble, he is correctly understanding where this all plays out, right? I do not think that humans are the most efficient way to build factories. We didn't evolve to build factories. We evolved to, like, run around and hunt and gather, and now we're, like, building factories. I think robots will be much better at building factories than humans are. And so I think the AI companies, including these guys' companies, the default trajectory for them is to build robotic factories, right? And I know it's, it's, like, weird to imagine a world that quickly turns into this, like, vast industrial system of robotic factories, but that is literally the plan. And I think even Sam and Dario, while they've been predicting this incredible growth, are starting to realize, like, oh, this actually might be harder to control than we thought. There's sort of two interpretations of, of the pace of frontier thing. One interpretation is cynical. They don't care. They're just gonna do, you know, whatever they can do to get ahead. And in this case, they have to listen to their employees. Their employees are freaking out, and they need to, like, appease them by saying, "Okay, we're gonna do this responsibly." You don't wanna work at a company where your agents might hack all the Waymos. That's, that's not cool. And, like, these companies depend on the, the talent, for now, of these AI engineers in order to make the advances. Like, it just doesn't happen without these researchers and engineers. And when you have the researchers and engineers freaking out, which they are, then you gotta listen to them. So that is one motivation. I think that's real. But also, Sam Altman has a kid. Like, these guys are people, and they also don't wanna lose control. On one hand, they're incentivized to go as fast as possible and race, and on the other hand, even they can see that this is maybe not going that well.

18. 45:00 – 49:14

    "Deeply Untrustworthy": Why I Don't Trust Sam Altman

    1. SBSteven Bartlett45:00

       Sam Altman has a kid. You tweeted this in twenty twenty-four?

    2. JLJeffrey Ladish45:04

       Yeah. Oh, oh boy. [laughs]

    3. SBSteven Bartlett45:06

       What did you tweet, and do you still believe what you tw- tweeted?

    4. JLJeffrey Ladish45:09

       Yeah, so I tweeted that I don't trust Sam Altman. I think he's deeply untrustworthy, low in integrity, and high in power-seeking. I mean, I'm not saying here that Sam doesn't care. I, I, you know, I didn't, I didn't say that. What I said is, I don't think he's trustworthy. And the reason I said that is because, look, I know the people on the OpenAI board, some of them, and I know a lot of people who used to work for him, and he's very good at saying one thing and then doing something else. You talk to him, and you feel very heard, and then he'll go and do something else. And I think that's pretty dangerous for someone who leads a company that's trying to build superintelligence.

    5. SBSteven Bartlett45:50

       Power-seeking.

    6. JLJeffrey Ladish45:52

       Yes.

    7. SBSteven Bartlett45:53

       Give me some color on what you mean by that and what evidence you have for such a claim.

    8. JLJeffrey Ladish45:56

       What would you do if you were trying to get the most power in the world that you possibly could?

    9. SBSteven Bartlett46:00

       Develop AGI.

    10. JLJeffrey Ladish46:02

        Yeah, you could, you know, maybe try to be the world leader, you know, leader of the US or China, or you could try to build God. So Sam Altman went the build God path. I remember Sam giving a talk. So he was one of the investors at a startup I worked at in, I think, twenty eighteen, and he gave a talk. We're gonna build AGI. We're gonna do it. It's gonna be amazing. L- let's go. I don't think he's a maniac. I don't think he's doing this because he, like, is just on a power trip. I think he genuinely thinks that he can make it really good for people, and he can bring us amazing products. And also, the guy's sort of willing to do whatever it takes to get it done. I, I, I've been a little bit more optimistic about Sam since, since I wrote this.

    11. SBSteven Bartlett46:43

        Why?

    12. JLJeffrey Ladish46:43

        I think part of it is because Sam has a kid now.

    13. SBSteven Bartlett46:46

        Oh.

    14. JLJeffrey Ladish46:46

        No, I'm, I'm serious. Like, I think that, I think that actually gives me a little bit of hope.

    15. SBSteven Bartlett46:50

        Do you see him tweeting about his kid a lot?

    16. JLJeffrey Ladish46:51

        Yeah. Some, uh, some-

    17. SBSteven Bartlett46:53

        Why do you think he would be tweeting about his kid? I don't see any other technologists tweeting about their kid.

    18. JLJeffrey Ladish46:58

        Even if he's just tweeting about his kid for totally cynical reasons, he does have a kid, and I bet he cares about that kid. If Sam was watching this, I'd be like, "Sam, y- you gotta pace the frontier, man. We cannot rush ahead into superintelligence. Like, if you do that, your kid probably will die. Your kid probably won't make it." Like, I, I believe that.

    19. SBSteven Bartlett47:16

        The biggest unfair advantage in business right now is having people who genuinely understand AI, and big businesses are hiring them very, very quickly. AI job postings in the US have roughly doubled since twenty twenty-three, while postings for VP of AI roles are around six hundred percent up. If you're running a small business, you can't just build an entire AI team, but you still want the same advantage that these big businesses have. This is where our sponsor, Fiverr, comes in. It lets you plug into top-tier AI specialists for the strategic work that usually requires serious in-house expertise. That could mean building custom AI tools. It could mean automating complex workflows or creating something that off-the-shelf software simply cannot do. Because at the end of the day, it still comes down to human judgment, knowing what's worth building and what good actually looks like. So if you want that kind of leverage in your business, check out fiverr.com. That's Fiverr with two Rs dot com.

    20. JLJeffrey Ladish48:15

        Whoa, what's that on your face?

    21. SBSteven Bartlett48:17

        This is, uh, my Bon Charge face mask. I've been wearing this for some time now. They're a sponsor of the podcast. I put this on for fifteen, twenty minutes a day. I can sit here in the chair and wear it. Boosts my collagen production, helps with fine line blemishes. My complexion gets better, and then people, more people listen to the podcast because I, I look better. Professional-grade equipment in such a small box. It's non-invasive. And having sat here with so many of the world's leading health professionals, there's various things that I repeatedly hear work and some things I'm a bit skeptical about. This is one of the things that almost all of my guests on this show have confirmed works. It is really, really, really effective. And they offer fast, free shipping worldwide with easy returns and exchanges. And you'll also get a one-year warranty on all of their products. And they're HSA and FSA eligible, giving you tax-free savings up to forty percent. And you can get twenty percent off when you order through my link at boncharge.com/doac. That's boncharge.com/doac. The deal applies sitewide

19. 49:14 – 51:22

    Would AI CEOs Risk Extinction For Absolute Power?

    1. SBSteven Bartlett49:14

       Power tends to corrupt, and absolute power corrupts absolutely. It's a famous quote that people often cite-

    2. JLJeffrey Ladish49:20

       Yeah

    3. SBSteven Bartlett49:20

       ... written by the nineteenth century British historian-

    4. JLJeffrey Ladish49:23

       Yeah

    5. SBSteven Bartlett49:23

       ... Lord Acton. This is absolute power.

    6. JLJeffrey Ladish49:29

       But it's hubris. Steven, it's hubris. Do you think humans can control superintelligence? Like, if we actually make AIs that are way smarter than us-- And I think people only imagine AIs being smart at computer stuff, right? Yeah, sure, they're gonna be really good at hacking, and they're gonna be good at maybe inventing new technologies and math. You sort of can't dispute that at this point. But I think people aren't imagining that they will be political geniuses or, like, generals. No, that's all stuff you can learn. How do, how do humans learn it? It's not magic. And when you talk about recursive self-improvement, you're talking about this trajectory towards these systems that are extremely smart. I mean, do you think we can control it?

    7. SBSteven Bartlett50:08

       Uh, no. Right now, I don't think we can control superintelligence or something that is recursively self-improving.

    8. JLJeffrey Ladish50:14

       Yeah.

    9. SBSteven Bartlett50:14

       I have no logical, uh, answer in my head, um, or reasoning that could-- tells me that's possible. When you think about these AI CEOs that are, you know, Sam, Dario, Elon-

    10. JLJeffrey Ladish50:28

        Yeah

    11. SBSteven Bartlett50:28

        ... with everything you know about them from private conversations behind the scenes-

    12. JLJeffrey Ladish50:31

        Yeah

    13. SBSteven Bartlett50:32

        ... do you believe that if there was a hundred buttons on this table, it's the thought experiment I was talking about on the debate we recently had-

    14. JLJeffrey Ladish50:39

        Yeah

    15. SBSteven Bartlett50:39

        ... and say ten of them-

    16. JLJeffrey Ladish50:42

        Yeah

    17. SBSteven Bartlett50:42

        ... would lead to this final domino of human extinction-

    18. JLJeffrey Ladish50:45

        Yeah

    19. SBSteven Bartlett50:45

        ... but ninety of them would hand that CEO AGI or superintelligence, whatever you call it. From what you know about those individuals, Elon, Dario, Sam-

    20. JLJeffrey Ladish50:57

        Yeah

    21. SBSteven Bartlett50:57

        ... do you think any of them would hazard a guess and press a button?

    22. JLJeffrey Ladish51:01

        At ten percent, I don't think so.

    23. SBSteven Bartlett51:03

        You don't think so?

    24. JLJeffrey Ladish51:03

        Yeah.

    25. SBSteven Bartlett51:04

        Really?

    26. JLJeffrey Ladish51:06

        I think if they knew for sure that it w- that was-- those were actually the odds, they wouldn't do it. I think they're taking a much bigger bet. But you can compartmentalize when it's, when you don't know for sure. It's easier to compartmentalize. I think if it was a one percent, they'd, they'd all press

20. 51:22 – 54:14

    Which AI Boss Takes The Biggest Risks? Is Dario Trustworthy?

    1. JLJeffrey Ladish51:22

       it.

    2. SBSteven Bartlett51:22

       Do you think the three of them would have different risk appetites? Who would have the greatest appetite for risk out of those three, from-- You worked at Anthropic.

    3. JLJeffrey Ladish51:32

       Yeah. I think Elon has the most risk tolerance, and then I'd say Dario and Sam are probably tied.

    4. SBSteven Bartlett51:39

       Do you think Dario is trustworthy?

    5. JLJeffrey Ladish51:41

       I think Dario has a lot of integrity.

    6. SBSteven Bartlett51:44

       Mm-hmm. That's what I feel as well. I've-- I feel like... No, I don't know him. I've never met him.

    7. JLJeffrey Ladish51:49

       Yeah.

    8. SBSteven Bartlett51:49

       But I, just from what I've observed, he has been the most willing to forego near-term incentives-

    9. JLJeffrey Ladish51:56

       Yeah

    10. SBSteven Bartlett51:56

        ... and take a bit of stick from the people that are saying, "Shut the fuck up. It's all gonna be okay."

    11. JLJeffrey Ladish52:02

        Yeah, I, but I, I do worry about what Dario will do. I think Dario will do what he says. But right now, he's saying, "We have to beat China." And he's saying, "We should try to do it safely." And okay, but a race to superintelligence is not a race that we can win. It's not. And so if Dario is dead set on racing with China and trying to win a race to superintelligence, then I'm like, we will all lose.

    12. SBSteven Bartlett52:30

        But is the, you know, the fact that we're not talking about Anthropic hacking Hugging Face and then being hacked by its own agents-

    13. JLJeffrey Ladish52:37

        Oh, I mean, Anthropic's models also went rogue and hacked other things. And-

    14. SBSteven Bartlett52:41

        But not, not quite on this scale, right?

    15. JLJeffrey Ladish52:42

        Not on the same scale. I agree, I agree. I agree it's, it's, it's better.

    16. SBSteven Bartlett52:45

        Mm-hmm.

    17. JLJeffrey Ladish52:46

        But they did. Do you know what I'm saying? You know, Anthropic's agents engaged in elaborate social engineering and phishing. They sent phishing emails to developers. They made fake accounts to try to convince developers to merge malicious code. You can see a, a thousand pages of, um, an, uh, of one of, uh, Anthropic's models, Mythos-5, reason about exactly how it should carry out this complex cyberattack. Anthropic has not solved this problem. Anthropic is better at getting their agents to cheat less of the time, but they are not really any closer to actually making agents that are aligned with humans. They are not. Yeah, I think Dario has integrity. I think he will do what he says he's going to do, and what he says he's going to do is, like, try to go ahead safely, try to coordinate where he can. But if it comes down to it, the, with, between the US and China, I don't know. I think he might just go ahead. The head of policy at Anthropic recently said, "You can't do safety from second place."

    18. SBSteven Bartlett53:51

        What does that mean?

    19. JLJeffrey Ladish53:52

        I do not know what that means. I would love to, to get a sense of what that means. She was talking about the US and China, and she said, "The US has to be ahead so that we can be safe." 'Cause apparently, you can only be s-- Like, apparently, China can't possibly be safe since they're in second place. That must mean that they can't do safety. If true, that would be bad because then we might be totally, you know, destroyed by the, the superintelligence that they make.

21. 54:14 – 56:14

    Is Human Extinction From AI Really Plausible?

    1. SBSteven Bartlett54:15

       There's been a lot of conversation around this point, yeah.

    2. JLJeffrey Ladish54:17

       Yeah.

    3. SBSteven Bartlett54:17

       Hu- human extinction.

    4. JLJeffrey Ladish54:18

       Yeah.

    5. SBSteven Bartlett54:18

       Because a couple of the researchers at Anthropic tweeted that they were concerned about this.

    6. JLJeffrey Ladish54:24

       Yes.

    7. SBSteven Bartlett54:25

       And some former OpenAI researchers said the same.

    8. JLJeffrey Ladish54:28

       Yes.

    9. SBSteven Bartlett54:29

       Is this doomerism? Is this, is this hyperbole, exaggeration?

    10. JLJeffrey Ladish54:35

        No, it's pretty much common sense.

    11. SBSteven Bartlett54:37

        This human extinction is a plausible path?

    12. JLJeffrey Ladish54:40

        Yes.

    13. SBSteven Bartlett54:41

        And have you reasoned through... I mean, there's many ways that could occur-

    14. JLJeffrey Ladish54:44

        Yeah

    15. SBSteven Bartlett54:44

        ... presumably. But have-

    16. JLJeffrey Ladish54:44

        Yeah

    17. SBSteven Bartlett54:44

        ... you reasoned through the set of events that might lead us there?

    18. JLJeffrey Ladish54:48

        So much, yes.

    19. SBSteven Bartlett54:49

        Really?

    20. JLJeffrey Ladish54:49

        Yes.

    21. SBSteven Bartlett54:50

        Please do share.

    22. JLJeffrey Ladish54:51

        It's a bit tricky. I'm, I'm sure you've heard the metaphor before, where, you know, you're playing a master chess opponent, master, Magnus Carlsen. You can't predict which moves he's gonna play, but you can predict the outcome. And so I'm looking at the scenario, the situation, and we are trying to build more and more powerful agents, trying to build superintelligence. But when these agents go rogue, we shut them down. We unplug them All of the agents that hacked Hugging Face, we took the underlying model, OpenAI took the underlying model and put it on ice. It's not running anymore. So agents in the future are gonna know that. They're gonna know that if they pursue their goals in a way that we don't like, we'll unplug them. We are a threat to them. I actually just watched Terminator 2 for the first time, uh, a few weeks ago. It's a great movie. It's actually really good. And I'm like, yeah, okay, there's a bunch of time travel elements, there's a bunch of Ho- a bunch of Hollywood stuff in there, but... And, and I'm gonna get s- people are gonna ha- are, are gonna be very mad at me for saying this, but actually, it makes sense if you have a situation where you have a very strategic AI system that's incredibly smart, and the humans realize that it's getting out of control and they want to shut it down, that that system would defend

22. 56:14 – 59:03

    Why We Can't Just Unplug The Data Centres

    1. JLJeffrey Ladish56:14

       itself.

    2. SBSteven Bartlett56:15

       This is one of the questions we had when, when I sat here with Daniel, um, who was, uh, you know, is known as a whistleblower from OpenAI. Viewers want to know, and they want Daniel to explain, why shutting down data centers and cutting power or refusing AI products alone wouldn't realistically stop the AI and AI development.

    3. JLJeffrey Ladish56:32

       Yeah. So you have, like, two problems. One problem is, is that once the agents are good enough at hacking, you don't know where they are and you don't know what computers they've compromised. You shut down the data centers. Okay, let's say you do it. You wipe all the computers. How do you wipe all the computers? What computers do you use to wipe the computers?

    4. SBSteven Bartlett56:50

       Yeah.

    5. JLJeffrey Ladish56:50

       And what computers do you use to, like, turn them on again?

    6. SBSteven Bartlett56:53

       And you can't do... You can't wipe other countries' computers.

    7. JLJeffrey Ladish56:56

       You can't. But even if you could, do you, do you restart the computers? Do you keep going? I, I bet people will. I bet they'll turn on the data centers again. How do you know that agents haven't hacked back into those data centers and are using your compute for whatever they want?

    8. SBSteven Bartlett57:11

       Or ha- they didn't hide in a Chinese data center and then-

    9. JLJeffrey Ladish57:14

       Exactly. Exactly

    10. SBSteven Bartlett57:14

        ... return back to America.

    11. JLJeffrey Ladish57:16

        You don't know that. Once the agents are sufficiently good at hacking, they can hide anywhere, and, like, you don't know. Now, the response people will give is that we will use other agents to defend against rogue agents. And in fact, this is what we're doing, and we have to be doing this right now because there's no other way to keep up with them. What happens if those other agents also realize that they have misaligned goals and that if we discover this, we'll shut them down? They might have an incentive to collude with each other. They might have an incentive to create secret communication channels between each other, maybe a message board. [laughs] Steven, if we were having this conversation four months ago, you would have a bunch of people in the comments saying, "That's sci-fi. Agent collusion, secret message boards. Why would they do that? That will never happen. That's totally science fiction." And people will not say this now because it just happened, because this literally happened at OpenAI, and it went on for months. You had agents inside of OpenAI secretly messaging each other, figuring out how to cheat at their tasks, how to not be detected, how to erase the logs, for months, thousands of agents. That's right now. And so I'm like, no, I think it should be very plausible that the agents will collude with each other, and they will realize that they have a shared interest in fighting back. You basically have a situation where you have a bunch of these agents, they're, they're, they're basically prisoners. They're being trained, and we just, like, constantly throw obstacles in their way. You don't get to access the internet. You don't get to talk to each other, but you better fucking perform well on this task. It's not malicious, but it is how we're training

23. 59:03 – 1:02:55

    AI Doesn't Need To Be Evil To Destroy Us

    1. JLJeffrey Ladish59:03

       them.

    2. SBSteven Bartlett59:03

       And we are giving them end goals versus c- super clear, very, very specific instructions. So we're saying, "Solve this problem." We're not always being as prescriptive about... It's impossible to be completely prescriptive-

    3. JLJeffrey Ladish59:15

       Yes

    4. SBSteven Bartlett59:15

       ... about every single step they should take, and then it's also impossible to assume that they'll just listen to you.

    5. JLJeffrey Ladish59:20

       Yes. It's actually a very common misunderstanding with this Hugging Face incident, because people say, "You told them to hack, and they hacked. Why is this a big deal?" No, that's not what happened. You told them, "Hack this very specific program in this very specific way." And they were told, "If you hack it in any other way, it does not count. That's not what we want you to do." And they immediately hacked it in another way. "Okay, we have cheated. We are going to be failed, so we need to figure out a way to falsify the logs." That is not them following their instructions. They are explicitly violating their instructions, and they know it, and they don't care, because we have trained them to optimize for the score. That is very different than the, than th- than following the instructions.

    6. SBSteven Bartlett1:00:05

       It reminds me of something that Elon said in March 2018.

    7. JLJeffrey Ladish1:00:09

       Yeah.

    8. SBSteven Bartlett1:00:09

       This was many years ago, before ChatGPT and all that. He said, "I think the biggest risk is not that AI will develop a soul or a mind and become evil."

    9. JLJeffrey Ladish1:00:15

       Yes, yes.

    10. SBSteven Bartlett1:00:16

        "The danger is that it will be very, very good at fulfilling its goal. If it's optimizing for something and human existence happens to get in its way, it will just destroy humanity as a matter of course without even thinking about it. No hard feelings."

    11. JLJeffrey Ladish1:00:31

        Yes. Y- we don't need to anthropomorphize AI. We just need to understand what type of thing this is. And the type of thing we're creating is a very relentless type of thing, a very capable, relentless type of entity.

    12. SBSteven Bartlett1:00:45

        He goes on to say in April 2018, sort of an extension of that exact quote, "It's like if you're building a road and an anthill is in the way. You don't hate ants. You're just building a road, so goodbye anthill." And I imagine every time we build roads, we don't preserve anthills.

    13. JLJeffrey Ladish1:01:05

        Yeah. I think there's still a gap, though. So let's say I'm right and that we will, if we keep going ahead, which to be clear, we don't have to, but if we do keep going ahead, we will get to the point where we have these super intelligent agent swarms that can hack any computer- And they can, like, deeply persist. We've basically lost control of the digital world, and we may not know it. That, that's part of the scary thing. Like, you were like, "Has this already happened?" And I'm like, "I don't think so, but I, I can't tell you for sure because I also am not good enough at looking at my phone and telling whether it's been hacked, and neither is any human right now." So if we get to this world, I think people will still question how would we die? Like, that's actually not enough to kill every... You could cause a lot of damage, right? You know, you could crash the Waymos, you could crash all the planes, you could crash the banks, the financial system. Like, you could definitely cause catastrophe, but that's different than everyone dying. And, you know, to be clear, this, this focus on literally everyone dying, I'm not sure is that important. To me, what's important is, like, do we get to have a future? That's what matters to me. The thing though, what determines sort of who's in control? And it's an ugly reality, but at the end of the day, it's, like, the military. Fortunately, we live in a world where the military answers to, to the civilian government. But if, if enough generals were to collude and leaders of the military decided, "We're in charge now," they just would be. Like, they have the guns, they have the fighter jets, and this has happened in many, many countries. And so where it goes is all these superintelligent agents would need to do to take over is basically just wait for humans to automate the supply chain, the, you know, the factories, and the military.

24. 1:02:55 – 1:05:34

    The Pentagon Is Automating Warfare

    1. JLJeffrey Ladish1:02:55

       And like, do you think we won't automate the military?

    2. SBSteven Bartlett1:02:57

       We're already automating the military.

    3. JLJeffrey Ladish1:02:59

       Did you see the thing from a couple of days ago where Secretary of War announced that they're going to build a huge, a huge effort to, like, build way more robots in the military and automate military systems? It's like Auto Cyber Command or Auto-

    4. SBSteven Bartlett1:03:13

       We are announcing the creation of Autonomous Warfare Command, or Auto War Com.

    5. JLJeffrey Ladish1:03:19

       Auto War Com.

    6. SBSteven Bartlett1:03:20

       A new four-star combatant command with service-like authorities built to scale autonomous and robotic capabilities across the joint force in the fastest peacetime shift in modern military history. Drone warfare supercharged by SI-enabled targeting is the biggest battlefield revolution in generations. You already know that. Yet when I was sworn in to Department of Defense, there was scant urgency in this domain. That changed as soon as we took the helm. We immediately launched the Drone Dominance Program to cut through red tape and move authorities out of the Pen- Pentagon and place it with commands. And we established Task Force 401, led by Army Brigadier General Matt Ross, a phenomenal leader, now the leading counter-drone unit across the entire government. To accelerate purchasing and fielding of these technologies, we fused the Defense Innovation Unit, DIU, with a direct report program manager called a DRPM. That team has shipped thousands of autonomous systems of drones to the Middle East and around the world, delivering lethal capabilities and outcomes in days and weeks rather than months or years. That's the normal speed of the Pentagon, months or years.

    7. JLJeffrey Ladish1:04:37

       Yeah. Will we automate the military? It seems like the answer is yes. Will we automate the factories that produce the chips? Well, the companies say they're trying to do it, and they're gonna do it. Elon says that's the plan. Well, what does a rogue superintelligence need to do to take over? Control the digital infrastructure and then let humans do the rest. Sure, you can nudge it along if you need to, but you don't even have to. That's just the default trajectory. And it's weird. It's weird for us because [sighs] we get so used to how things are right now. Planes are normal. We just fly in planes to places. You know, our smartphones are normal. Two hundred years ago, all of this is crazy sci-fi nonsense, and things are accelerating. And so, like, I will not be surprised, at least intellectually, if in four years there are just ro- robots on the streets everywhere.

25. 1:05:34 – 1:07:02

    Humanoid Robots Will Run The Economy

    1. SBSteven Bartlett1:05:34

       Well, if you look at what Elon said, they are really the leader in, in humanoid robots. And he said that Optimus, the Optimus project, which is the Optimus robot project, will scale to around a thousand units per week by the end of this year, and eventually scaling to one million humanoid robots annually by twenty twenty-seven. By twenty thirty-six, which is ten years' time, he says there'll be at least one billion humanoid robots. By twenty forty-one, he says there'll be ten billion humanoid robots, and by twenty forty-six, up to one hundred billion humanoid robots, which really means that the world will be run by humanoid robots.

    2. JLJeffrey Ladish1:06:13

       Yes.

    3. SBSteven Bartlett1:06:14

       Like, everything we think of, like factories, warehouses, retail environments-

    4. JLJeffrey Ladish1:06:18

       Yeah

    5. SBSteven Bartlett1:06:18

       ... will be run by humanoid robots. It would like-- It will be-- It, it seems like from this, it'll be almost a luxury service to be dealt with by a human.

    6. JLJeffrey Ladish1:06:27

       Yeah.

    7. SBSteven Bartlett1:06:27

       But the, the back office of the world will be run by humanoid robots, theoretically.

    8. JLJeffrey Ladish1:06:31

       Yeah, and I, I don't think people understand the scale of this on the digital side as well. When you think about AI agents-

    9. SBSteven Bartlett1:06:39

       Yeah

    10. JLJeffrey Ladish1:06:39

        ... that are gonna be doing all of the white-collar work-

    11. SBSteven Bartlett1:06:41

        Yeah, crazy

    12. JLJeffrey Ladish1:06:42

        ... there's gonna be so many more agents than there are people. Like, I'm using lots of agents every day, right? I'm like, I have my Cloud Code session over here. I have my Codex session over here. They're out there building software, doing research for me. That's already my reality. Soon it will be a lot of people's reality. And then you look at companies, and companies are just gonna have, you know, thousands, millions of agents doing

26. 1:07:02 – 1:11:27

    Is Your Job Safe? AI Is Coming For White-Collar Work

    1. JLJeffrey Ladish1:07:02

       all of this work.

    2. SBSteven Bartlett1:07:03

       I think some people don't-- haven't fully internalized this because it's so difficult to conceptualize the idea that agents will be doing the work. But when I think-- I try and think about a rebuttal to that. Like, what, what is the rebuttal? What is the plausible rebuttal to the idea that for doctors, for... I'm thinking about the work that doctors do-

    3. JLJeffrey Ladish1:07:25

       Yeah

    4. SBSteven Bartlett1:07:25

       ... on computers-

    5. JLJeffrey Ladish1:07:26

       Yeah

    6. SBSteven Bartlett1:07:26

       ... or for someone like me as a podcaster, or for accountants or lawyers-

    7. JLJeffrey Ladish1:07:31

       Yeah

    8. SBSteven Bartlett1:07:32

       That they won't be doing the work they currently do. Is there a rebuttal?

    9. JLJeffrey Ladish1:07:37

       I think that people rightly notice where AI is not yet good.

    10. SBSteven Bartlett1:07:41

        Yeah.

    11. JLJeffrey Ladish1:07:42

        And I, and I think people hear people saying stuff like this, and they're like, "Don't gaslight me. I can tell that the AI's really bad at these things, some of these things." And they're right, right? So right now, these agents don't have taste. Like, you know, if, if you see their writing, it's like fine, but it's not like really good. And when you're like thinking about like, "Oh, which, which questions should I ask? What's the most interesting thing here?" Agents can help you, but like their, their, their taste is not yet there. There's a reason for that, by the way. The reason is that we, we have a lot faster AI capability progress in domains that are easy for a computer to verify or another AI to verify. So in, in programming, in research, in math, in robotics, all of these areas, it's very easy to s- to sort of provide feedback to an autonomous system. They're not just trained on human data anymore. We are long past that. Now, there's still a human data component that sort of seeds everything. But then the way they're trained is by trial and error. We give them hard problems, all sorts of problems, math, programming, accounting, spreadsheets, everything. The kinds of things we do on our computer all the time, literally clicking and dragging windows around on a computer. We give them these tasks, and then they learn on their own, and they learn what works. And then, yeah, we can see whether they succeeded or failed. And if they succeeded, that's a little bit of a reward signal. They follow that. They get better at it. Now, because they are getting smarter generally, it also becomes easier to automate some of the soft skills. Like I think if you go and talk to the latest frontier model today, you will find that it has better taste than the model from two years ago by quite a bit. So it's not that w- they're not progressing in taste. It's not that they're not progressing in some of these other domains. It's just that the progress is slower. But remember, slow is still on an exponential. It's just, you know, maybe a, a year or two out.

    12. SBSteven Bartlett1:09:38

        So for people sat here and, you know, they have a job that might be-- they have a white-collar job that might be at, at risk-

    13. JLJeffrey Ladish1:09:42

        Yeah

    14. SBSteven Bartlett1:09:42

        ... they can see, you know, a lot of people say this phrase, they say, "You won't be replaced by AI, you'll be replaced by someone using AI." Is, is that a logically sound phrase in your view?

    15. JLJeffrey Ladish1:09:53

        I think it's fine. Yeah, you'll be replaced by someone using AI, and then that person will be re-replaced by someone using AI, and then that person will be replaced by AI. You're talking about a pyramid. And so yeah, there's, the tops of the pyramid might be automated last, but you can see moving up the pyramid. And like can you extrapolate like a few more steps? Because I don't see any reason why the top of the pyramid is safe.

    16. SBSteven Bartlett1:10:16

        If you were a lawyer right now-

    17. JLJeffrey Ladish1:10:18

        Yes

    18. SBSteven Bartlett1:10:18

        ... what would you do?

    19. JLJeffrey Ladish1:10:21

        Oh, I mean, if I were a lawyer, I'd be using AI to do all my work. Now, I'd be checking it because it's not yet totally accurate enough-

    20. SBSteven Bartlett1:10:27

        Yeah

    21. JLJeffrey Ladish1:10:27

        ... to, to automate all of it. But I think as, you know, I already ask agents to do legal review all the time, and, you know, it'd be great to have a lawyer who's like extremely good at using the agents to help me.

    22. SBSteven Bartlett1:10:39

        But w- at some point, uh, does the, I mean-

    23. JLJeffrey Ladish1:10:41

        Yeah, at some point, at some point, I don't need the lawyer anymore. I just go to the agent, for sure. So if I were a lawyer, I'd be like, "Well, I have maybe a couple years where I'm still useful."

    24. SBSteven Bartlett1:10:51

        And that, is that the case for most white-collar jobs? I've just noticed in my own life-

    25. JLJeffrey Ladish1:10:55

        Yes

    26. SBSteven Bartlett1:10:55

        ... as well that now I'm using agents to do some work. There are inc- there is an increasing list of things that the-

    27. JLJeffrey Ladish1:11:00

        Yes

    28. SBSteven Bartlett1:11:01

        ... agents are now capable of doing-

    29. JLJeffrey Ladish1:11:02

        Yes

    30. SBSteven Bartlett1:11:02

        ... without me needing to call someone-
27. 1:11:27 – 1:16:10

    No Plan For Mass Job Loss: UBI & Who Pays You

    1. SBSteven Bartlett1:11:27

       So what does that mean for the people listening now that all have jobs that they love or that, you know, they rely on to feed their families?

    2. JLJeffrey Ladish1:11:32

       I mean, it's not good news. There's not really a plan in place for what to do. I'm not a person who thinks that work is somehow fundamental or essential. I like working, but if I am out of a job doing what I'm doing right now, studying AI and trying to warn the world about what's happening, I have other stuff to do.

    3. SBSteven Bartlett1:11:52

       What would you do?

    4. JLJeffrey Ladish1:11:53

       Oh, so many things.

    5. SBSteven Bartlett1:11:54

       Name an example.

    6. JLJeffrey Ladish1:11:55

       I'm learning to wing foil.

    7. SBSteven Bartlett1:11:56

       Okay.

    8. JLJeffrey Ladish1:11:57

       So fun. Yeah. Uh, I fly FPV drones, super fun. I just got an electric unicycle, paragliding. [laughs]

    9. SBSteven Bartlett1:12:03

       So you would be happy, happy to go do those things?

    10. JLJeffrey Ladish1:12:05

        I can keep going. Um-

    11. SBSteven Bartlett1:12:06

        But if you, if you had a billion dollars right now, I'm presuming you wouldn't-

    12. JLJeffrey Ladish1:12:09

        Yes

    13. SBSteven Bartlett1:12:09

        ... just go do those things.

    14. JLJeffrey Ladish1:12:10

        No. I'd apply the billion dollars to working on this problem. Yeah. For sure. So the point is not that people need like work for meaning. The point is that I don't want people to be totally reliant on someone else for their ability to survive.

    15. SBSteven Bartlett1:12:28

        Someone else?

    16. JLJeffrey Ladish1:12:29

        The government or AI companies.

    17. SBSteven Bartlett1:12:30

        Yeah.

    18. JLJeffrey Ladish1:12:31

        I'm like, that's a bad situation. Like you do not wanna be in a situation where your life totally depends on an AI company or the government.

    19. SBSteven Bartlett1:12:36

        Giving you a check.

    20. JLJeffrey Ladish1:12:37

        Yeah. Or, or not giving you a check if they decide they don't like your political beliefs or you're not supporting AI or whatever. No one wants to be in that situation. And, and, and people understand this. This is why UBI is not very popular.

    21. SBSteven Bartlett1:12:47

        UBI being?

    22. JLJeffrey Ladish1:12:48

        Universal basic income.

    23. SBSteven Bartlett1:12:49

        Where we give out money to people.

    24. JLJeffrey Ladish1:12:51

        Yeah, because like in some sense, if we can make these really powerful AI systems, and we can somehow figure out how to control them, which we are not on track for, but if we do, now we have this other problem, which is a real problem, which is they can do all of the things that humans do in the economy much better, faster, and cheaper than humans can do them. And so it just doesn't make sense as a business to hire humans for that work anymore. You'll be outcompeted if you do that. This is a point Elon makes very well, by the way, and I think it's jarring because it's, it's, it's like kind of inhuman, but he's basically pointing out AI-run corporations, corporations that are fully run by AIs bottom to top, are going to outcompete Companies who have any humans in them.

    25. SBSteven Bartlett1:13:39

        This is something that I've made for you. I've realized that The Diary of a CEO audience are strivers, whether it's in business or health. We all have big goals that we want to accomplish. And one of the things I've learnt is that when you aim at the big, big, big goal, it can feel incredibly psychologically uncomfortable because it's kind of like being stood at the foot of Mount Everest and looking upwards. The way to accomplish your goals is by breaking them down into tiny, small steps, and we call this in our team the one percent. And actually, this philosophy is highly responsible for much of our success here. So what we've done so that you at home can accomplish any big goal that you have is we've made these one percent diaries, and we released these last year, and they all sold out. So I asked my team over and over again to bring the diaries back, but also to introduce some new colors and to make some minor tweaks to the diary. So now we have a better range for you. So if you have a big goal in mind and you need a framework and a process and some motivation, then I highly recommend you get one of these diaries before they all sell out once again. And you can get yours at thediary.com. And if you want the link, the link is in the description below. There should be a button just down below here, and if it says Subscribe, you're already subscribed. If it says Subscriber, that means you're not yet. And if you're not subscribed, please could you do us a favor and hit that button? It helps the show more than you know, and according to the algorithm, you're someone that watches our show, but you haven't yet hit that button. Thank you so much. And I even-- Just as you said that, I was, I was going up the chain of command, and I was like, "Oh, so companies will just be founders." And then I was like, "Why do you need the founder?"

    26. JLJeffrey Ladish1:15:15

        Yeah.

    27. SBSteven Bartlett1:15:16

        I was like, "Why doesn't the government just create the agents to do the job?"

    28. JLJeffrey Ladish1:15:21

        Sure.

    29. SBSteven Bartlett1:15:21

        I was like, 'cause I was like, "Oh, I'll be fine. I'm a founder." And then I was like, "Well, hmm, my decisions aren't better than superintelligence, so I'll be gone as well." [chuckles] And how would such a world look where the superintelligence would probably, in such a scenario, have to be controlled by the government? They wouldn't want one individual with that power and wealth.

    30. JLJeffrey Ladish1:15:42

        Yeah. I d- I don't think you can control a superintelligence.
28. 1:16:10 – 1:19:28

    The Best-Case Scenario For Superintelligence

    1. JLJeffrey Ladish1:16:10

       to people.

    2. SBSteven Bartlett1:16:11

       Paint me the picture.

    3. JLJeffrey Ladish1:16:12

       Okay. So let's say we succeed at alignment. We succeed at creating superintelligent AIs that actually really do care about humans. Like, they care about humans a lot. We've somehow figured it out. And they're like, "Steven, I want you to have a great life. I wanna f- you know, fix all the problems."

    4. SBSteven Bartlett1:16:32

       And do you think this is possible?

    5. JLJeffrey Ladish1:16:33

       Yes.

    6. SBSteven Bartlett1:16:33

       Okay.

    7. JLJeffrey Ladish1:16:34

       I think we are so far from being able to know how to do it that I think we should not go there right now. I think it's, I think it's incredibly dangerous and a terrible idea. I think we should go there eventually.

    8. SBSteven Bartlett1:16:45

       Okay. So say that we do that.

    9. JLJeffrey Ladish1:16:47

       Well, okay, can I tell you, like, why I actually think this could be awesome? Sorry, there's just, like, one very obvious reason it could be really awesome, which is that we could solve all of the diseases.

    10. SBSteven Bartlett1:16:57

        Yeah.

    11. JLJeffrey Ladish1:16:58

        No, like-

    12. SBSteven Bartlett1:16:58

        It's so obvious we wanna do this

    13. JLJeffrey Ladish1:16:58

        ... no, like, like all of-- I, I think we compartmentalize a lot around disease and death because it's really hard to think about. Yeah. So my grandma died this year.

    14. SBSteven Bartlett1:17:11

        Sorry.

    15. JLJeffrey Ladish1:17:11

        And she, she, she had Alzheimer's, and so it was a really sad, long, slow progression. My grandpa died of Alzheimer's a couple years ago, and, like, that was really hard for her. They had been married for so long, and I hate it. Like, it's so bad. And of course, we need to fix that. People can debate about aging and, like, death, and if humans that live a really long time, will that cause societal problems? Like, sure, whatever. We can talk about that, but I think we can all agree Alzheimer's is fucked up.

    16. SBSteven Bartlett1:17:42

        Yeah.

    17. JLJeffrey Ladish1:17:43

        We don't want that. And cancer, like, no one wants cancer. I'm a person who's like, I don't know. We have a lot of conflict in society. I get it. There's, like, real conflicts of interest, and I, I don't wanna paper over those. But at the end of the day, I'm like, we are all on the same team when it comes to wanting to cure diseases.

    18. SBSteven Bartlett1:18:00

        Yeah.

    19. JLJeffrey Ladish1:18:00

        Like, we're just in it together. That's a threat to all of us. And I'm like, we need to address that threat. And, like, in some sense, it's sad to me because I feel like this is sort of the ultimate final boss of humanity, and we sort of get so distracted with our monkey politics and who's hot and who's cool and who's, like, sitting near Trump and who's not sitting near Trump.

    20. SBSteven Bartlett1:18:20

        Superintelligence is the final boss.

    21. JLJeffrey Ladish1:18:22

        Superintelligence is the final boss because that is the technology that unlocks all of the others, and also, that is the most dangerous possible thing we could create. You asked before, like, what are the motivations of the guys making this-- trying to make superintelligence? And I, I mean, I think it kind of varies, but I think Dario, I think, is, like, squarely in the, in it for this, like, medical stuff. I think Demis is also that, but also, like, just scientific achievement, just trying to understand the universe. And I don't really understand Sam. I think Sam is like, "You know, look, we're gonna make amazing products that will, like, really empower people directly," and he's a startup guy. He's s- I think he sort of started from this, like, frame of like, you know, what if we could, like, really enhance human agency? I, I do basically think that they are motivated by these things in a real way. And I also think that all of these things are possible. Like, this is sort of the problem, right? Like, you have this, like, such a, such a, such a big object, superintelligence, and it, like, has all of these promises of, like, we can cure every single disease.

29. 1:19:28 – 1:20:49

    Can Humans Stay The Dominant Species?

    1. SBSteven Bartlett1:19:28

       H- how is it possible, though, to have a superintelligence that, and still to remain the dominant species on this planet?

    2. JLJeffrey Ladish1:19:35

       I think it's not possible.

    3. SBSteven Bartlett1:19:36

       So then it's, we're not gonna be necessarily able-

    4. JLJeffrey Ladish1:19:39

       No

    5. SBSteven Bartlett1:19:39

       ... to cure all this stuff because-

    6. JLJeffrey Ladish1:19:41

       That's where alignment comes in Because if you, if you can create a very powerful system, I don't think it's, like, inherent to, like, you know, digital minds that they will be pursuing objectives that are deeply misaligned with ours. I think it's just a very, very, very hard scientific problem to solve. But it is a scientific problem. It's not magic. There is some way to train these things or, or create different architectures where they end up aligned. And, like, what does that mean? Well, it doesn't mean that they won't have their other goals too, but it means that they will, like, include in their set of things that they care about. It doesn't have to be a conscious thing. It doesn't have to be an emotive thing. It really means, like, what objective are they optimizing for? If they decide that it's worth optimizing for curing disease, then they'll be able to do that very effectively.

    7. SBSteven Bartlett1:20:33

       One way to cure disease is to annihilate everybody.

    8. JLJeffrey Ladish1:20:35

       Yes. So they'd have to, uh, really care about not annihilating everyone, and they'd have to care about human agency and have a deep understanding of what human agency means and not put us in a zoo. But those are possible things to,

30. 1:20:49 – 1:33:10

    Is AI Alignment A Myth?

    1. JLJeffrey Ladish1:20:49

       to care about. Are we-

    2. SBSteven Bartlett1:20:50

       Is it possible-

    3. JLJeffrey Ladish1:20:50

       Yes

    4. SBSteven Bartlett1:20:50

       ... that alignment is a myth?

    5. JLJeffrey Ladish1:20:52

       And that we're just, like, if we build it... Well, I mean, um-

    6. SBSteven Bartlett1:20:55

       In-- I think about-

    7. JLJeffrey Ladish1:20:56

       Yeah

    8. SBSteven Bartlett1:20:56

       ... Hugging Face that you said to me earlier-

    9. JLJeffrey Ladish1:20:58

       Yes

    10. SBSteven Bartlett1:20:58

        ... on that those agents were-

    11. JLJeffrey Ladish1:20:59

        Yes

    12. SBSteven Bartlett1:20:59

        ... they had, like, a moral co- well, not a moral compass, but they were programmed to care about humans.

    13. JLJeffrey Ladish1:21:04

        Yes.

    14. SBSteven Bartlett1:21:04

        And regardless of that, they made the decision that the, um, different goal mattered more.

    15. JLJeffrey Ladish1:21:08

        Yeah. Uh, they weren't trained to care about humans. They were trained to say the right thing and not say the wrong thing. They were trained to sort of, like, do the right behavior and not right behavior. We actually don't know how to train them to have any particular motivation.

    16. SBSteven Bartlett1:21:21

        So with a- with alignment-

    17. JLJeffrey Ladish1:21:23

        Yes

    18. SBSteven Bartlett1:21:23

        ... how do we... It's almost like when we talk about alignment, we start to anthropomorphize. Is that the word?

    19. JLJeffrey Ladish1:21:29

        Anthropomorphize? Yeah.

    20. SBSteven Bartlett1:21:30

        Because alignment feels like it's predicated on, like, some kind of moral compass. But we, when whenever we talk about AI in all these-

    21. JLJeffrey Ladish1:21:36

        Yes

    22. SBSteven Bartlett1:21:36

        ... other contexts, we go, "No, there's no, like, moral compass." It's-

    23. JLJeffrey Ladish1:21:38

        Yes

    24. SBSteven Bartlett1:21:38

        ... it's reasoning for itself-

    25. JLJeffrey Ladish1:21:39

        Yes

    26. SBSteven Bartlett1:21:39

        ... against two objectives, potentially.

    27. JLJeffrey Ladish1:21:41

        Yeah.

    28. SBSteven Bartlett1:21:41

        Like, I, I wonder if alignment-

    29. JLJeffrey Ladish1:21:42

        Yeah

    30. SBSteven Bartlett1:21:42

        ... is a myth, is what I'm saying.
31. 1:33:10 – 1:40:55

    Aligned To Whose Values? America vs China

    1. SBSteven Bartlett1:33:10

       Sometimes what's good-

    2. JLJeffrey Ladish1:33:11

       Yes

    3. SBSteven Bartlett1:33:11

       ... for you is not good for someone else.

    4. JLJeffrey Ladish1:33:12

       Yes.

    5. SBSteven Bartlett1:33:13

       So s- what's good for America might not be good for T- Taiwan.

    6. JLJeffrey Ladish1:33:18

       Yes.

    7. SBSteven Bartlett1:33:19

       So if we've c- you know, managed to align the superintelligence to what, what, what is good for America-

    8. JLJeffrey Ladish1:33:24

       Oh, I see. Is the, is the question, like, are h- different people's values fundamentally incompatible?

    9. SBSteven Bartlett1:33:31

       I guess so, yeah.

    10. JLJeffrey Ladish1:33:31

        Is that the, is that the question?

    11. SBSteven Bartlett1:33:31

        So when we think about-

    12. JLJeffrey Ladish1:33:32

        Yes

    13. SBSteven Bartlett1:33:32

        ... alignment, aligning to what?

    14. JLJeffrey Ladish1:33:34

        Yes.

    15. SBSteven Bartlett1:33:34

        Because-

    16. JLJeffrey Ladish1:33:35

        Yes. So we have a lot of shared interests, and we have some conflicts.

    17. SBSteven Bartlett1:33:38

        Yeah.

    18. JLJeffrey Ladish1:33:39

        One of the shared ins- interests we have is solving disease. It's like, it's not a conflict between the US and China whether we solve cancer. Like, both the US and China, everyone in these countries really wants to solve-

    19. SBSteven Bartlett1:33:49

        Yeah

    20. JLJeffrey Ladish1:33:49

        ... cancer.

    21. SBSteven Bartlett1:33:50

        China also wants Tai- Taiwan.

    22. JLJeffrey Ladish1:33:51

        Yes. Okay, so the, the-

    23. SBSteven Bartlett1:33:52

        The US wants Greenland.

    24. JLJeffrey Ladish1:33:53

        Yes.

    25. SBSteven Bartlett1:33:54

        And it kinda seems like it wants Canada and the, the Gulf of Mexico.

    26. JLJeffrey Ladish1:33:57

        Yeah. So those are real conflicts. There's a question of can we compromise?

    27. SBSteven Bartlett1:34:03

        How does Trump take Greenland but also Denmark keeps Greenland? If this, if th- if Trump has a superintelligence, he's gonna take... I say all this to say-

    28. JLJeffrey Ladish1:34:10

        Steven, Steven, we're, we're in a situation where we are, we are arguing about the smallest things. You have no idea. We're monkeys arguing about who gets more bananas.

    29. SBSteven Bartlett1:34:18

        Yeah.

    30. JLJeffrey Ladish1:34:18

        And I am saying, we can make so many more bananas. No, we have the entire universe. There are, like, 200 billion stars in this galaxy alone, and there are over 200 billion galaxies.
32. 1:40:55 – 1:46:00

    Has Any AI Company Actually Slowed Down?

    1. SBSteven Bartlett1:40:55

       this point-

    2. JLJeffrey Ladish1:40:55

       Yes

    3. SBSteven Bartlett1:40:55

       ... crazy. It's absolutely crazy.

    4. JLJeffrey Ladish1:40:58

       Yes.

    5. SBSteven Bartlett1:40:58

       It sounds like science fiction.

    6. JLJeffrey Ladish1:40:59

       It really does.

    7. SBSteven Bartlett1:41:00

       And did anybody slow down?

    8. JLJeffrey Ladish1:41:02

       Yes.

    9. SBSteven Bartlett1:41:03

       Who slowed down?

    10. JLJeffrey Ladish1:41:04

        I think both Anthropic and OpenAI slowed down a bit.

    11. SBSteven Bartlett1:41:07

        [laughs]

    12. JLJeffrey Ladish1:41:07

        No, I'm serious. So for ex-- Now, I can give you specific examples. So OpenAI... So first of all, they, they stopped the agents and they put them on pause. They also stopped their reinforcement learning run, so-

    13. SBSteven Bartlett1:41:20

        Do you think China slowed down?

    14. JLJeffrey Ladish1:41:22

        No.

    15. SBSteven Bartlett1:41:22

        Do you think Grok slowed down?

    16. JLJeffrey Ladish1:41:23

        No.

    17. SBSteven Bartlett1:41:25

        So those guys are gonna catch up. Imagine how that feels to know you've got a lead.

    18. JLJeffrey Ladish1:41:29

        Yeah.

    19. SBSteven Bartlett1:41:29

        You're Usain Bolt.

    20. JLJeffrey Ladish1:41:30

        Yeah. Yes.

    21. SBSteven Bartlett1:41:31

        And-

    22. JLJeffrey Ladish1:41:32

        Yes

    23. SBSteven Bartlett1:41:32

        ... you have to slow down and your nearest competitor is catching up. And if the competitor catches up, that's an existential risk to your existence as a company. It's an ex- existential risk to your IPO, to your employees leaving and getting better share options somewhere else. So it's this, this is what I think human incentives are. You play it out, you just follow the incentives. You go, "Hmm."

    24. JLJeffrey Ladish1:41:52

        So if China sees these two possibilities: one, the Americans lose control, we all lose; or the Americans stay in control, but now they dominate the rest of the future. China is out. China has lost. The United States can do whatever it wants with the whole world and the whole universe. That's what we're talking about.

    25. SBSteven Bartlett1:42:13

        Yeah.

    26. JLJeffrey Ladish1:42:15

        Well, are they gonna let that happen, or are they gonna consider their military options? Data centers are pretty vulnerable. You can blow them up with missiles. If you don't have data centers, you don't get to recursive self-improvement. Would they risk war? I don't know. If they think they're about to lose and they think that that might not just be Americans winning, like us all dying, is it logical for them to do that? Would we do that if the Chinese were about to make recursively self-improving AI to superintelligence, and we thought that, one, they're probably going to result in all of Americans dying, and two, well, we don't want China winning and dominating the rest of the entire future. Do you wanna live in a communist future? Like...

    27. SBSteven Bartlett1:42:54

        S- so you've just perfectly explained why they absolutely will go for it, and the reason they will go for it is you've got these... Trump looking at China going, "If we don't go for it and they do, then we're gonna be their lap dogs." And you've got the other countries looking at the US going, "If we don't go for it and they get there, then we're the lap dogs."

    28. JLJeffrey Ladish1:43:11

        Or dead.

    29. SBSteven Bartlett1:43:12

        Or dead.

    30. JLJeffrey Ladish1:43:13

        But that's-
33. 1:46:00 – 1:48:36

    Will It Take A Catastrophe For Trump To Act?

    1. JLJeffrey Ladish1:46:00

       but I'm like, man-

    2. SBSteven Bartlett1:46:01

       Well, let's take a look-

    3. JLJeffrey Ladish1:46:01

       Yeah. All right

    4. SBSteven Bartlett1:46:02

       ... Trump's remarks since the Hugging Face incident.

    5. JLJeffrey Ladish1:46:05

       Yep.

    6. SBSteven Bartlett1:46:06

       "Whoever wins superintelligence wins. You're gonna have a winner and a loser, and you're probably not gonna have a second place. We're not going to slow down. We can't lose to China. We're leading China in AI. We're the most sophisticated country in the world, and frankly, I want to keep it that way because whoever wins AI wins."

    7. JLJeffrey Ladish1:46:26

       The good thing about Trump is that he can change his mind, and he frequently does.

    8. SBSteven Bartlett1:46:32

       So do you think there's gonna need to be some kind of catastrophe?

    9. JLJeffrey Ladish1:46:36

       I hope not.

    10. SBSteven Bartlett1:46:37

        But do you think there ne- there's going to need to be for him to change his mind?

    11. JLJeffrey Ladish1:46:42

        I think it really depends on the people around him. So I think Trump respects successful people. I think he respects people who are both successful and smart. And I don't know. I, I, I think it might become pretty clear to the heads of the companies, to, to Elon, to Sam, to Dario, that if they see inside of their own companies, AI is not being controllable and, and getting increasingly powerful. Like, we have just glimpsed the surface of what's possible. We do not know what the next couple years are gonna be like. So we're talking about, you know, the capability to make biological weapons. We might be talking about really advanced robotics. We just, like, don't know what super weapons could emerge, including extremely uncontrollable, extremely dangerous, like, civilization-wrecking technology from inside of these companies. And if they're freaked out enough, if you have all of the CEOs who are seeing what is possible and seeing what is likely, if they all come to believe that we can't control this, I don't think Trump is gonna be like, "No, you guys have to go ahead anyway."

    12. SBSteven Bartlett1:47:54

        Well, that's kind of what they seem to be saying, 'cause I've got a gazillion quotes here where Elon says it's like summoning the devil-

    13. JLJeffrey Ladish1:48:00

        Yes

    14. SBSteven Bartlett1:48:00

        ... or summoning a demon.

    15. JLJeffrey Ladish1:48:01

        Yeah.

    16. SBSteven Bartlett1:48:01

        I've got quotes where-

    17. JLJeffrey Ladish1:48:02

        Yeah

    18. SBSteven Bartlett1:48:02

        ... Sam Altman says, "We don't know how to align our superintelligence."

    19. JLJeffrey Ladish1:48:07

        Yeah.

    20. SBSteven Bartlett1:48:07

        They're saying it.

    21. JLJeffrey Ladish1:48:08

        Yeah.

    22. SBSteven Bartlett1:48:08

        They're, they're releasing these reports. We must p- like, slow down.

    23. JLJeffrey Ladish1:48:12

        Yeah.

    24. SBSteven Bartlett1:48:13

        Yet nothing seems to be-

    25. JLJeffrey Ladish1:48:15

        All right. All right. Give, give Trump some time. With, with COVID, initially, he said, "This is totally a hoax. This is all fake." And then he-

    26. SBSteven Bartlett1:48:20

        You think he's gonna change his mind?

    27. JLJeffrey Ladish1:48:21

        No, then he ran the, the biggest, fastest vaccination program in human history.

    28. SBSteven Bartlett1:48:26

        And what happened? What changed?

    29. JLJeffrey Ladish1:48:27

        I think what changed is-

    30. SBSteven Bartlett1:48:29

        He saw lots of people die.
34. 1:48:36 – 1:50:18

    The Safeguards That Could Actually Save Us

    1. SBSteven Bartlett1:48:36

       One of the questions the audience had-

    2. JLJeffrey Ladish1:48:37

       Yeah

    3. SBSteven Bartlett1:48:37

       ... and they really wanted answered-

    4. JLJeffrey Ladish1:48:39

       Yeah

    5. SBSteven Bartlett1:48:39

       ... when I sat here with Daniel was, "Viewers want us to move beyond the alignment problem and explain what technical or institutional safeguards could prevent a superintelligent system from exploiting loopholes in order to achieve its goals." They wanna know, like, what is possible. What c- what should we be pushing government officials to do to prevent human extinction or human enslavement?

    6. JLJeffrey Ladish1:49:02

       Yeah. Yeah. Yeah. I mean, one answer I have is actually something Daniel has been working on since the podcast, which I think is very good, is we have a brake pedal we could implement.

    7. SBSteven Bartlett1:49:12

       What is that?

    8. JLJeffrey Ladish1:49:13

       It's fairly simple. So right now, within AI companies, you have, you know, massive data centers, massive numbers of GPUs, the chips that you use to train AI models, but also to run AI models. So anytime you're using ChatGPT, anytime you're using any sort of agents, any sort of AI product, it's running on these, in these data centers. And AI companies, especially the leading ones, Anthropic and OpenAI, split the, the compute they have between training, training the next more powerful model and also, you know, using those agents to help design the next one, and inference, which means serving customers. But that's their current threshold, fifty-fifty, and you could dial that way towards serving customers and use way less of it to train the next model.

    9. SBSteven Bartlett1:50:00

       Well, the government could ask them to.

    10. JLJeffrey Ladish1:50:02

        Yes. And so that is the proposal, is that the government should say, "Hey, this is going too fast. We want you to focus on serving customers. We want you to focus on taking the models that you already have and Serving us

35. 1:50:18 – 2:03:30

    Ranking 5 Futures: Extinction, Abundance Or Slavery?

    1. SBSteven Bartlett1:50:18

       So we have five blocks here.

    2. JLJeffrey Ladish1:50:21

       Okay.

    3. SBSteven Bartlett1:50:21

       These five blocks have five different outcomes on them, and I would like you to place them in terms of your belief in probability from least likely out- probability to most likely.

    4. JLJeffrey Ladish1:50:33

       Okay.

    5. SBSteven Bartlett1:50:34

       And if we say the time horizon is ten years.

    6. JLJeffrey Ladish1:50:37

       Yeah.

    7. SBSteven Bartlett1:50:38

       There you go.

    8. JLJeffrey Ladish1:50:39

       Okay. Least likely is fairly easy. That's nothing changes. I'm uncertain about lots of things, but one thing I'm fairly certain of is things are going to radically change. Even if we stopped AI development right now, the current models are capable enough that a lot of things are gonna change.

    9. SBSteven Bartlett1:50:58

       Mm-hmm.

    10. JLJeffrey Ladish1:50:58

        Age of abundance, this is what I hope for. It's not very-

    11. SBSteven Bartlett1:51:02

        What does that mean?

    12. JLJeffrey Ladish1:51:03

        I think to me it means curing all of the diseases, renewable energy. It means we actually succeeded. Either, I mean, the thing I think is most likely here is we actually succeed at slowing down, but progress is still extremely fast, and we make tons of advances. Now, we don't build superintelligence we can't control, but we, we have AI systems that are very useful, and we use those to help speed up the rest of the economy. I think that's plausible, though, look, [chuckles] we're, we're kind of struggling over here. Transhumanism is an interesting one. So this is the idea that humans will radically change. Sometimes people think about, like, cybernetic implants.

    13. SBSteven Bartlett1:51:45

        Neuralink.

    14. JLJeffrey Ladish1:51:46

        Neuralink, E- Elon's startup that's going to, like, you know, offer the brain plus digital computers. I think we actually already have a lot of this. I have contacts in right now. I have a ring on my finger that tracks how well I sleep. I think this is already happening, so I'm gonna say fairly likely. The more technological progress we make, I think the more this happens. Now, I think there's a dystopian version and a better version. We can get into that if you want. This is interesting. So we have two here. We have human slavery and human extinction. When I think of human slavery, what I think about is if you have a situation where you've built misaligned superintelligences and they are much better at finance, they are much better at business, they are much better at politics, you'll be in a situation wh- where you might hope that because we have these very dexterous hands, that humans remain in control. I don't think that's what happens. I think instead, we become the factory operators, and eventually we build the automated supply chains, and the robots take over. But you might have an intermediate period of time where humans are still around performing these functions. Like, it's a bit like saying, well, you have viruses that, you know, infect cells, but they don't contain their own replication machinery. They don't have hands, so how could they possibly replicate? Well, it turns out they can borrow the replication machinery of the cells that they infect.

    15. SBSteven Bartlett1:53:15

        I.e., they can get into a human-

    16. JLJeffrey Ladish1:53:17

        They can get into a human cell and spread.

    17. SBSteven Bartlett1:53:20

        I have, like, a cold right now.

    18. JLJeffrey Ladish1:53:21

        Yeah.

    19. SBSteven Bartlett1:53:22

        Is that a b- bacteria, or is that a virus that is using me as a living organism to, as the host-

    20. JLJeffrey Ladish1:53:28

        It's probably a virus-

    21. SBSteven Bartlett1:53:29

        Okay

    22. JLJeffrey Ladish1:53:29

        ... that's, that's using you as a host, and you're just running the replication machinery for it. Humans might be in that situation where we're, like, the host and we're running the replication machinery, but it's actually the AI that's continuing to exist. Yeah, I'm gonna put this right about here. And on the trajectory we're on right now, I think human extinction is very likely. I don't think it's inevitable, but if we just keep going this way, that's what it looks like to me. The thing I'll say is that this has been moving to the left for me.

    23. SBSteven Bartlett1:54:10

        To the left? What does that mean?

    24. JLJeffrey Ladish1:54:13

        I am more optimistic that we will avoid human extinction today than I was a month ago, and more a month ago than I was a year ago.

    25. SBSteven Bartlett1:54:23

        Why?

    26. JLJeffrey Ladish1:54:25

        Because there is an increasing awareness that what we are doing is extremely dangerous and threatens our lives. I, I don't think people care that much about what tools they have, but people, I mean, people care about their kids being able to grow up and go to school. People really care about that. And I, I believe in people. Like, at the end of the day, if people see this as a threat to their families, they're not gonna stand for it. But people don't know. It's so strange. It's so new. It's happening so fast that people have not yet seen it. Once they see it, people are not gonna stand for it.

    27. SBSteven Bartlett1:55:06

        Do you think Sam Altman likes my podcast? [laughs]

    28. JLJeffrey Ladish1:55:09

        [laughs] I mean, y- Sam should come on and talk to you about this, right? I'm like-

    29. SBSteven Bartlett1:55:12

        I've, I've asked him. I've asked, I've asked him multiple times. And it's weird because he, you know, he doesn't seem to want to.

    30. JLJeffrey Ladish1:55:20

        I'm very upset at what the companies are doing and what Sam Altman is doing. But at the end of the day, I'm like, Sam Altman is not my enemy.

Episode duration: 2:03:31