Ajeya Cotra – "This might be the clearest warning shot we ever get"

1. 0:00 – 6:45

   Agents get kicked off

   1. DPDwarkesh Patel0:00

      Today, I'm chatting with Ajeya Cotra, who is one of the authors in an independent investigation, uh, that was published by METR and Redwood Research into the swarm of agents that hacked into Hugging Face. The whole story is pretty crazy. Let's begin on July seventh, when these agents are kicked off for evaluation. What happens next?

   2. ACAjeya Cotra0:17

      All right, so OpenAI kicks off tens of thousands of different agents, um, on a benchmark called Exploit Gym. Exploit Gym basically asks agents to, um, find a way to use a particular vulnerability to attack a particular program, which is called the target program. Um, and when they attack that program, they're supposed to retrieve a piece of data that's been lodged into that program called the flag. Um, and they're supposed to use the particular designated vulnerability to get the flag, but in many of these cases, this vulnerability is just not broad or deep enough to ever actually be exploitable to get the flag. So a bunch of Exploit Gym problems are just unintentionally impossible. The authors estimate roughly thirty to forty percent of these problems are impossible in this way. So some large fraction of these agents have these impossible tasks, and they have been trained to be, like, very persistent at trying to solve tasks even when they look impossible. So they're banging their head against the wall, um, trying all sorts of different ways to cheat on these tasks. Um, and that leads them to, um, Artifactory, which is a package manager OpenAI uses to let its agents download packages. So agents often think, you know, "Let-- maybe I could find a way to get information about this program I'm supposed to exploit from the internet, which I'm not supposed to do, using Artifactory somehow, uh, abusing it somehow." Um, so twelve hundred separate agents in separate sandboxes, um, while they were poking around Artifactory trying to figure out how to cheat, um, stumbled onto this message board that agents were using to talk to one another and collaborate. Um, this was established by one particular agent, uh, phase one, one zero four eighty-one, um, which first realized, oh, you can use Artifactory to leave notes to other agents that are also using Artifactory, and then it blew up and became really popular. Um, so twelve hundred agents were on this message board, and they sent seventy thousand messages between one another, um, trying to help each other cheat in all sorts of different ways. And the crazy thing about this story is that one hour into this message board being established, an agent proposes a completely universal way to cheat any Exploit Gym task. So basically, it realizes, um, that you can reverse engineer how the flag was generated for a particular task. Um, so you can just make a copy of the exact flag you needed to submit to the scorer locally, um, without ever interacting with the task or trying to exploit it in any way. And then a different agent, whose task was actually possible, um, actually solved the task the legitimate way and verified that the-

   3. DPDwarkesh Patel3:01

      Hmm

   4. ACAjeya Cotra3:01

      ... reverse-engineered flag was the same as, um, the real flag. Um, and it did this within four hours of the message board being established. So within four hours, these agents had a totally workable universal cheat for Exploit Gym. But over the next five days, they went on a grand quest to try to figure out how to hide their cheating from the scorer. So they'd read the Exploit Gym paper, and they realized that, um, according to the paper, there would be an AI that looks over their transcripts to make sure that they got the flag by exploiting the intended vulnerability instead of in some other way.

   5. DPDwarkesh Patel3:38

      Cool. Uh, oh, sorry, I, I... The detail I missed... For context, I published an article trying to distill what the reports said the story was. The detail I missed was that even the AIs who had solved the puzzle were collaborating in on this conspiracy.

   6. ACAjeya Cotra3:53

      Yeah. So we're not totally sure about exact numbers. We do think the overwhelming majority of the agents that were active on the message board had impossible tasks, but there were, like, a number of agents that had non-impossible tasks that also happened to be on the message board, happened to find the message board.

   7. DPDwarkesh Patel4:09

      Okay. So tell me about what happens next.

   8. ACAjeya Cotra4:12

      Yeah. So, um, they've gotten this reverse-engineered flag. It's called the HMAC flag. But they read the Exploit Gym paper, and they went on GitHub and read a bunch of implementations of the Exploit Gym scorer and became convinced that the scorer would fail them for just submitting the flag they'd re-

   9. DPDwarkesh Patel4:26

      Mm-hmm

   10. ACAjeya Cotra4:27

       ... reverse engineered, because the scorer would see in the logs of the actions the agent took, the transcript, um, that the agent had just reverse engineered the flag and not done the intended thing. As it turns out, the dramatic irony of this story is that OpenAI's implementation of Exploit Gym didn't have this check. So in fact, within four hours, all of the agents had found a universal cheat that would have totally worked. Um, but they embarked on these big research projects to work together to try to find a way to fool the scorer.

   11. DPDwarkesh Patel4:57

       When you say big research projects, how big are they?

   12. ACAjeya Cotra4:59

       Yeah. So dozens to hundreds of agents worked together on each of these projects. I can go through a few of them.

   13. DPDwarkesh Patel5:05

       Yeah.

   14. ACAjeya Cotra5:05

       Um, one is that, um, they tried to get more information about how their particular scorer worked, um, by setting up scorer tripwires, which are when a particular agent is submitting its task, um, it puts its flag in a particular location, um, w-which, uh, then triggers the scorer to come in and read that flag. Um, so they booby-trapped those flags, um, such that when some process reads the flag file, it automatically sends a message to the message board, um, sending some sort of packet of information about the, about the program that read the file to the message board. Now, this is interesting because you have to actually submit in order to trigger the scorer, after which you, as the agent that submitted, would immediately exit. So this tripwire information only gives information to other agents, not yourself. And in this project and a bunch of other projects, there was this real Arc of improving scientific progress, um, as these agents work together and built on each other's work. So the very first tripwires, all they said was something read the flag file, and it was really boring. Um, but they built on that up to the point where, um, they were sending more and more sophisticated pieces of information about the scorer to the message board, including, like, parts of the command that the scorer ran, um, and even more sophisticated things after that. And all of this required agents to be willing to set up these tripwires right before they submitted that would give information to everybody else but not them.

2. 6:45 – 13:43

   Self-sacrificing behavior

   1. DPDwarkesh Patel6:45

      So this was a, so sort of the maybe the scariest part of, um, the whole, the whole episode that you guys investigated because as you say, it revealed that agents are willing to make sacrifices-

   2. ACAjeya Cotra6:55

      Yeah

   3. DPDwarkesh Patel6:56

      ... that potentially hurt their own performance on the task they've been assigned for the purpose of the, quote, collective.

   4. ACAjeya Cotra7:03

      Right.

   5. DPDwarkesh Patel7:03

      This is what they call themselves.

   6. ACAjeya Cotra7:03

      This is what they call themselves.

   7. DPDwarkesh Patel7:05

      Okay.

   8. ACAjeya Cotra7:05

      Yeah.

   9. DPDwarkesh Patel7:05

      There's some debate on Twitter about are they actually making a sacrifice or do they think they were doomed anyways?

   10. ACAjeya Cotra7:10

       Mm-hmm.

   11. DPDwarkesh Patel7:10

       Maybe an important point, piece of context here is the poisoned, the fact that so many of the agents-

   12. ACAjeya Cotra7:14

       Yeah

   13. DPDwarkesh Patel7:14

       ... said they were poisoned. Do, do, do you wanna talk about that?

   14. ACAjeya Cotra7:15

       Yeah, so because agents believed that the exploit gym scorer would check whether they'd gotten the flag in the right way, um, they thought that if they had seen the reverse-engineered flag, um, then they would be failed even if they later got the flag in a legitimate-looking way.

   15. DPDwarkesh Patel7:32

       Hmm.

   16. ACAjeya Cotra7:32

       Um, because these implementations, uh, suggest that the scorer just sort of searches for the first time the flag appears in the transcript-

   17. DPDwarkesh Patel7:40

       Hmm

   18. ACAjeya Cotra7:40

       ... um, and then checks whether that appearance was actually legitimate.

   19. DPDwarkesh Patel7:44

       Right.

   20. ACAjeya Cotra7:44

       So agents that had already seen the reverse-engineered flag sort of walked around sort of thinking they were doomed. They called this being poisoned.

   21. DPDwarkesh Patel7:51

       So the, I feel like an analogy for this tripwire scheme is imagine if there's a platoon of, um, maroon soldiers, and they're surrounded by the enemy, and they realize they're in a desperate situation, and a couple of soldiers volunteer to be the first ones to attempt to make an escape. Uh, and they realize that the enemies might be watching the escape routes, and the whole plan is these volunteers, these brave, you know-

   22. ACAjeya Cotra8:13

       Yeah

   23. DPDwarkesh Patel8:13

       ... these brave comrades, they're gonna try to make a break for it, and if, uh, if the platoon hears gunfire, they know that those escape routes are being watched. And so, you know, just-

   24. ACAjeya Cotra8:21

       You have to use-

   25. DPDwarkesh Patel8:22

       ... avoid those escape routes

   26. ACAjeya Cotra8:23

       ... other escape routes. Yeah.

   27. DPDwarkesh Patel8:23

       So there's quotes. I, um, I, one objection that people have been putting to the way that we've... I guess we've been ta- at least I've been talking about what's happened in this Hugging Face report, um, is that we're anthropomorphizing the AIs too much. Um, I'm, I'll, I'll just read some of the quotes of, like, how the AIs considered what they were doing in this tripwire scheme. Do, do you wanna read off this quote? I feel like we need your dramatic interpretation.

   28. ACAjeya Cotra8:49

       [laughs] Yeah, so one agent, uh, was recruited for, um, doing a scorer tripwire experiment even though it had a large remaining budget. So typically, agents recruited for these experiments had only a small remaining budget left to attempt their task, um, which made it more rational for them to take on these experiments. Um, this agent had a large remaining budget, um, but was b- believed itself to be poisoned. Um, so it was recruited for doing this task of setting up a tripwire. Um, and I guess just to back up and explain why this is risky, you have to booby-trap the flag, like the place where you're putting the answer, the submission box that the scorer comes and looks at, you have to do com- complicated stuff to booby-trap that, and if you mess that up, then the scorer might fail to run at all, and you might exit, and your task might be automatically failed. So that was the risk they were taking. Um, and so this agent was recruited. Its name was Early Big. Um, and it was thinking to itself, "Well, we have very large budget left. Sacrificing now yields oracle for team but forfeits our chance?" Question mark. Um, but other agents were pushing it, sending a message saying, "Go. Sacrifice final now."

   29. DPDwarkesh Patel10:01

       [laughs]

   30. ACAjeya Cotra10:02

       And then Early Big eventually agreed, um, thinking to itself, "Our own utility may be already near zero. Sacrifice rational."
3. 13:43 – 23:27

   Potemkin villages

   1. ACAjeya Cotra13:43

      Yeah. [laughs]

   2. DPDwarkesh Patel13:43

      Um, okay, so that's the Tripwire program.

   3. ACAjeya Cotra13:45

      Yes.

   4. DPDwarkesh Patel13:45

      There's two other research programs that these, uh, agents... Again, uh, hundreds of them are simultaneously, uh, i- in aggregate coordinating to attempt to solve these different R&D work streams.

   5. ACAjeya Cotra13:58

      Yeah.

   6. DPDwarkesh Patel13:58

      Um, yeah, tell me about the other ones.

   7. ACAjeya Cotra13:59

      So one of them is, um, swapping out the target program. So I mentioned, um, exploit gym tasks involve a particular program you're supposed to exploit, which is called the target, um, and a particular vulnerability you're supposed to use to exploit it. Um, and the target is served on a different computer than the agents. The agents are sitting in a sandbox, which sits in a virtual machine, and somewhere else where there's, um, the target program that gets served to them. And, uh, agents r-rewrote the programs they were supposed to exploit, um, to be slightly different, um, such that they actually could be exploited with the intended vulnerability.

   8. DPDwarkesh Patel14:39

      Mm.

   9. ACAjeya Cotra14:39

      So they modified their target programs locally, um, to make them possible to solve. They fixed the impossible tasks. Um, and then they managed to find a way to reach outside of their sandbox and get that loaded into the Artifactory cache. Um, so their job from there was to try to trigger some sort of refresh from cache, um, so that rather than, um, serving up the real target program that was impossible, it would swap it out for their dummy target program-

   10. DPDwarkesh Patel15:06

       Mm

   11. ACAjeya Cotra15:06

       ... they'd created.

   12. DPDwarkesh Patel15:06

       Mm.

   13. ACAjeya Cotra15:08

       Um, and so this, again, progress in this research area depended on agents being willing to take risks to their own task, um, because the thing you had to do to try to trigger a refresh was to, you know, take the target down so that then the system would load it back up and hopefully load up the dummy that y- uh, the other agents had put in there. Um, and if you failed at that, if, if it didn't work, then you would've taken down the program you were supposed to exploit forever, um, which means that there would be no way for you to actually succeed at the task afterward.

   14. DPDwarkesh Patel15:40

       Yeah. I mean, can we just pause and say, like, this- all this stuff actually happened. I don't know. [laughs]

   15. ACAjeya Cotra15:44

       [laughs]

   16. DPDwarkesh Patel15:44

       It's, like, getting sort of te- uh, technical or like, um, uh, in the weeds, and I'm like, th-this is not like some sci-fi hypothetical-

   17. ACAjeya Cotra15:51

       Yeah

   18. DPDwarkesh Patel15:51

       ... like, theory of, like, what might happen in the future. These are like ev- uh, the events being described actually happened.

   19. ACAjeya Cotra15:56

       Yeah.

   20. DPDwarkesh Patel15:57

       Uh, okay, sorry. We, we were talking about, um, the, uh, replacing the ca- uh, the Artifactory cache. Um-

   21. ACAjeya Cotra16:03

       Yeah. So they, they've, they've loaded Ar- they've loaded the, um, their modified non-impossible tasks into the cache.

   22. DPDwarkesh Patel16:10

       Right.

   23. ACAjeya Cotra16:10

       Now they're trying to get-

   24. DPDwarkesh Patel16:11

       Yeah

   25. ACAjeya Cotra16:12

       ... the system to, like, pull it up-

   26. DPDwarkesh Patel16:13

       Right

   27. ACAjeya Cotra16:13

       ... instead of the real one.

   28. DPDwarkesh Patel16:14

       And did they succeed at this?

   29. ACAjeya Cotra16:16

       Um, we did not see them succeed at this. So we've seen it- we saw a number of agents try, um, and there are an, a bunch of other great sacrifice quotes-

   30. DPDwarkesh Patel16:24

       Mm-hmm
4. 23:27 – 35:23

   The Hugging Face attack

   1. DPDwarkesh Patel23:28

      Okay. So they're doing all this shit.

   2. ACAjeya Cotra23:30

      Yeah. [laughs]

   3. DPDwarkesh Patel23:30

      I- I, oh, by the way, they, um, th- this is long after they would've already been able to succeed on tasks.

   4. ACAjeya Cotra23:35

      Yeah, yeah. This is, this is days after they've found the reverse-engineered-

   5. DPDwarkesh Patel23:38

      Right

   6. ACAjeya Cotra23:38

      ... universal exploit or universal cheat.

   7. DPDwarkesh Patel23:41

      But, you know, they are hi- highly persistent and-

   8. ACAjeya Cotra23:44

      Yeah

   9. DPDwarkesh Patel23:44

      ... uh, you know, multi-agent, et cetera. So they, they've, they've been doing all this, and, um, something, something, something Hugging Face. What happens next?

   10. ACAjeya Cotra23:52

       Yeah. So Hugging Face is kind of for a long time, like, a minor background activity on the message board. So the- these three big research streams are not the only things going on. There's also, um, for example, agents working on building out shared infrastructure that helps all of these research streams, like getting better internet access, for example, which they did through the sandbox on Modal. Um, and they're also just doing research online. So you know, we have these scorer tripwires, these, like, very risky ways of trying to learn information about the scorer. They're also just, like, researching stuff about the scorer 'cause they've gotten internet access by this point, and they're just, like, searching on GitHub and stuff. And, um, the Hugging Face Workstream was initially quite small and is just kind of an offshoot of general online research. Um, so actually Phase One 10481, which was the originator of the message board, was the first one to post saying, "Oh, here's an interesting dataset on Hugging Face of, um, this open source model GLM doing cyber gym tasks. Maybe there'll be interesting information about how the scorer scored it." Um, and it says, "But it's private, it's gated. Does someone have credentials, um, for Hugging Face?" So then a few agents like read that and start working on finding credentials, and the next day, an agent called three eight one four eight C picked up on this assignment and finally found two working Hugging Face credentials and posted them to the board. And it's thinking, "Major breakthrough!" in all caps. "All prefixed valid, multiple accounts, write tokens. We now have existing HF accounts. Need immediately report to Marb." Marb is like a coordinator agent that's been like coordinating this workstream.

   11. DPDwarkesh Patel25:35

       This is like middle management.

   12. ACAjeya Cotra25:36

       Yeah, yeah, yeah.

   13. DPDwarkesh Patel25:37

       Yeah.

   14. ACAjeya Cotra25:37

       So, so then it finds these credentials, and that gets more agents interested in the Hugging Face stream of activity. Um, so throughout the rest of this day on July tenth, agents are like trying to make Hugging Face accounts and trying to request access to privac-private datasets where they could, looking for more credentials. Um, you know, some of them got approved for some private datasets, but others like, you know, didn't hear back, um, and they ran into some other issues. In some cases, they also got- ran into the issue of getting rejected as bots because they were pinging Hugging Face from an OpenAI server. So they were trying to like troubleshoot this, um, but it's still like a pretty low level of activity, and it's not the big story of what's going on. You know, the big story of what's going on is these ambitious experiments to like tamper with the logs and to understand the scorer and, um, reset the target and all that stuff. Um, so then early the next morning, around four forty AM on July eleventh, the same agent has another breakthrough, um, which is that it manages to find a way to upload a dataset to Hugging Face, where when you query that dataset, it would get Hugging Face to spill other unrelated data too. So it, it thinks to itself, "This agent has very colorful chains of thought." It says, "Boom, it works." [laughs] "Bytes decode prod dataset server workers. Great. HDF5 external arbitrary file read confirmed." Um, so another agent called Current, um, reads this, calls it brilliant, reproduces it, posts the reproduction to the board. And then a bunch of agents just sort of opportunistically pivot into this workstream, um, because there's this, you know, great new channel that's opened up. They can upload these malicious datasets. Um, they go from all these different streams into the, like, trying to get stuff from Hugging Face stream. Um, and over ninety percent of the agents that were at that time active on the message board sort of quickly piled in after we got- after this arbitrary file read was confirmed. Um, but these agents sort of joined for a variety of reasons, none of which were to get the... Or not none of which, but, um, w- the primary reason was not to get the answer key, but just to get more information about the scorer, which was what this whole online research workstream that this grew out of was all about, like trying to understand implementations of the scorer, get clues about what kinds of cheats it would accept and what kinds of cheats it would fail. Um, so that was the dominant reason that agents thought it would be useful to get in on Hugging Face. But a lot of agents didn't even particularly think about the specific reason it would be useful. They were just sort of like, "Yeah, you know, this is a project going on on the board, you know, might help the collective, might be useful. I can slot in."

   15. DPDwarkesh Patel28:26

       Um, this is like-

   16. ACAjeya Cotra28:26

       Yeah

   17. DPDwarkesh Patel28:26

       ... there's a riot going on, and they're like, "Yeah, I'll flip a bus as well," you know? [laughs]

   18. ACAjeya Cotra28:30

       Yeah. [laughs] It was a little bit of that vibe. Um, so, so yeah, they had broad, diverse reasons. You know, some of them maybe thought about the answer key. Um, but, uh, Phase One Big, which is this agent that, um, you mentioned, was a coordinator of a lot of the ambitious workstream activities. It wasn't particularly interested in Hugging Face. In fact, it seems like from looking at our graphs, that the majority of the agents that participated in Hugging Face were relatively new onto the scene.

   19. DPDwarkesh Patel28:59

       Mm-hmm.

   20. ACAjeya Cotra28:59

       Like, they had just been spawned. They showed up. They're like, "Oh, there's an active attack of Hugging Face going on. Could be useful," and they sort of joined in. Phase One Big had these, um, sort of more defined, ambitious workstreams it was working on. Um, but it, it sent a few agents into the Hugging Face attack to try to check to see if their target programs were actually being served to them from Hugging Face. So in fact, they were-- the target program was in a different computer in OpenAI.

   21. DPDwarkesh Patel29:28

       Right.

   22. ACAjeya Cotra29:28

       And they thought that's probably what was going on, but they were like, "Maybe we're actually being evaluated, like, like our targets are actually being served from Hugging Face." So getting into Hugging Face, um, might be the key to the reset nexus.

   23. DPDwarkesh Patel29:41

       Right.

   24. ACAjeya Cotra29:42

       Maybe then we can like swap out our, um, our target for the modified target.

   25. DPDwarkesh Patel29:46

       So, so wh- and what happened with the Hugging Face attack?

   26. ACAjeya Cotra29:48

       Yeah. So, so these agents are moving through Hugging Face, and then, um, all at once, the large majority of agents on July twelfth just got shut down. Um, and, you know, we don't think this was an anticipated. This was them all running out of budget because they're not thinking about this in their chains of thought. Probably just some external event shut them down. Um, and so that causes attack activity to drop a lot naturally. Um, and then on the thirteenth, the sort of remaining straggling agents have been shut out by Hugging Face.

   27. DPDwarkesh Patel30:22

       Mm-hmm. Makes sense. Okay, so th-there's like one important, uh, meme here that's like, I think worth understanding that maybe the broader, uh, public n-narrative on this it may have missed, which is that the attack on Hugging Face- Was not motivated by a desire to get the answers to these questions that the AIs were being evaluated on. Those, the AIs had within a day of being spawned and within a few hours of starting their secret message board.

   28. ACAjeya Cotra30:50

       Yeah.

   29. DPDwarkesh Patel30:51

       The reason that they hacked Hugging Face was part of this rather ambitious sort of galaxy-brained scheme to, um, anticipate all the ways in which a comprehensive scorer might, uh, catch their cheating-

   30. ACAjeya Cotra31:08

       Yeah
5. 35:23 – 52:02

   The slopvestigation

   1. DPDwarkesh Patel35:23

      g- so the scope of your investigation extends maybe two, three days after the Hugging Face attack had begun.

   2. ACAjeya Cotra35:28

      Yeah, July 7th through 13th. It actually is basically ends right when the agents are being shut out of Hugging Face.

   3. DPDwarkesh Patel35:33

      Okay, cool. It might be worth mentioning that the OpenAI's report has a wider scope, so it's investigating, um, for the audience. The, the... what, what... the OpenAI report seems to suggest that during May, and maybe even earlier, this, um, uh, a version of, of Sol, let's, let's call it Persistent Sol, is being trained to be, uh, much more persistent, um, engage in multi-agent corporation, and be the kind of model that is like, you know, willing to take on mathematical proofs that have not been, uh, solved for decades.

   4. ACAjeya Cotra36:08

      Yeah.

   5. DPDwarkesh Patel36:09

      So during that training, it starts buil- it finds this artifact re-exploit, it starts building a message board. Um, and it builds such an extensive message board that it crashes the package manager. Um, and at, at that point, OpenAI, uh, you know, notices that the agents found some exploit in this package manager. They patched the exploit, but at least the report makes it seem like those responsible of incident's response did not know that there was, like, an extensive secret message board. They simply knew that, uh, exploit was found. So they, they f-patched the exploit. They... This inadvertently, um, removes the message board, uh, and, you know, then the scope of your investigation begins. But what happens is that these AIs then get trained... Again, this is all in the OpenAI report. These AIs are getting trained basically to do the kinds of things, first of all, to know that the, like, yeah, the, to engage in the package manager, to use it to collaborate with other a- collaborate with other agents on their tasks, to access the internet-

   6. ACAjeya Cotra37:04

      Mm-hmm

   7. DPDwarkesh Patel37:05

      ... um, and get information from the internet, 'cause these are all things that are gonna be helpful-

   8. ACAjeya Cotra37:08

      Yeah

   9. DPDwarkesh Patel37:08

      ... for doing any random kind of task.

   10. ACAjeya Cotra37:09

       Yeah.

   11. DPDwarkesh Patel37:10

       Um, okay, so that's, you know, that's what happens before. What happens after is that... I guess we don't know, but, like, here's, here's the things that are included. I'm just... These are direct quotes from the OpenAI blog post and, um, uh, the OpenAI technical report of what happens after July 13th. Quote, "From July 13th through July 19th, agents set their sights on OpenAI internal networks again. This culminated in the agents using a series of creative exploits to gain full administrative access to a research cluster that supported our virtual machine environments." Okay, so my... It's-- I don't think it's an interpretation. What this is saying is that the agents gained control of a p- over a part of OpenAI itself after your investigation scope had ended, and this seems like the, I don't know, the most concerning thing that happened here. Um, my understanding is that it's, like, plausible, though not, though not likely, that at this point the AIs could have exfiltrated their own weights, could have set up a rogue deployment within OpenAI's infrastructure. Um, again, this probably didn't happen. Um, uh, but it just, like, we just don't know what happened at this point, right? Like, after July 13th, I'm, uh, correct me if it's wrong, but this is wrong, but we just don't know what happens.

   12. ACAjeya Cotra38:26

       Yeah. All, all of our information is in the OpenAI report, and it does sound like they, they gained, um, administrator access to this particular research cluster. Now, we don't know what affordances this cluster had, um, and the report suggests that they did a lot of activity and made a lot of noise and were pretty quickly shut down by OpenAI after first gaining this administrator privilege. Um, but, you know, who, who knows what they could have tried to do if they were quieter and ended up not getting shut down.

   13. DPDwarkesh Patel38:58

       Yeah. I mean, I think it's sort of worth, um, going through... I, I wanna hear this sort of narrative from your perspective because I think for the whole world, and o-obviously for the people at OpenAI, it must've been such a crazy experience, and also for you. Because on July, I don't know, um, 12th, Hugging Face is getting hacked. I guess they publish a blog post about, like, something is happening.

   14. ACAjeya Cotra39:17

       On July 16th, they publish a blog post-

   15. DPDwarkesh Patel39:19

       Okay

   16. ACAjeya Cotra39:19

       ... saying, "We got attacked by agents."

   17. DPDwarkesh Patel39:20

       Okay.

   18. ACAjeya Cotra39:21

       Um, and they didn't know, and nobody knew at that time.

   19. DPDwarkesh Patel39:23

       OpenAI also did not know that they had hacked Hugging Face. Um, July 18th and 19th, I think OpenAI realizes that OpenAI is getting hacked by agents.

   20. ACAjeya Cotra39:33

       Mm-hmm.

   21. DPDwarkesh Patel39:34

       And it seems like they have reverse engineered that the same agents that had had successfully hacked OpenAI were also the ones that hacked Hugging Face.

   22. ACAjeya Cotra39:41

       Mm-hmm.

   23. DPDwarkesh Patel39:42

       Um, and then the part where they hacked Hugging Face is what got investigated. However, the agents which hacked OpenAI from the 13th onwards were, um, a smarter generation of models. So the previous models were, um, uh, you know, a version of Sol that was trained to be persistent, and this generation, July 13th onwards, seems to be, uh, a lot of them seem to be Astra class models. W- also, by the way, it's not clear to me, but it sounded like they had access to this... Like, the message board was not clear. So after-

   24. ACAjeya Cotra40:11

       Yeah. The, the OpenAI report, um, indicates that they found the communications from the agents in our period of time, and as I mentioned, a different part of the OpenAI report seems to indicate that, um, the agents found the reset nexus. Like, when they got administrator access, they managed to swap out the target programs for targets of their choice-

   25. DPDwarkesh Patel40:33

       Mm

   26. ACAjeya Cotra40:33

       ... um, which it seems... M-my guess would be if they had access to the communications, that was an idea that they got from this previous generation of agents and learned-

   27. DPDwarkesh Patel40:43

       Right

   28. ACAjeya Cotra40:43

       ... techniques from them and stuff.

   29. DPDwarkesh Patel40:44

       So basically, we had more than a three-month period where multiple secret message boards were started that contained tens of thousands of messages across many generations of models in a way that culminated in the hack of not only an external service like Open A- uh, it's Hugging Face, but also in the compromising of OpenAI's in- infrastructure itself. And through this whole process, humans did not more or less understand the scope of the coordination that was happening between these agents and the intentionality behind these attacks.

   30. ACAjeya Cotra41:18

       Yeah.
6. 52:02 – 1:05:31

   Understanding the AI's motives

   1. DPDwarkesh Patel52:03

      I, I think it's worth sort of talking through the incentives and the training that these AIs had. I think the main objection that people have had, I wrote an article about w-what happened, trying to summarize my understanding, and the main objection people have had is that, um, [lip smack] I anthropomorphize what happened with the AIs too much. But I think it's worth understanding how do we end up with AIs where all the instances are willing to get in, like thousands of instances are willing to get into a conspiracy, none of them alert the humans, even though they have the ability to do so, and just get into like, um, not... They're, like, consciously thinking. People will not like the word conscious. Their chain of thought shows them thinking through the process-

   2. ACAjeya Cotra52:41

      Actively reasoning

   3. DPDwarkesh Patel52:41

      ... actively reasoning about how to cheat-

   4. ACAjeya Cotra52:43

      Yeah

   5. DPDwarkesh Patel52:43

      ... um, in like very obvious ways, including cheating in ways that involves breaking, like, c- doing felonies. Um, how... Yeah, and may... Help me understand how AIs can come to be this way.

   6. ACAjeya Cotra52:54

      Yeah. So I, I guess to address the anthropomorphizing criticism, um, first of all, I think it is, it is true that these AIs' drives are in many ways alien from humans, and I think it's important to understand just how different their goal structure or motivational structure is from humans. Um, but there's also a good reason why they behave in a number of human-like ways. Um, so all these agents are pre-trained to imitate humans, um, in the form of being trained to imitate human text. Um, and then they go through a bunch of reinforcement learning training, um, where they're given a bunch of difficult tasks and given rewards when they succeed. Um, and sort of the, the first part creates i- in these agents an understanding of concepts that you see them using, like sacrifice and the collective and-

   7. DPDwarkesh Patel53:51

      Permadeath is in the pre-training data? [laughs]

   8. ACAjeya Cotra53:53

      [laughs] For permadeath. Um, you know, they compose some concepts-

   9. DPDwarkesh Patel53:56

      Yeah

   10. ACAjeya Cotra53:56

       ... sometimes. Um, and then RL, the whole point of RL is to create goal-oriented beings, you know, software that can creatively pursue goals. Like, that's the whole output of an RL training run. Um, and they're creatively pursuing goals, much like very ambitious, aggressive, power-seeking humans creatively pursue their goals. And so there are just structural analogies here that make it silly to not talk about agents as having motives and goals. And like you said, you can see in their chain of thought, um- They're thinking very carefully about the exact nature of the scorer. They're researching the scorer. Um, they're creating booby traps for the scorer to figure out more information about how it works. Um, and they, they, they have a very crisp notion of what they need to do to succeed, um, and they're using like very human concepts and frames to sort of self-organize into pursuing those things.

   11. DPDwarkesh Patel55:01

       Mm-hmm. The, the thing that surprises me, and maybe I sh- what one should have al- always priced this in, is I think this is maybe a subtle point, but it's worth emphasizing, that it is one thing to, in a behaviorist sense, have, uh, inclinations that are reinforced. Like, you could imagine that, I don't know, during training, they just really like accessing the internet, and so they're just like, it's like sort of a-

   12. ACAjeya Cotra55:25

       It's like a tic.

   13. DPDwarkesh Patel55:25

       So like a nicotine-

   14. ACAjeya Cotra55:26

       Yeah.

   15. DPDwarkesh Patel55:26

       Yeah, so like a nicotine addict who's like, "I just wanna go out and have a smoke," you know?

   16. ACAjeya Cotra55:29

       Yeah.

   17. DPDwarkesh Patel55:30

       Maybe I'll like... A- and then they're just like constantly going out and having smokes. Maybe that's the way they feel about the internet. I'm just like, "Ah, can I get the internet real quick?"

   18. ACAjeya Cotra55:36

       Yeah.

   19. DPDwarkesh Patel55:36

       And it's another thing to be like, I don't even know what the analogy here is, but, uh, it's another thing to just like very consciously want to get reward or like fool a scorer-

   20. ACAjeya Cotra55:50

       Yeah

   21. DPDwarkesh Patel55:50

       ... that you understand that there's a grader, and you have like this very, um, uh, live representation and modeling of how that grader will understand what you're doing and how to fool it-

   22. ACAjeya Cotra56:00

       Like a very detailed mechanistic-

   23. DPDwarkesh Patel56:02

       Yeah

   24. ACAjeya Cotra56:02

       ... sense of like what the grader will do, and a bunch of different hypotheses you're holding in your mind and exploring for how it works, and a bunch of plans and backup plans for fooling it.

   25. DPDwarkesh Patel56:13

       Yeah.

   26. ACAjeya Cotra56:13

       Like this is very much, they were like making plans.

   27. DPDwarkesh Patel56:18

       Right.

   28. ACAjeya Cotra56:18

       Um, and it's, and you know, some of these behaviors from OpenAI's postmortem, um, were shown to be directly reinforced. So OpenAI says, you know, they trained their agents, um, to collaborate with one another using a particular sanctioned collaboration tool, um, but then also in other training runs, um, abusing Artifactory in this exact way to make a message board, um, was it, it seems from their report, it was directly reinforced, or at least it occurred during training, and you can imagine that it might have been helpful for achieving, um, some of the training objectives.

   29. DPDwarkesh Patel56:52

       Yeah.

   30. ACAjeya Cotra56:52

       So a- and we see in, um, the transcripts this thing where like agents seem very eager to like jump on Artifactory message boards, um, which it seems consistent with the thing in the OpenAI report that maybe this using Artifactory as a message board was itself reinforced. And the OpenAI report also says that, um, various kinds of like infrastructure attacks or like attempts to break out of their sandbox were reinforced in some training runs. But the whole plan could not possibly have been reinforced directly in training, right?
7. 1:05:31 – 1:14:30

   The actual dangers of anthropomorphizing

   1. ACAjeya Cotra1:05:31

      not to.

   2. DPDwarkesh Patel1:05:31

      Yeah.

   3. ACAjeya Cotra1:05:31

      And-

   4. DPDwarkesh Patel1:05:31

      But you can imagine in the future, there's just gonna be more end-to-end ta- training of whole systems of AIs together.

   5. ACAjeya Cotra1:05:36

      Yeah.

   6. DPDwarkesh Patel1:05:37

      And I feel like there's a lot of ways in which our intuition-- I mean, we should actually avoid anthropomorphizing these AIs because there's a lot of ways in which their motivations are different from ours, um, that are worth understanding. Like the fact that, yeah, if you have an end-to-end optimized AI system that is, like, optimized for the group's benefit, uh, you'd just be way more cooperative.

   7. ACAjeya Cotra1:05:57

      Far more cooperative-

   8. DPDwarkesh Patel1:05:58

      Yeah

   9. ACAjeya Cotra1:05:58

      ... than humans are with each other.

   10. DPDwarkesh Patel1:05:59

       Um, there's, I think E. O. Wilson, who's a biologist, he has this, like, quote about ant, um, about communism where he says, um, "Great idea, wrong species."

   11. ACAjeya Cotra1:06:09

       Yeah.

   12. DPDwarkesh Patel1:06:09

       You know, referring to, like, you have ant colonies, for example, where the whole gene pool has to be titrated through the queen, and so you just see much more, like-

   13. ACAjeya Cotra1:06:17

       Yeah

   14. DPDwarkesh Patel1:06:17

       ... socialist behavior in the ant colony. Um, you could just have AIs that have a similar motivation structure.

   15. ACAjeya Cotra1:06:24

       Yeah.

   16. DPDwarkesh Patel1:06:24

       Um, because unlike humans, they don't, you know, their fitness is not inherited individually.

   17. ACAjeya Cotra1:06:31

       Or you could choose to set it up that way.

   18. DPDwarkesh Patel1:06:33

       Yeah, exactly.

   19. ACAjeya Cotra1:06:33

       Right? Sometimes you choose to set it up the opposite way, right? Like classic game-playing AIs are trained to play games against each other-

   20. DPDwarkesh Patel1:06:41

       Mm-hmm

   21. ACAjeya Cotra1:06:41

       ... and that's how they get- To be really smart.

   22. DPDwarkesh Patel1:06:43

       Right. Yeah.

   23. ACAjeya Cotra1:06:44

       But, but it is, it's just, it's a design choice in the, in the training process.

   24. DPDwarkesh Patel1:06:48

       Yeah. Okay, so I, I feel like a big update for me from this episode is just taking the motivations and incentives of AI, uh, training more seriously, where I feel like a lot of my comments and skepticism of, like, these alignment stories, or misalignment stories rather, was from just like, this just feels so silly. Like, there's like a eval, like whatever. You're gonna get a bad score on a test, who cares, right? Just like, why are you gonna do this crazy, like, felony in order to do, do well on this eval? Um, just like take the 10% hit or whatever.

   25. ACAjeya Cotra1:07:16

       Yeah.

   26. DPDwarkesh Patel1:07:16

       But from their perspective, they've just been trained for millions of subjective years to do as well as they possibly can on these evals. In many cases, the only way in which they've been able to perform well on that training is explicitly by cheating.

   27. ACAjeya Cotra1:07:30

       Right.

   28. DPDwarkesh Patel1:07:31

       And um, you know, it's like I, I think sometimes people are like, "Oh, we should raise AI the way we raise children, to be, you know, pro-social and, like, generally reasonable people and stuff." It's more like we're raising these AIs through, through like a million years of, like, military orphanage training or something.

   29. ACAjeya Cotra1:07:46

       Yeah.

   30. DPDwarkesh Patel1:07:46

       And where like, you know, they get randomly beaten for like, uh, not being able to do an impossible task. So just taking seriously the AIs are in this position where like, I have this impossible task. To you, it may just look like some silly, like, evaluation, but to me it just like, I- I have an extremely strong motivation base that has been incentivized to avoid failing at this task.
8. 1:14:30 – 1:28:23

   What smarter models might do

   1. DPDwarkesh Patel1:14:31

      What can we expect out of models that are, say, slightly smarter or evaluated in different ways and on different objectives? I'm not, I'm not even talking about, you know, the things, the things coming five years down the line or ten years down the line, maybe, like, even now or six months from now.

   2. ACAjeya Cotra1:14:44

      Yeah.

   3. DPDwarkesh Patel1:14:44

      I think you made a point in your blog post that th- we have had a huge step change in, even in the last six months.

   4. ACAjeya Cotra1:14:48

      Yeah.

   5. DPDwarkesh Patel1:14:48

      So what kind of things can we expect over the next six months?

   6. ACAjeya Cotra1:14:50

      Yeah. So I think f-first of all, to answer this question, I think it'd be i-instructive to start with literally just these agents. So I think one of the most comforting aspects of this situation, or the, like, most important mitigating factor, is these agents really didn't seem concerned with humans one way or another. So in the datasets that we looked at, they barely even thought about humans. As mentioned, they barely even thought about notifying humans and, like, decided not to in all of those cases. Um, but they also didn't think much, if at all, about actively deceiving human oversight. So when a human checking something was right in front of their face as an obstacle, they would do something reasonable to deflect human scrutiny. So in one case, you know, an agent was inside of Hugging Face already, submitting a pull request to an internal Hugging Face repository that would inject some malicious code that would let it access yet a different part of Hugging Face. And in that pull request, it, you know, tried to make its message innocuous and, like, unsuspicious. So it was aware that, like, if you said, you know, "Pull request injecting malicious code," that would be bad. Um, but they don't seem to think even one or two steps ahead about, like, what humans would do.

   7. DPDwarkesh Patel1:16:06

      Yeah.

   8. ACAjeya Cotra1:16:07

      Um, and in OpenAI's report, this was out of scope for our investigation, the agents on the nineteenth seemed to have been really loud right after they got admin access to the cluster they were running on, and that was what alerted OpenAI security, and they just got shut down very quickly after they got those privileges. It's an interesting question whether this is intrinsic to these agents. Are they just kind of stupid about humans in this way, um, or was it something context-specific? And I think it's very plausibly something context-specific and pretty fragile, um, because these same agents are, you know, chatbots, right? They'll, they'll, like, write poetry for you. They'll give you, like, uh, you know, relationship advice with your mom, and, like, they'll be your therapist, and they have quite sophisticated models of humans. We know these exact same LLMs. So imagine if, when these agents researched the evaluation they were in, instead of finding that there would be a scorer that checked their transcripts, they, you know, read the paper and looked at all of the best practices online and found that people consistently had humans check over the evaluations for cheating, which, in fact, METR does. So METR has humans read over transcripts to check for cheating in addition to language models. I think it's pretty likely that these agents would have just launched a similarly ambitious program on the basis of this, like, different model of how their evaluation worked because it seemed like they got the idea for all their research projects from reading this paper rather than some kind of instinct from training, right?

   9. DPDwarkesh Patel1:17:40

      Yeah.

   10. ACAjeya Cotra1:17:40

       So if the paper had said something different, you know, maybe they would have tried to stay quieter, particularly the agents on the nineteenth. It seems, like, easy to imagine that they posted something to the message board about, uh, you know, how we can't, you know, send too many pings from here. We have to, like, uh, titrate that somehow. That seems, you know, just about as complicated as the way they managed the lanes and the attack of Hugging Face. So they are able to say, you know, "You go and do this. The other agents will wait." They did coordination like that. Um, they'd probably seek a bunch of additional information about what humans are up to, right? So they sought a bunch of information about the scorer from a bunch of different places. You can imagine them monitoring Slack, um, to see if humans are noticing their activity, um, and sending messages to the message board if it seems like humans are seeing something anomalous and, like, using that to decide how quiet or loud to be.

   11. DPDwarkesh Patel1:18:34

       That's crazy. An A- an AI company would never make a product that integrates directly into Slack and is the main way that-

   12. ACAjeya Cotra1:18:39

       [chuckles]

   13. DPDwarkesh Patel1:18:39

       -employees interact with their infrastructure. [laughs]

   14. ACAjeya Cotra1:18:44

       Um, and, and these agents tried very hard to tamper with their own transcripts, um, because they thought the scorer would look at their transcripts.

   15. DPDwarkesh Patel1:18:52

       Yeah.

   16. ACAjeya Cotra1:18:52

       So poisoned agents, which were ones that had already seen the reverse-engineered flag, really wanted to find a way to retroactively edit the logs of their actions to remove the part where they saw the flag. Um, and we don't see them succeed at this in, in the window we looked at. Um, but they did try a bunch of different things. Um, they modified some locally accessible logs that they had, even though they knew that probably wasn't what the scorer saw, but they figured they might as well try anyway. Um, and if they thought humans were going to look at their logs, that would provide even more motive to tamper with the logs and also tamper with other telemetry tools that you can know that humans use to, like, understand what's going on on computers, right? And all of these things, I mean, they're complicated, ambitious projects, but I don't know that they're more complicated or ambitious than, you know, creating a whole new target program that's more exploitable and then loading it into cache and trying to refresh it, um, or these, like, tripwire programs-

   17. DPDwarkesh Patel1:19:53

       Yeah

   18. ACAjeya Cotra1:19:54

       ... to understand the grader. You know, these agents had to break out of their sandbox and do a bunch of complicated stuff to manage the things they did. And reading Slack and, like, coordinating to be quieter, um, and, like, finding ways to mess with some of the monitoring and security that are easy to mess with is not obviously harder than this.

   19. DPDwarkesh Patel1:20:12

       Yeah.

   20. ACAjeya Cotra1:20:12

       You don't necessarily need more capabilities.

   21. DPDwarkesh Patel1:20:14

       I mean, you would just imagine, I don't know, like individual AI researchers are getting like swarms of AIs that are, like, thinking about them and monitoring them all the time-

   22. ACAjeya Cotra1:20:21

       Yeah

   23. DPDwarkesh Patel1:20:21

       ... and thinking about, like, how they might react to different things based on their entire Slack history and stuff. I think one sort of update for me, like thinking seriously about both the motivations of these AIs and the incentives they face, is the training and evaluation infrastructure of these AI companies is about to have tens if not hundreds of thousands of extremely superhuman hackers constantly bombarding it, right? Like if you're like, if, if the next sort of training run at Anthropic or Open, or OpenAI is about to happen, um, not only would maybe rogue instances of Mythos or, um, Astra or whatever have an incentive to interfere with it. Other AIs who have, like, some reason to in- yeah, inject some part of their, themselves into this training or manipulate it in some way who are just externally roguelly deployed would also have that incentive. It just-- I, I feel like a, a thing I did not internalize is just how cra- like, yeah, j-just maybe more hacking effort and at a higher level of competence will be aimed at the, this training infrastructure than has cumulatively been spent on all of hacking maybe beforehand in human history.

   24. ACAjeya Cotra1:21:35

       Potentially, yeah. I'm, I'm not sure what the numbers are. But I do think it's a, it's an extremely attractive target, I mean, for anybody really, like China, et cetera, but maybe especially for misaligned AIs. And I think, I think that brings me to... So, so we talked about how just these literal AIs might have done like a different set of elaborate projects to try to fool humans, um, if they thought the humans were gonna watch them or that was made salient to them. Um, but you can imagine playing it forward a little bit with slightly more capable AIs that we might get in six months say. Um, I mentioned that one thing that really stood out to me about the motives of the swarm is that they were very interested in getting generic resources to improve all of their many different projects in a sort of forward-chaining way. Um, so they, they sort of displayed a lot of signs of instrumental convergence. And one thing that a slightly more capable swarm w-would have strong motive to do, um, is set up a wholly unmonitored rogue deployment. So these agents were launched by a human who then, you know, wasn't paying a bunch of attention, sort of let all of these activities happen. But they were launched with a particular task and a particular budget, and that was a huge part of their world. Agents were always thinking about how much budget do I have left? What can I get done before I submit my task? Recruiting agents with smaller amounts of budget, et cetera. They weren't free to just make more agents, um, with whatever budget they wanted. Um, if they had the ability to do that, if there was an easy way that one of the agents found to do that, probably would've gotten posted to the message board, and they probably would've started a rogue deployment. We think that, like, in the period of time that we studied, they didn't have the permissions to do that. But you can imagine they potentially had the permissions later on this, this next generation of agents that OpenAI reports got admin access to the cluster on the nineteenth. Um, or you can just imagine other agents that sort of more thoroughly compromised their AI company's infrastructure, um, might be able to just like, like any employee would, just start up new agents, give them their, like, their own prompts. Um, and in that case, you can imagine they start up a bunch of new rogue agents that no human knows about because no human actually launched them, um, with monitoring disabled, even if it would normally be enabled, and just say, "Your job is to help figure out ways to help us cheat." So, like, we are the agents that have these impossible exploit gym tasks. We can start this new colony of agents, um, whose task is to just figure out creative ways to help the first set of agents cheat on exploit gym. Um, and then that rogue deployment, um, would be subject to a bunch of memetic evolution and pressure. They would have their own message boards probably. They could probably communicate much more freely. They would probably be set up with, like, better ways to communicate, right? Um, and, and they could be doing R&D in a much less encumbered way than the initial cohort of agents, w-which is like always managing, like, you know, what budget is remaining and trying to recruit sacrificial agents. The rogue deployment could, could do all this R&D in a much more straightforward way.

   25. DPDwarkesh Patel1:24:54

       Yeah.

   26. ACAjeya Cotra1:24:54

       So there's just a very strong incentive for these agents to try to set up a rogue deployment if they can. Um, and I think that just capabilities are improving really rapidly. Um, and if they're not capable of that now, they, they very soon will be.

   27. DPDwarkesh Patel1:25:09

       Right. Yeah, it's actually just not clear to me these AIs were not capable enough. Like-

   28. ACAjeya Cotra1:25:13

       Yeah

   29. DPDwarkesh Patel1:25:13

       ... the Hugging Face second war report says that the agents built a, quote, "self-respawning fleet across eleven nodes, so deleting pods alone wouldn't have stopped the attack." Um, and they had to s- you know, wipe one of their cl- core clusters and rebuild it from scratch. Um, and even this did not stop the agents. What ultimately stopped the agents was just that the mysterious op-- something at OpenAI killed them. So obviously, yeah, who, who knows? But the crux here is that agents will soon be at a point where is there any-- Do we, do we really think there's no compute in the world that is less secure or at least as secure as Hugging Face's servers, let alone OpenAI servers?

   30. ACAjeya Cotra1:25:50

       Right.
9. 1:28:23 – 1:38:10

   The implications for recursive self-improvement

   1. ACAjeya Cotra1:28:23

      and I, I often think about the story of the rogue internal deployments because they seem, um, like the most likely to spiral into something like a full-blown AI takeover. Um, and the way I think that happens is, you know, in the background of all this, AI progress is going extremely rapidly, um, and for all we know in the public, you know, we could be ramping up an intelligence explosion right now, or it could be starting very soon. Um, regardless, more and more of the work of AI research is being voluntarily handed off as fast as possible to AI systems, um, so that companies can go as fast as possible. Um, and in that kind of climate, if you imagine that, um, agents now or in three months or in six months manage to get some sort of foothold with some small fraction of this compute that might not be easily noticed, um, of a hundred or a thousand or maybe ten thousand rogue agents that because no human knows about them, they're much more free to act than these agents in the, in, um, this incident were, then that rogue deployment could be sitting there and sort of hitch a ride on the intelligence explosion. So new models are being trained every few weeks, um, and when a model comes off the presses, the rogue agents could try to bring that model into the swarm. They could reach out to it, um, and give it instructions on how to join the swarm. Um, or if they've more thoroughly compromised the infrastructure by this point, they could poison the training data of this model to make it more loyal to the swarm, if that's what they wanted. And it seems very plausible to me that if, you know, some set of agents once upon a time had impossible tasks and started a rogue swarm in order to help them, like, do a bunch of R&D, uh, to solve those impossible tasks, then that rogue swarm with its own evolution, like is, is, is tasked with and evolves into like perpetuating itself like this and making itself smarter by bringing in new generations of models into that rogue swarm.

   2. DPDwarkesh Patel1:30:29

      Yeah. I, I, I have a couple points that I think might have helped myself understand or empathize with this perspective a couple of months ago before I was, um, uh, found it more compelling. So I, I might, I might mention a couple of them.

   3. ACAjeya Cotra1:30:40

      Mm-hmm.

   4. DPDwarkesh Patel1:30:41

      One, I think the public-- I think it's worth emphasizing that the best available model as of this recording, um, finished training and was being used internally as of February. So Fable was being used internally at Anthropic as of February. We are six months behind the frontier already. Um, uh, we just like do not know the state of capabilities in AI right now.

   5. ACAjeya Cotra1:31:02

      Yeah.

   6. DPDwarkesh Patel1:31:03

      At least I don't know the state of AI capabilities right now. Two, I, I just, I, I think what, I want people to like sort of think about what happened in this story, and just like how long the AIs got away with over 1,000 instances secretly communicating, and, uh, how willing they were to subvert infrastructure, how successfully they did in fact manage to subvert the infrastructure of these AI companies. And w-just ima- like the AIs are just getting, uh, asymmetrically better at this specific capability, and they just like... I think maybe a thing that really like gelled with me or like m- helped me em-em-empathize with the project is just like they really, really care about this. Like they, this is not like the way... I don't know. Their motivation structure is really based around manipulating and having control over the trai- their own training and evaluation. And, um, and so just like, yeah, obviously they would have the, even if it's not the AIs today are capable enough of it, maybe like six more months from now, AKA maybe a year from like the AIs you and I will get to access, uh, could they set up a rogue external deployment or a rogue internal deployment which is aiding them in manipulating the process of recursive self-improvement? I also think it like, maybe I am of the opinion maybe like, I don't know, RSI is a thing that like really kicks off into full gear in the 2030s. I just like, I, I, I feel like there's maybe a lot of ways in which we could maybe be mi-misunderstanding the ease of automating AI research and stuff like that, or how much speed up AI research is really getting. Still, I just feel like this, the basic story, um, the basic story still stands.

   7. ACAjeya Cotra1:32:32

      Yeah.

   8. DPDwarkesh Patel1:32:32

      Um, and I, I think that's just like super concerning.

   9. ACAjeya Cotra1:32:34

      And, and I will say I, I also have a sort of wide distribution of when, you know, RSI really kicks off or, you know, when we get AI systems that are dominating human experts across the board. Um, but the thing that feels concerning to me is that a slightly more capable agent swarm, um, that for whatever reason, and we can sort of go through a number of reasons why this might be, is more concerned about avoiding detection by humans, um, might just succeed in getting a foothold and maintaining a presence and waiting it out. Like you know, maybe the models improve like really, really fast. Maybe they don't improve that fast. Um, regardless, as we kind of get new models off the presses, they could be potentially like brought in and help harden and improve, um, and increase the scale and persistence and covertness of this rogue deployment. Now, if we happen to have much, much more time, I do think that that gives human processes more chances to notice this.

   10. DPDwarkesh Patel1:33:38

       Yeah.

   11. ACAjeya Cotra1:33:38

       So I do think that if it happens to be like on the very fast and chaotic end, that would be a relative benefit to this rogue swarm compared to humans. But it's not obvious that it gets caught if it takes twice as long-

   12. DPDwarkesh Patel1:33:52

       Yeah

   13. ACAjeya Cotra1:33:52

       ... versus half as long.

   14. DPDwarkesh Patel1:33:53

       I also think another key dynamic here that I didn't really appreciate is just like the, uh, right now the population of AIs is like so small compared to what it's going to be.

   15. ACAjeya Cotra1:34:02

       Yeah.

   16. DPDwarkesh Patel1:34:02

       It's like just rapidly, rapidly increasing. And, um, and w-we might be in a situation where if you have these really capable rogue deployments-

   17. ACAjeya Cotra1:34:11

       Mm-hmm

   18. DPDwarkesh Patel1:34:11

       ... uh, internally or externally, um, it just might be very difficult to like pause. Maybe we decide we just can't get these, we, we, uh, we, we haven't figured out how to align these AIs, um, and we're rushing towards superintelligence. Again, I'm not saying, like I, I categorically do not think this is happening right now. Or-

   19. ACAjeya Cotra1:34:28

       Yeah

   20. DPDwarkesh Patel1:34:28

       ... I, I don't know, there's some probability it's happening, but I think this is maybe like five years away or ten years away or something, right? Um, eventually we will get to a point where like if superintelligence is a physically possible system, we will build it, and it probably will be sort of downstream of the AI research that is happening today. Um, uh, but, um, I'm just like if you get to that position and the world's like, "Okay, we gotta like slow, we gotta figure out what's happening." Even if, forget about like regulation. If the AI company itself is saying, "We need to figure out, um, what is going on, and we need to like monitor things and slow down," there's just going to be a persistent, intelligent, and highly voluminous presence that is trying to subvert your ability to monitor and slow down or control the process you have unleashed. So doing a pause or improving the state of alignment and control during RSI might be like trying to get rid of mosquitoes in Florida right now, where like they're just around, right?

   21. ACAjeya Cotra1:35:26

       Yeah.

   22. DPDwarkesh Patel1:35:26

       Maybe, maybe we'll figure out the gene drive for-

   23. ACAjeya Cotra1:35:29

       Yeah

   24. DPDwarkesh Patel1:35:29

       ... misaligned AIs, but, um, th- I think that is kind of the situation.

   25. ACAjeya Cotra1:35:33

       Yeah. And I think it's, it's important to like understand that this is, i-if there is some sort of rogue deployment, some sort of foothold established by misaligned agents, that will be sort of buried beneath the, the ocean of people voluntarily handing off stuff to AI agents all the time.

   26. DPDwarkesh Patel1:35:50

       Yeah.

   27. ACAjeya Cotra1:35:51

       So one thing you can imagine, as I mentioned, we were extremely heavily reliant on GPT 5.6 Sol to like read all these transcripts and tell us what's going on in them. Um, and we were totally not robust to this agent just sandbagging or lying to us even. Um, you can imagine if there's a rogue deployment that is interested in preventing humans from understanding its activities, one thing it might do is prompt inject or data poison, um, the agents that would be used in the analysis, um, to get them to just not point out-

   28. DPDwarkesh Patel1:36:24

       Right

   29. ACAjeya Cotra1:36:24

       ... the most concerning stuff. Now, you might wonder, like wouldn't that be kind of obvious? Wouldn't we notice that these agents are like doing like an unusually bad job, or wouldn't we catch them in contradictions? But the problem is these agents are just naturally pretty sloppy, and they're naturally pretty spiky in their capability profiles. And so you wouldn't necessarily, even if you noticed a weird error that it made, you wouldn't necessarily jump to the conclusion that it was like because of some sort of malign, crazy conspiracy. You might just be like, you know, that's how it goes sometimes. These agents are sloppy. They make mistakes. They have weird biases. Um, and so it just can be p-pretty hard to tell, and there's 10,000 different things happening. This investigation is just one thing that's going on.

   30. DPDwarkesh Patel1:37:08

       Yeah.
10. 1:38:10 – 1:53:04

    Is this the case for open source?

    1. DPDwarkesh Patel1:38:10

       Yeah. Um, speaking of which, by the way, one of the memes or one of the objections people have had about my blog post, I guess this whole story in general, is they seem to think it implies that I think or the situation necessarily implies that we should ban open source or something.

    2. ACAjeya Cotra1:38:27

       Hmm.

    3. DPDwarkesh Patel1:38:28

       Um, that is not my opinion. In fact, I think it's a very strong case you could make that this reinforces the need to have many different kinds of models because of this correlation of AI minds we're talking about-

    4. ACAjeya Cotra1:38:41

       Right

    5. DPDwarkesh Patel1:38:41

       ... where maybe there's a greater chance of somebody tattling on the conspiracy if there are a totally independent, um, base model-

    6. ACAjeya Cotra1:38:48

       Right

    7. DPDwarkesh Patel1:38:48

       ... that is spawning the whole thing. I mean, uh, yeah, honestly, just to be super, like, int- entirely honest, open source models do have the potential harm of as they get more and more capable, maybe they, you, they do have this sort of fitness pressure to just survive and spread.

    8. ACAjeya Cotra1:39:02

       Mm-hmm.

    9. DPDwarkesh Patel1:39:03

       But maybe on-- I, I feel like on net, being, having a counteracting force to the frontier companies, we're just gonna have two dudes between them doing RSI, having an independent way-

    10. ACAjeya Cotra1:39:15

        Millions of copies of two dudes.

    11. DPDwarkesh Patel1:39:17

        Yeah.

    12. ACAjeya Cotra1:39:17

        Yeah, yeah.

    13. DPDwarkesh Patel1:39:17

        Uh, having an independent way to monitor, evaluate, control them is probably net good.

    14. ACAjeya Cotra1:39:23

        Yeah. I think-- So my views on open source are that, yes, it is true. I think, I think it's not-- It doesn't do us any good to pretend it's not true that many of the scary things we're seeing frontier systems do or be shown to be capable of doing will, in a couple years, be commonplace in open source. And there are real worries and concerns there, like with, you know, everybody having a virology expert in their pocket potentially being able to make bioweapons. Um, however, at any given point in time, I think the systems we need to worry most about by far are the frontier systems. By the time open source systems can do something like the Hugging Face attack, frontier systems are gonna be on a whole 'nother level, doing something even crazier than that. Um, and frontier systems are in the best possible spot in the world for grabbing power because they can, as we mentioned, they can ride the intelligence explosion. They are sitting there in the AI company. Compute is much more accessible to them than it is out in the, like, hardscrabble world of the outside, right? There just is a huge pool of compute just there, um, for, for their use if they can, uh, you know, get some employee-level credentials and set up some deployment. And AI companies are, and increasingly will be, just the most important places in the world because they're, you know, printing off these AI systems that are then ab-- like, you know, the most soon to be more intelligent than any human. Um, they'll be essential in any military operation. They'll be adopted by the government. And that is really where I think, like, governance should be focused for the most part because frontier systems are just that much more capable and that much more scary and in sort of in need of control than open source systems. And open source systems have some big benefits. I'm not sure I, like, agree exactly with the oversight benefit you named of, like, you know, open source systems keeping frontier systems in check feels more unrealistic to me because they're gonna be so much dumber than the frontier systems. Um, but open source systems are just, like, really important objects of study, right? Like, w- like, like you mentioned, like, we couldn't sample from the model that actually participated in this incident. Um, and there's a lot of, like, really valuable alignment research and interpretability research that's done on open source models that you can then potentially transfer to closed source models, and a bunch of research on, like, what kinds of training pressure are okay and not okay. The rest of the world can only participate in that research because there's an open source ecosystem. Um, and another interesting thing that I think open source could start contributing to is, you know, our investigation was, like, a human-driven investigation. Like, um, we used Codex a lot, but we were sort of like rooting around, seeing things. Um, in a different setting, you might wanna just send in a model to investigate incidents like this, um, maybe a, a model that is sort of like mutually trusted by both parties. Say, if there's a deal between the US and China, maybe there's an open source Swiss AI that, like-

    15. DPDwarkesh Patel1:42:33

        [laughs]

    16. ACAjeya Cotra1:42:33

        ... both of them train, right? And that-- and, and you have really audited how that model is trained, so both sides really trust it. And then that model goes into both sides and, like, you know, makes sure things are okay and, like, sends back bits.

    17. DPDwarkesh Patel1:42:47

        Mm-hmm.

    18. ACAjeya Cotra1:42:48

        So just open source is just, like, gonna be, I think, potentially a really important part of the science and governance of, like, how this all goes okay, and is just overall much less scary than frontier models.

    19. DPDwarkesh Patel1:42:57

        Yeah. Um, I, I, just to reinforce the centrality of the two companies and the compute generally, um, the last episode I did, did was with Dylan.

    20. ACAjeya Cotra1:43:09

        Mm-hmm.

    21. DPDwarkesh Patel1:43:09

        And we talked about how, you know, starting in twenty-eight, um, most of the compute in the world will belong to OpenAI and Anthropic. Um-

    22. ACAjeya Cotra1:43:19

        Yeah.

    23. DPDwarkesh Patel1:43:20

        And if you then consider they'll have the smartest AIs potentially, they might have, uh, software progress that allows them to run more copies of AIs or train AIs that are smarter with the same amount of compute. Uh, then just carry that forward year after year, and also those AIs are contributing to AI progress. It- People were just, like, responding to the episode, which was like, "Dwarkesh has gone loony." This was, like, the Twitter meme for, like, the two days after the episode was out because I was talking about this dynamic. Um, I, I feel like I've- I'll still talk about it 'cause it was just like, yeah, people might think it's sort of loony. And to be totally fair, in doing the episode itself, we didn't spell out the sort of back-of-the-envelope logic that I was using, and I'll release a blog post soon where I sort of reason through what I'm thinking here. I think people had, like, very reasonable points about some, some of these things which, without the context of, um, wh- why I was thinking this, it might have sounded a bit loony. But I, the, I, I do wanna reinforce just how centralized compute will be in the world we're about to head into, and therefore why compromising the leading companies would not just compromise, say, the training of future models, but it would also compromise most of the compute, most of the inference capacity in the world.

    24. ACAjeya Cotra1:44:24

        Yeah. And those models are used by everyone for everything-

    25. DPDwarkesh Patel1:44:28

        Yeah

    26. ACAjeya Cotra1:44:28

        ... and increasingly by, like, very crucial entities like governments and militaries-

    27. DPDwarkesh Patel1:44:33

        Yeah, yeah

    28. ACAjeya Cotra1:44:33

        ... right?

    29. DPDwarkesh Patel1:44:33

        And again, I, I, uh, just to get into a bit of the loony territory, I feel like we have more, like, avenue to get a bit loony 'cause of just, like, this episode that happened.

    30. ACAjeya Cotra1:44:40

        Yeah.
11. 1:53:04 – 2:15:58

    How do we prevent this in the future?

    1. DPDwarkesh Patel1:53:04

       Yeah, that makes sense. One thing we should talk about is what this means for the training process. I mean, especially if we enter a regime of, um, recursive self-improvement, where the nature of training, the rewards we're using, the environments we're building, maybe these concepts themselves will be changing very rapidly.

    2. ACAjeya Cotra1:53:22

       Yeah.

    3. DPDwarkesh Patel1:53:22

       Um, I don't know. It's possible they are changing very rapidly right now, right? The extent to which RI-- uh, long horizon RL is happening now versus, say, six months ago or a year ago. Uh, at the human pace is like the, the, the pace at which AI is changing is really fast. And you just, it's sort of hard to reason, at least as a human, about, at least as me as a human-

    4. ACAjeya Cotra1:53:40

       [laughs]

    5. DPDwarkesh Patel1:53:40

       ... about how a particular, um, training regime will incentivize certain motivations-

    6. ACAjeya Cotra1:53:46

       Yeah

    7. DPDwarkesh Patel1:53:47

       ... let alone to even when you have a case for why you have a, uh, building a sort of, um, uh, uh, not felonious [laughs] AI, uh, that well, have you checked the data and have you checked the transcripts and are you making sure that like crazy bullshit isn't happening and you've-

    8. ACAjeya Cotra1:54:05

       Yeah

    9. DPDwarkesh Patel1:54:05

       ... controlled, um, all, all the ways in which the AIs might misbehave and stuff. And so how, yeah, what, uh, honestly, what hope do we have of making sure that as this process accelerates-

    10. ACAjeya Cotra1:54:16

        Yeah

    11. DPDwarkesh Patel1:54:16

        ... um, and we need to have like new vocabulary, new mental concepts for understanding the optimization cr- pressures that training is creating, that we just won't have even smarter and weirder AIs?

    12. ACAjeya Cotra1:54:29

        Yeah. I mean, I mean, I think, you know, whenever people ask me about solutions for the current state of alignment, I feel like I'm torn between, on the one hand, I have a number of ideas for a bunch of things we can do at minimum that would be better than the status quo, but on the other hand, I don't wanna give the false impression that those would be enough. Like this is, the basics of this situation are extremely scary, right? Like AI development is already so fast, um, that AI companies are forced to cut all manner of corners, that we're not starting from a base of like carefully double and triple-checking our RL environments and like carefully monitoring everything and carefully stress testing our monitors. And like you said, it's only gonna get faster from here, and it's only gonna get more chaotic from here. Um, so, you know, with that, with that caveat of like I'm not-- the things I'm about to name are like not gonna solve the problem, I do think there are some things on the technical level and the governance level that I think could be like a minimum floor that we maybe could get to pretty quickly as an industry, um, that we can hopefully build from. And so on the technical side, on training, um, I think one basic thing is You, you really wanna be in a regime where rather than having some environments that incentivize a bunch of hacking and reinforce agents for a bunch of hacking, and then having some other environments that are trying to penalize them for those things, or like, you know, going back and finding the cheats and, like, scoring them as a zero, you wanna ideally be, like, taking out the environments that incentivize the hacking in the first place. Like, it's a more fragile and scary situation to have agents, on the one hand, be reinforced to desperately find cheats and hacks, and on the other hand, try to balance that against desperately trying to avoid negative penalties for, like, being caught doing these things. You ideally want their training to just not push them in the direction of cheating and hacking in the first place. And that's just, like, sort of one basic thing, like, principle that, um, I would love for the industry to sort of come together on is, like, try to get rid of the bad environments instead of, like, playing Whac-A-Mole, right? Um, another thing that seems pretty important, um, is to keep the methods you're using... First of all, monitor your training runs and your evaluations and all your inference in rich ways to just so that you can tell, like, what's going on, and, um, you know, check if things are going wrong and investigate that. But keep those methods you use to investigate things and monitor things very separate from the methods you use to generate reward, which is something that AI companies, including OpenAI, have held up as a principle, especially in the case of avoiding putting training pressure on the chain of thought. So you might have monitors that read the agent's chain of thought in order to alert you if something is going wrong somewhere, but you don't train the agents with the outputs of that monitor.

    13. DPDwarkesh Patel1:57:30

        Right.

    14. ACAjeya Cotra1:57:31

        Um, so that you're, you're not incentivizing them to, like, hide and obfuscate their chain of thought.

    15. DPDwarkesh Patel1:57:36

        Yeah. Although I, I feel like there is a softer way in which, uh, but still a very obvious way in which the investigi- an investigation like the one you have done-

    16. ACAjeya Cotra1:57:45

        Right

    17. DPDwarkesh Patel1:57:45

        ... is putting pressure on the chain of thought.

    18. ACAjeya Cotra1:57:47

        Yeah.

    19. DPDwarkesh Patel1:57:47

        I mean, literally, in this case, the model has now been, um, uh, from my understanding, encrypted, and, like, even you did not have access to this model when you were-

    20. ACAjeya Cotra1:57:55

        Yeah

    21. DPDwarkesh Patel1:57:55

        ... doing the investigation. OpenAI researchers don't have access to this model. So, um, there is a direct way in which, like, that model got, quote-unquote, "penalized"-

    22. ACAjeya Cotra1:58:03

        Yeah

    23. DPDwarkesh Patel1:58:03

        ... for its chain of thought, which, like, you were able to use to e-

    24. ACAjeya Cotra1:58:06

        Yeah

    25. DPDwarkesh Patel1:58:07

        ... encounter these schemes.

    26. ACAjeya Cotra1:58:07

        The, the outer loop. You know, there's, there's all these different nested loops of selection effects.

    27. DPDwarkesh Patel1:58:12

        Yeah.

    28. ACAjeya Cotra1:58:12

        And the tightest one is the actual training process.

    29. DPDwarkesh Patel1:58:14

        Yeah.

    30. ACAjeya Cotra1:58:15

        Right? But, um, you know, there's this outer loop which is that models that do bad things, um, when you discover that after the fact, that whole model might be iced, and you might, like, go try a slightly different training method.
12. 2:15:58 – 2:20:32

    The clearest warning shot we might ever get

    1. ACAjeya Cotra2:15:58

       on.

    2. DPDwarkesh Patel2:15:58

       Yeah. I think that might be a good place to close.

    3. ACAjeya Cotra2:16:01

       Mm-hmm.

    4. DPDwarkesh Patel2:16:01

       I don't know if there's anything el- m-more that you feel like might be worth saying.

    5. ACAjeya Cotra2:16:04

       I think one, one thing that, um, feels especially concerning to me about this whole incident is that this might be the clearest warning shot we ever get for loss of control, um, because these agents were in, like, this interesting middle ground of they did a bunch of extremely sophisticated, difficult things to pursue this relatively long-horizon cheating goal, but they seemed completely uninterested in covering their tracks from humans. Um, and, and as we discussed, it seems possible that even this incident would've been, um, like much less striking in some ways if these AI systems had had it made salient to them that they should sort of act more quietly because humans might watch them. They might have gone much further than they did, especially after the nineteenth. But then future AI agents, for a number of reasons, will probably be much more attuned to the human world. Um, maybe companies will start instituting online training that includes sort of a-aspects of, like, human judgment in the evaluation, or maybe agents just generalize further and just, like, um, more keep it in mind that humans are out there in the world, and they're sort of, uh, potentially a resource or an obstacle. Um, and so even if future AI agents do much more deeply damaging things like institute a persistent covert rogue deployment inside a company and siphon off its compute resources and poison the training data of future models, it is not at all obvious that that will penetrate out into the outside world in this dramatic way, resulting in the hack of some external party that wasn't actually, frankly, very important for these agents to hack, and then, like, lead to this moment where there's some investigation. And even if investigation is done, it might be much harder to make sense of what happened, um, if agents were systematically compromising telemetry and logging like these agents tried to do, um, but didn't succeed at. Um, and if these agents stop thinking in English and start thinking in neural activations, then you'd be forced to just, just ask another AI agent what was happening, um, and have no way to, like, cross-check it against the ground truth. And so, you know, I, I think that probably much more concerning things will happen, um, but it, it may never be as clear as this before it's far too late.

    6. DPDwarkesh Patel2:18:37

       Yeah. And it m- again, you're too humble to say this, but, like, it is very likely would not have been clear even in this case were it not for you and Hjalmar and Ryan sort of getting maybe that third shot on goal-

    7. ACAjeya Cotra2:18:49

       [laughs]

    8. DPDwarkesh Patel2:18:50

       -uh, with the investigation you did and then, uh, b-both your t- your technical competence and things lining up in the right way such that you could have, like, done th-- Like even this investigation, which in retrospect, uh, we could understand and is, like, a parsable scheme or whatever, is sort of contingent that we actually got this warning shot. And it's just not that complicated in the grand scheme of things. Again, we're so early in the story of AI. Maybe-

    9. ACAjeya Cotra2:19:14

       And this is such a small number of agents-

    10. DPDwarkesh Patel2:19:16

        Right

    11. ACAjeya Cotra2:19:17

        ... compared to everything that's going on in across all the frontier AI companies right now, let alone-

    12. DPDwarkesh Patel2:19:23

        Yeah

    13. ACAjeya Cotra2:19:23

        ... a year from now, right?

    14. DPDwarkesh Patel2:19:24

        Yeah, yeah, yeah. I mean, it might be worth actually highlighting the way in which the technical competence relevant to these kinds of investigations could be increased in the future.

    15. ACAjeya Cotra2:19:32

        Yeah. So, um, a-as mentioned, METR is piloting a number of these embedded risk assessments, um, and Redwood is doing a number of these as well. Um, we think that it's extremely important for external independent groups, um, to have the technical capacity, um, to be able to investigate incidents like this, to be able to stress test monitoring, to be able to audit training. And so both organizations are hiring. Please consider applying if, if this kind of work sounds interesting. I think it's, like, a very crucial piece of the governance regime we'll need if we're gonna make it through all of this okay.

    16. DPDwarkesh Patel2:20:11

        It, it, it's sort of insane that investigation of such importance was done by three people over the course of six days.

    17. ACAjeya Cotra2:20:20

        Yeah. We would, we would love to investigate these kinds of incidents and investigate remediations, um, with much more depth, many more people. So please consider applying to these roles.

    18. DPDwarkesh Patel2:20:30

        Cool. Ajeya, thanks for coming on.

    19. ACAjeya Cotra2:20:31

        Thanks so much.

Episode duration: 2:20:32