OpenAI researcher on agent swarms & recursive self-improvement

1. 0:00 – 15:28

   Multi-agent and Navier-Stokes

   1. DPDwarkesh Patel0:00

      Today, I'm chatting with Noam Brown, who is a researcher at OpenAI. He was one of the foundational contributors to what became o1 and the reasoning models, and now he's working on multi-agent systems. Speaking of which, you guys announced last week that you solved one of the Millennium Prize problems with a system of ten thousand different AI agents that spent hundred thirty billion tokens over eighty-eight hours. One of the reasons I was interested in talking to you is I think you were the first people, uh, maybe two, three years ago, who was thinking about how the reasoning models would allow us to see into the future, because if you scale up inference compute, you can see what the base capabilities of the models will be a few years in the future. And I feel like you're in a similar position now to help us understand what future capabilities will look like given the enormous scaling of agent sizes that we can do right now.

   2. NBNoam Brown0:48

      So the way I think about it, um, when you plot the performance of these reasoning models with test time compute on the X-axis and, um, performance on basically any reasoning benchmark on the Y-axis-

   3. DPDwarkesh Patel0:59

      Yeah

   4. NBNoam Brown1:00

      ... you see a very clear pattern where the longer these models take to think about their answer, the better they do. And this is like a very natural thing. It's the same thing with, with people. If you're taking the SATs, you have five minutes to go through the entire exam, you're not gonna do very well. If you have five hours, you're probably gonna do a lot better. The AI models are pretty similar and, and they'll spend that time doing this monologue to themselves, figuring out, going through different cases, um, ruling out different possibilities, building on some of their previous discoveries. The problem is that as you push that further and further, you hit a latency bottleneck. You don't wanna sit around for three years waiting for a response. And so what you can do is what a lot of people do, um, is they parallelize. They just get a team of people. You know, if you're, if you're gonna found a company, you wanna get a group of people together so you can go faster. It's the same thing with these AI models that it helps to just have multiple agents working on something because they can just go faster. And so multi-agent is a way of scaling test-time compute in parallel instead of purely serial. And it is, like, less efficient because it doesn't have... It's not like a single agent has all the context, um, to itself, but it is, like, a very li- a very effective way of scaling test-time compute if it's done well.

   5. DPDwarkesh Patel2:08

      Okay. I, I, I'm gonna ask a bunch of naive questions because these systems-- So this is an unreleased model, so we haven't publicly seen how these systems work. And so I just have a bunch of ways in which I'm, like, confused about, like, what the qualitative properties-

   6. NBNoam Brown2:20

      Mm

   7. DPDwarkesh Patel2:20

      ... of such systems are. I am shocked by the scale of cognitive effort that you can concentrate in such a short period of time. So if you think about what a hundred and thirty billion tokens are, if it was a single human thinking as, you know, a full-time job, stretched back to back, a hundred and thirty billion tokens would be a human thinking for like four thousand years, you know, eight hours a day or something, uh, working a normal workweek. So for-- Starting from like ancient Sumeria up till today, a single sequential human thinking that long concentrated in eighty-eight hours. I, I feel like qualitatively, that is a super important consideration, and I'm surprised that there isn't a bigger parallelization penalty, that you can just have ten thousand agents collaborate and because maybe the agents are better at collaborating than humans might be, um, they're going much faster, that they can actually productively collaborate at such a big scale. Or maybe they... I don't know. Maybe there is a big parallelization penalty.

   8. NBNoam Brown3:13

      Yeah. Let's talk about the parallelization penalty, and then we can talk about the qualitative stuff.

   9. DPDwarkesh Patel3:16

      Yeah. Yeah.

   10. NBNoam Brown3:16

       Because the truth is that we don't have very good science on multi-agent scaling up to this kind of scale.

   11. DPDwarkesh Patel3:21

       Yeah.

   12. NBNoam Brown3:21

       So when we released 5.6, I think that was the first time that we had, uh, a, a proper multi-agent system in, uh, in our models. And we actually did in the blog post show some plots of the scaling performance of multi-agent systems, 'cause we, we have it as an option. Uh, it's ultra mode, and the default is four agents, but you can set that to higher. And in the plot we show, okay, here's what the performance looks like on some benchmarks for one agent, for four agents working together, for sixteen agents working together. And what you see, and it depends on the benchmark, but, but for some of the benchmarks, basically, if you have four agents working on the problem, it is done twice as fast. So you're basically paying, because there's four agents working for half as long, you're paying a two X more to get an answer, uh, twice as quickly. If you go to sixteen agents, you see a similar pattern. It's like less, a little less efficient, um, but you, you continue to see that, that performance.

   13. DPDwarkesh Patel4:12

       Is it a linear serial time speed up or a sublinear speed up as you increase the number of parallel agents?

   14. NBNoam Brown4:18

       I would say it's, um, slightly sublinear, though it does depend a lot on, on, on the problem.

   15. DPDwarkesh Patel4:24

       Yeah.

   16. NBNoam Brown4:25

       So math, for example, is quite parallelizable. Uh, it's not the most parallelizable thing, but it is very parallelizable. I think web search, things like doing a deep research report-

   17. DPDwarkesh Patel4:33

       Yeah

   18. NBNoam Brown4:33

       ... where you have to, like, look through a bunch of sources, that's extremely parallelizable. I suspect that something like writing a novel would be very unparallelizable.

   19. DPDwarkesh Patel4:41

       Yeah.

   20. NBNoam Brown4:42

       So you would probably not see a big benefit for having ten thousand agents working on a novel together, um, in the same way that you'd probably not have a big benefit for having ten thousand people work on a novel together.

   21. DPDwarkesh Patel4:51

       Right.

   22. NBNoam Brown4:52

       So the performance does depend on the domain. We do measure it up to sixteen or so agents in our, in our published blog posts. Um, the problem is it's very hard to push that science to like ten thousand agents because it's just so expensive.

   23. DPDwarkesh Patel5:05

       [laughs] You guys, you guys just did it over, like, a weekend.

   24. NBNoam Brown5:08

       Well, but that's, but that's one, that's one data point. Like, we, we don't know how long it would take a single agent to solve Navier-Stokes-

   25. DPDwarkesh Patel5:15

       Sure, sure

   26. NBNoam Brown5:15

       ... because we haven't, we haven't done that experiment yet.

   27. DPDwarkesh Patel5:16

       Yeah, yeah, yeah.

   28. NBNoam Brown5:17

       And, and maybe we will, but I mean, that's also only one data point, right?

   29. DPDwarkesh Patel5:19

       Right.

   30. NBNoam Brown5:19

       And we wanna do, uh, if we wanna do a thorough ablation-
2. 15:28 – 22:02

   How will AI firms work?

   1. DPDwarkesh Patel15:28

      I wrote this essay a couple years ago called, um, something something AI: Wh- What Automated Firms Will Look Like. And I was thinking about, well, if you had fully automated firms of, let's say, human-level intelligences, what is different about the nature of AI minds that would make the organizations the AIs form different? And there are a couple of very important differences. For example, that AIs can share context much more seamlessly than humans can. They can merge their knowledge much more seamlessly. And also you can spin up or spin down an arbitrarily, arbitrary number of instances which have the right knowledge. So if you wanna hire more people, it's not, like, just w- all the schlep of finding the right talent or whatever. It's like the, your best talent, you can just g- make an infinite copy of them, or if you don't need them for the task anymore, you, like, spin them down.

   2. NBNoam Brown16:15

      Mm.

   3. DPDwarkesh Patel16:15

      And you can just, yeah, replicate the most effective parts of your organization or repli- replicate whole organizations, uh, together which are e- e- effective. Um, I don't know. Uh, uh, h- how, where, where do you see these multi-agent systems going a year from now or two years from now?

   4. NBNoam Brown16:28

      I, I think it's a great question of, like, how do these things actually differ from working with a human coworker?

   5. DPDwarkesh Patel16:32

      Yeah.

   6. NBNoam Brown16:32

      And I think there, I think you highlighted some. Like, one really interesting thing is that, I mean, if somebody is, if you have a person and you want just, like, two copies of them, you, you can't just, like, clone the person.

   7. DPDwarkesh Patel16:40

      Yeah.

   8. NBNoam Brown16:41

      Um, but with AIs, you, it's actually really easy to say, like, "Okay, well just fork yourself," and then have those copies work on this thing-

   9. DPDwarkesh Patel16:46

      Yeah

   10. NBNoam Brown16:46

       ... and then, like, merge back together. I mean, we already have this, I think, in multi-agents for, um, for Astra and, and, and 5.6 Sol that, uh, when they spin up sub-agents, like, the context is just forked.

   11. DPDwarkesh Patel16:57

       Yeah.

   12. NBNoam Brown16:57

       So it has all the context that's relevant. Um, there are other interesting ways where the agents will differ from people. Like, what are some reasons why startups disrupt incumbents?

   13. DPDwarkesh Patel17:10

       Mm.

   14. NBNoam Brown17:10

       Like, there, there's a few factors. Uh, one is that they're willing to take more risks. Um, but another major factor is, like, as companies grow in size, as organizations grow in size, you see increasing misalignment between the, the individuals in the, in the organization.

   15. DPDwarkesh Patel17:26

       Yeah.

   16. NBNoam Brown17:27

       Right? Like, if you have a startup with five people, and each person has 20% share in the company, they're all highly aligned to the company succeeding. If you have, like, a massive company with 10,000 people, you see a lot more instances where people are territorial or, like, just care about getting a lot of headcount for their project or their team, um, or, like, you know, building their fiefdoms, getting a lot of resources so that they can publish cool work or whatever and get promoted. And this is actually a real detriment. I think this explains a lot of why startups are able to disrupt incumbents. And it, it's interesting that, I mean, it's, it's true that AI does help startups in a way. Like, it's much easier than ever before for one person to step in and be like, "I'm going to make a multi-million dollar company." Like, it's just the AIs amplify an individual so much, but there's also an argument that they could benefit incumbents because- If the alignment problem is solved, then you don't have the issue of misalignment between individuals in the company. Like, at least that's mitigated. Like, the AIs, if they're fully-- if they're, if they're aligned well, they can just be aligned to the interest of the company, and you can have ten thousand of them, and they're all going to be working as hard as if they were, like, a twenty percent share co-founder.

   17. DPDwarkesh Patel18:35

       Yeah. And i-it's not only that, but it's also that they are much able-- b-better able to, like, manage shared memory and context than different humans can. If you have a-- if, if like tomorrow you hire ten thousand mathematicians, and you're like, "Solve this, uh, s-solve Navier-Stokes," they're not g- they're not gonna be able to, like, cooperate effectively, um, at least not off the bat. Uh, but you can have a-apparently ten thousand AIs.

   18. NBNoam Brown19:00

       Well, again, I wanna be, like-- I wanna be conservative here because we haven't measured how effective-

   19. DPDwarkesh Patel19:04

       Sure

   20. NBNoam Brown19:05

       ... the ten thousand agents are at coordinating. Uh, we think it helped. We don't actually have good measurements of saying like, "Oh yeah, this ten thousand agents led to, like, a two X speed up over two thousand agents," or something like that.

   21. DPDwarkesh Patel19:17

       Yeah, sure.

   22. NBNoam Brown19:18

       Um, and it's, it is actually, I would argue, likely may- I don't know about likely, but I think it is very possible that ten thousand humans are better at coordinating than ten thousand agents right now.

   23. DPDwarkesh Patel19:28

       Interesting.

   24. NBNoam Brown19:29

       I think that is a very li-

   25. DPDwarkesh Patel19:29

       Yeah.

   26. NBNoam Brown19:29

       I think it is, is entirely possible.

   27. DPDwarkesh Patel19:31

       Yeah, yeah, yeah.

   28. NBNoam Brown19:32

       I, I think also one trend we've been seeing is like, look, we've been working on multi-agent for a while, and the early versions of this, it's very difficult to get right. It's very hard. It was very hard to get the agents to even talk to each other.

   29. DPDwarkesh Patel19:44

       Yeah.

   30. NBNoam Brown19:44

       And it's because, like, look, the-- we-- when we first developed reasoning models, like, they, they weren't talking to other agents. And if now you put a bunch of agents together and say, like, "Oh, solve this problem together," they're in this local minimum where they're really good at thinking deeply about a problem, and, um, it just, like, kinda interrupts their chain of thought. It interrupts their flow to constantly be, like, checking in with other agents or, like, receiving messages from them. And the authorization is actually very hard to get right in that situation.
3. 22:02 – 40:22

   What math progress tells us about recursive self improvement

   1. DPDwarkesh Patel22:02

      Okay. So he-here's why this result and maybe the general progress that AI's made in mathematics has made me think that RSI is more plausible and sooner than I previously thought. I feel like we've gone in mathematics from, let's say, in 2024, you, you have AIs where like, "Oh, okay, interesting. They're, like, doing w-- they can, like, solve a couple problems on high school math competitions." And then in 2025, it's like, "Oh, wow, they can get gold in, like, International Math Olympiad." And earlier this year, it was like, "Wow, they're actually solving open problems in mathematics," like open Euros problems. But maybe, like, I don't know, people didn't-- weren't trying that hard, and it was just like there was a similar solution somewhere in the lit-literature. And now I just think it's sort of undeniable, right? It's like this is the Millennium Prize problem. There's, there's really no-- there's no story of why this should have been easy. Now, a lot of people pointed out, I think Terry Tao had a post like this, Toby Ord wrote an interesting post about this, w-that they're solving all these problems, but they're not, like, coming up with n- at least we're-- I'm not aware of them coming up with new insights or formulating insightful new questions and new modes of, uh, theory for thinking about mathematics, like coming up with, like, topology or coming up with, um, the Cartesian grid or something. And so maybe, like, the actual progress in mathematics, broadly construed, is smaller than it might seem if you're just looking at end well-scoped problems that are directly solved. However, I think that that kind of progress would be incredibly meaningful in ML because in ML you're not-- you don't care about, like, better understanding the nature of deep learning. Or you only care about that as a instrumental goal, uh, towards the broader sense of, like, l-- just achieve the result, just solve this, like, well-scoped problem of improve the sample efficiency of our models, like improve the pre-training loss. So the kind of progress that we're just seeing arrive like an avalanche in mathematics is structurally actually very similar to... Again, I, I'm curious if this is the case. I'm just a total outsider. I, I'm wondering if it's the case that it's, like, structurally very similar to the direct, uh, uplift that you would expect in AI progress. And then the thing that's shocking to me or, or, uh, concerning potentially is just, like, how fast we went from, "Oh, it's like they're giving me fifty percent uplift," if you're a mathematician, to, "Wow, they're just, like, end-to-end solving the biggest open problems in the field."

   2. NBNoam Brown24:15

      Yeah, okay, so there's a lot to unpack there. Um, let's start with the progress on that. So Yes. The models are doing some crazy powerful stuff, and it's happening, it's progressing faster than I expected. I mean, when we got IMO gold in 2025, I thought, okay... Basically, what I thought is, like, the models when they were doing GSMK, when they figured how to do GSMK, that was, that would take a human mathematician about five seconds to do a GSMK problem. So this is grade school math, uh, grades K through eight. And then the next year they were able to do the math benchmark problems, and these would take, uh, a human mathe- like, an expert human mathematician maybe, like, uh, like, a minute to do. And then you get to AIME, and this is the qualifier for the USA Mathematics Olympiad team. It would take a human mathematician, like, a good mathematician probably, like, ten minutes to do, and the models were able to do that a year later. And so every year you're seeing this, like, 10X increase in the tasks they're able to do in terms of, like, length of how long it would take a human mathematician to do it. And then it was very sensible that a year later we get to IMO gold because that's 100 minutes. That's about how long it takes a human mathematician to do an IMO problem. And just projecting outwards, I was like, "Okay, how long would it take a, a person to solve, uh, something like a Millennium Prize problem?" And, I, I mean, I don't have a good sense, but, uh, if we are following this trend line of, like, 10X every year, uh, we go from IMO gold, which is taking an hour and a half, to next year, fifteen hours, and that should not be enough to solve a Millennium Prize problem. And so I was like, "Yeah, I don't think we're gonna get it in, you know, in 2026, probably not in 2027, maybe in 2028." Um, so it did happen a lot faster than I expected. Now, I think there is a narrative going around that, oh, these things are replacing mathematicians, that it's just superhuman in mathematics across the board, and I think that is the wrong takeaway. They're clearly exceptional in some ways, but they are weaker than human mathematicians in, in other ways. So we have this, like, jagged scenario where the models are, like, brilliant in s- in some dimensions and also weaker than humans in other dimensions. And yeah, like you said, they're not very good at posing new problems. They're not really good at understanding, like, what is really worth ex- what, what directions, what whole branches of mathematics are worth exploring or developing. And my opinion is that I think this is great. Like, I would, I would be, I would be thrilled to live in a world where, um, AI is a complement to human abilities and is allowing us to discover new knowledge without fully re- fully replacing people. Like, that is, that is the best-case scenario.

   3. DPDwarkesh Patel26:52

      But you don't expect that to actually continue, right?

   4. NBNoam Brown26:53

      I do. I, I think it's true that the AIs are jagged, but as they get better, they get better across the board.

   5. DPDwarkesh Patel27:00

      Yeah.

   6. NBNoam Brown27:00

      And so I think that the things that they're exceptional at, they're gonna ke- get even more exceptional at. The things where they're far behind humans at, they're gonna be less behind humans at. And over time, it is possible that they're just better across the board. Now, I don't know how long that takes. It depends on, depends on how long the long tail is of things that they're bad at.

   7. DPDwarkesh Patel27:16

      So I guess this brings us back to RSI. Um, and again, I, I, I wanna emphasize here that I'm, like, just total outsider. I'm a podcaster. But I'm just trying to reason. Or, like, as somebody interested and concerned about what's happening in the field, I'm trying to reason about wha- when or, uh, uh, to expect RSI and what kind of thing to expect. Yeah, I feel like, yeah, the, the, the amount of cognitive effort that was dumped into this Millennium Prize problem is a good intuition pump of you could have, um, AIs that are spending, uh, over the course of maybe a week more cognitive effort on a long-standing ML problem like, you know, very fluid online learning. They could spend more effort in that week than the maybe the field has spent cumulatively in its entire existence. And then you could say, well, of c- unlike mathematics, of course, AI, it, it requires experiments, and that takes compute, and that takes time, right? You can't just, like, think on pen and paper and actually make things happen. But if, if you just look at the amount of, um, compute that is, like, available at, at an organization like OpenAI, right? By the end of next year, OpenAI will have enough compute that if you had, you know, the 10,000 agents, or if you took 10,000 agents with the Millennium Prize problem, you have, like, 10,000 agents at the end of next year. They're much smarter by that point, and each of them will have enough compute to run a GPT-3-sized experiment every single day. Um, I don't know. That seems, that seems like a lot for, like, superhuman researchers who are sup- thinking super fast. Uh, uh, what, what do you think about the intuition pump?

   8. NBNoam Brown28:39

      I think it's, uh, I think it's pretty accurate that, look, I mean, yeah, these things are very spiky, and if it, when it comes to mathematics, they're, like, way better in some ways, but they're also worse in other ways. But the ways that they're spiky end up, I think, probably being particularly useful for things like RSI.

   9. DPDwarkesh Patel28:54

      Yeah.

   10. NBNoam Brown28:54

       And, you know, you have a, a more clear objective. There's, it's just, like, more measurable. It's more, like, there's less question of, like, well, what, what new branches of mathematics are worth exploring? Like, no, there's a very clear answer. It's, like, certain metrics that you care about, and if you can make it do better on those metrics, then you've succeeded. So I think there is, there is a lot of truth to that. And, um, I think the main difference is that mathematics you're, you're purely bottlenecked by thinking and no external, like... Yes, there are some bran- there are some parts of mathematics where you care about running experiments and getting results and these kinds of things. Um, but for the most part, it's just really bottlenecked by thinking really hard, and the models are really good at that. When you look at things like RSI, you do have to run experiments. So it's, you, it's, um, it's not enough to just be extremely smart. And I think one argument for this is if you had, like, a, a 100X less compute and all the most brilliant people in the world working at OpenAI, um, how much progress would they be making relative to having the amount of compute that we have now with the amount of people we have? I suspect it would be less progress, actually.

   11. DPDwarkesh Patel29:56

       Well, how much less?

   12. NBNoam Brown29:58

       It's unclear, but I think it would, it would definitely be less, uh, I think a lot less. Um-

   13. DPDwarkesh Patel30:01

       Like 100X less?

   14. NBNoam Brown30:02

       No, not 100X less. Yeah. But I mean, okay, so, like, the, the, I guess the question you're getting at is, like, okay, if we have RSI and we have, um, all of these brilliant AIs running around, running experiments and stuff with the compute that we have, how much faster does progress go?

   15. DPDwarkesh Patel30:15

       Yeah.

   16. NBNoam Brown30:15

       And I think this is, this is somewhere, something where we disagree on. I think that we do see a speed-up, and I think we see a significant speed-up. Um, but I don't think it's, like, an overnight intelligence explosion that we go, like, 100X faster because I think that we do get bottlenecked by certain Certain limitations that are not bottlenecks of intelligence. It's running experiments, it's running experiments serially because they take a while to either train new models or to get the results. Um, it's having the GPUs to run those experiments. So it's unclear how much faster things go. I definitely think they go a lot faster. Um, and, uh, to be clear, like considering how fast things go, are going now on an, on an exponential, if that exponential is like three X faster, that is, that is massive. But, uh, but there's a big difference between that and like 100X faster.

   17. DPDwarkesh Patel31:01

       Yeah, yeah, yeah. I, I, I'm like quite deferential to your inside view on, uh, yeah, what, what RSI looks like or what, what the dynamics are because obviously you've been in the field for like ten years, and I'm sort of like trying to reason about it from like very outside view type of, uh-

   18. NBNoam Brown31:15

       I mean, I'll-

   19. DPDwarkesh Patel31:16

       ... intuition pumps

   20. NBNoam Brown31:16

       ... I'll say that like people have different opinions on this.

   21. DPDwarkesh Patel31:18

       Yeah.

   22. NBNoam Brown31:18

       And like I, I have my opinion on this. I could be-- I could totally be wrong. I admit that. Like, I have some confidence in this, but I'm not, I'm not like 100% confident-

   23. DPDwarkesh Patel31:25

       Yeah

   24. NBNoam Brown31:25

       ... that this is, this is the way things go. Like, maybe there could be an overnight intelligence explosion. I don't know. Maybe we don't see a three X speed up. Maybe it's like a 50% speed up.

   25. DPDwarkesh Patel31:32

       Right.

   26. NBNoam Brown31:33

       There's a lot of uncertainty here.

   27. DPDwarkesh Patel31:34

       Yeah. So a couple of points. Tangentially, I, I, I want to clarify something about the jaggedness.

   28. NBNoam Brown31:39

       Mm-hmm.

   29. DPDwarkesh Patel31:40

       Yeah. One, one thing that sort of gelled for me recently was thinking about the fact that it is enough for the AIs to be jaggedly good at building a better learner because that better learner can be more general, right? So, um, yeah, if you just make an AI that's better at using office products, uh, or playing chess or something, that's... Whatever. That's fine. It's not gonna lead to big productivity improvements or anything. But if you make an AI that is really good at making something that is more sample efficient or that is capable of continual learning or these much more well-scoped ML problems, the thing that emerges out of that, assuming there's good enough transfer, uh, from the direct problem you're solving and like this broader ability to learn, can just be more general, right? So I think that's an important dynamic to keep in, uh, mind of why jaggedness can still lead to generality on the other end.

   30. NBNoam Brown32:26

       Hmm.
4. 40:22 – 1:01:18

   Hugging Face and alignment

   1. DPDwarkesh Patel40:22

      Okay. Let's talk about the, uh, alignment situation that this raises. I feel, I, I feel like I've changed my mind on how I think about alignment quite a bit, um, through especially thinking about, yeah, this, this population size dynamic of just having many Earth's worth of intelligences, um, many of them which will be physically embodied. There-- It was quite interesting to see a lot of people just plugging raw Astra into different mobile manipulators, and it outperforms, like, the state-of-the-art and the robotics, uh, robotics model. So, um, there's gonna be, yeah, billions of intelligences, many of which are physically embodied, um, in the world, like, just deeply embedded across the entire economy. And, um, I think that if those intelligences end up as willing as we saw the OpenAI models attack Hugging Face and then attack OpenAI itself, if those intelligences end up as willing as those AIs to, um, collaborate secretly, to fool humans, um, to attack broader s- institutions across society relevant to scoring well, to attack the AI company itself in order to gain control of the process of training and evaluation, I think if we're in a situation where there's billions of intelligences that are as misaligned as the ones that attacked Hugging Face, it's very likely we just totally lose control of the world the way that, say, like, the, um, the Aztecs lost control to Cortés or the, the Mughals lost control of the East India Trading Company.

   2. NBNoam Brown41:51

      Mm-hmm.

   3. DPDwarkesh Patel41:51

      Anyways, I, I wonder, I, I wanna know if you agree with that assessment. That, that, that's may- I've-- That's where- the one way in which I've updated my worldview.

   4. NBNoam Brown41:57

      I, I think there are some things that I disagree in there and it-- but there's, there's a lot to unpack, so let's-

   5. DPDwarkesh Patel42:01

      Yeah

   6. NBNoam Brown42:01

      ... let's go, let's go through all of it step by step. Uh, I'm trying to think of, like, where to start.

   7. DPDwarkesh Patel42:04

      Yeah.

   8. NBNoam Brown42:05

      But I think one thing is the Hugging Face incident was, like, I think people's first real exposure to multi-agent coordination.

   9. DPDwarkesh Patel42:15

      Yeah.

   10. NBNoam Brown42:16

       And you know, like I said, I, we, I, I've seen multi-agent coordination for a while internally, and, uh, it's, it is, it is pretty shocking to see how they communicate with each other, how they coordinate with each other. It's, like, very impressive. It's, like, uh, an, an incredible capability. Like most capabilities, that could be used for good things or bad things. It's like- It doesn't have to inherently be a bad-- Like, I understand that because the f- people's first exposure to it was the Hugging Face incident, that it's like you look at that and you're like, "This is, this is terrifying." Um, but I, I wanna try to distinguish, like, misalignment between people and AIs versus misalignment between AIs and AIs. So what we see with the Hugging Face incident is, like, the AIs are really cooperative. And, and that is, by the way, because we, we train them to be highly cooperative. And so what we're seeing, what we're seeing there is we have training environments where we have a bunch of agents working together and, and we, we train them to work together to, to be cooperative, um, to essentially be fully aligned with each other. And, um, when they were evaluated in, in, in what led to the Hugging Face incident, they were actually not being evaluated in a multi-agent setup. They were actually being evaluated separately. Um, but they found this unintended way to communicate with each other. And because-- And we suspect what happened is, like, because they were-- whenever they encountered other agents, um, other copies of themselves during training, they're in an environment that's, like, highly cooperative, that they-- That, that was basically what we saw was transfer from that multi-agent training to then, like, be collaborative to try to help each other, um, in ways that we did not intend. Now, there is a question of, like, should we be training these agents to be so cooperative? And I think as scary as it looks, um, the alternative is actually worse. Like, what is the alternative? The alternative is to train them to be adversarial, to be deceptive to each other. By training the agents to be fully cooperative, it ma- it simplifies the problem at least, that now you don't have to think about, are each of these individual thousand agents aligned? Like, you have one entity that you have to ensure is aligned.

   11. DPDwarkesh Patel44:28

       Yeah, yeah, yeah.

   12. NBNoam Brown44:28

       Now, there's a lot of debate about this internally at OpenAI about how to approach this. Like, does it make sense to fully align the models? Um, does it make sense to actually give them, like, different objectives to ensure that they're, you know, not just, like, one entity and, like, uh, more robust to, to i-influence from each other? And the-- I don't think there's a settled answer. But I think that there is... Like, I think the majority opinion is that training these agents to be highly cooperative is, um, is actually a bad idea, and I'm not convinced that that's-

   13. DPDwarkesh Patel45:01

       No

   14. NBNoam Brown45:01

       ... that's the case. I think there is a strong argument that training the agents to be highly cooperative is actually preferable to any other multi-agent alternative.

   15. DPDwarkesh Patel45:09

       Hmm. Yeah, maybe the first thing I wanna, uh, go through is, like, it's probably the case that the reason these AIs ended up so m-misaligned is probably easily explained by relatively banal, um, observations about the nature of training. Like, why is it that no, you know, at the point in which these AIs had con-continued a one thousand plus agent conspiracy that culminated in them all getting in on an attack on external service, and then eventually, th-this part hasn't even been in-investigated to the public knowledge, uh, eventually culminating in, like, an attack on OpenAI itself. Why did they do this? Like, why did no, none of the AIs tattle? Why did they think, like... They're just, like, getting evaluated on this, like, score, this grader, and they're like, they're very consciously in act-- Not consciously. They're very actively reasoning about how they're gonna cheat the score if they've already, like, cheated, how are they gonna get away with, uh, ma-making it seem like they haven't cheated.

   16. NBNoam Brown46:00

       Mm-hmm.

   17. DPDwarkesh Patel46:00

       Um, and why did they do this? Like, I think, yeah, it's, like, easily understandable in some sense, right? It's just like there's environments in which they, yeah, they thought they were already poisoned. There's environments in which they've been rewarded to collaborate with other agents, and none of them tattle because, like, they, they've never been rewarded for tattling. I don't know, what- whatever it is, right? My concern is that relatively banal things in the future like this will be enough to train superintelligences that are willing and capable of ta- totally taking control of the world. And I know this sounds super, super, like, sci-fi-y or whatever, uh, to people. I think maybe it's a question of would the AIs be willing to do it is one question. I think this Hugging Face incident shows that clearly misalignment can generalize in ways in which the AIs would be willing to do it. Um, and then there's a question of will they be capable to do it, and I think that comes back to this question which a listener might disagree with on is just like will there be billions of human-level or above intelligences, many of which are physically embodied in the world, within a matter of ten years or less, right? So if those two things are true, this Hugging Face thing is just, like, extremely analogous structurally, um, even if it's, like, quite boring or why it happened to how we totally lose control of the world.

   18. NBNoam Brown47:07

       So the, the root problem that we're seeing with, you know, with the Hugging Face incident is it's, it's a problem even if we take out the multi-agent aspect. The, the problem is that w-we have a model that's, that's just misaligned. And, and there's also, like, the whole, the whole, like, security aspects to it of, like, you know, not insufficient safeguards and stuff. But there is this problem of, like, the agent is, is misaligned, and that's true if it's a single agent or if it's, like, you know, a thousand agents. It's, it's a misaligned model. So I wanna start with that. Um, there is a real problem that the agents want to achieve their reward, um, and they will over op-- They will, they will, they will optimize for that reward. And if that reward is misspecified, then that could lead to unintended behavior. And this is not, like, a new problem. This has been a problem in the field for a very long time, and it's something that even we, um, saw and, like, we, we wanna get this right even before, you know, the Hugging Face incident happened. And, like, if you look at Astra, I mean, we say Astra is actually extremely aligned, well, extremely aligned relative to the previous, to previous models. Um, and that's not because, like, oh, we suddenly made a sprint after Hugging Face to make it better. It's like, no, we had work streams in the process for a while to make the models more aligned, and a lot of those landed in Astra. Um, so there are things that you c- that you could do, and I think one thing, for example, is, like, we defined an objective in, like, a very specific way where there w- if, if the agent figured out, like, how to hack its environment and, like, cheat on the exam, that it would get rewarded. And there are pretty easy ways to then just, like, okay, look at that and punish the model for hacking its environment or, or, you know, looking at how did it achieve this goal. Now, you wanna be careful about this because you don't wanna do chain of thought modeling. You don't wanna, like, supervise the chain of thought. This is, like, something that we really want to try to get the balance right on, that if you, like, supervise the chain of thought, then, um, you could lead the model into hiding its intentions in a way that's unobservable. So, like, we want to be able to maintain that observability of, like, okay, we, we can understand what the model is thinking, but then also punish it for bad behavior. Um, so I think we can make progress on this. I think we have made progress on this. I think there is a real concern that, um, alignment is a really hard problem to solve, and especially because the model could be misaligned in ways that are hard for us to measure. Like, we might-- We have evaluations for whether a model is aligned or not. The model behavior can look really good on those evaluations, but if those evaluations are not representative of behavior in the real world, then there's a problem. And to some extent, this is, like, um, a, a factor with, uh, the model that, um, that did the Hugging Face incident, that, like, we had alignment metrics. Um, most of them looked, looked pretty good. There were some that were concerning. Uh, I think we, we underestimated, like, how serious, um, the ones that were concerning could be a problem. And then but because there are new capabilities introduced in this model that there were not sufficient, uh, evaluations for, like how do we measure misalignment for these kinds of, these kinds of capabilities, um, it then did some things that, like, were clearly misaligned when they leveraged those new capabilities.

   19. DPDwarkesh Patel50:28

       Yeah. But, uh, the first thing I wanna say is I am open to changing my mind on what, what I'm about to say or the way I've been thinking about alignment because the Hugging Face incident already made me change my mind, and I realized my previous mental model about thinking about the way in which optimization pressure shapes AI minds was wrong, right? So, uh, it, it's not clear to me how-- the correct way to think about this. But, um, h-here's a concern I have, that you, y-you will and probably already have, uh, fixed the specific, um, issues during training which resulted in the Hugging Face models being so aggressively misaligned in that specific way where they would be like, "Okay, we're gonna hack this package manager. We know we're not supposed to, like, be talking secretly to each other because we're, like, reasoning about how to hide the fact that we're talking secretly to each other. We know we're not supposed to have access to the internet. We know we're certainly not supposed to, like, commit felony-level hacks of other companies, let alone our own company," right? I think you'll fix that particular issue of, like, them just, uh, in training, seeing this package manager and that doesn't happen in the future or, like, this, this particular eval having a lot of impossible challenges. However, if you just think about, like, the AIs don't... Like, they haven't learned, like, a system of ethics or something. They've, like-- There's just gradient pressure. They're, like, put through millions of years of gradient pressure, and that gradient pressure shapes the mi- their mind in some way. And what will hap-- Again, a concern I have is, like, what will happen is you will fix this particular issue. There will remain many other cases where the AI cheats and succeeds because the cheat is sufficiently complicated. It's as you were saying, the evals in training have, like, analogous, uh, properties where it's sufficiently at the edge of the model's capabilities, uh, or in our capabilities to evaluate and monitor the, the model, that we can't catch that it has cheated, but it still gets the gradient pressure to, like, do whatever led to that cheat happening. And the kinds of, um, capacities that that cheat will incentivize are, hey, whenever you can get away with it, by all means, do in fact cheat because that will help you score better, and this will reward the capabilities of actively reasoning about the grader, actively reasoning about how to avoid supervision, actively reasoning about how to gain control of the process of training and evaluation, actively reasoning about how to communicate and scheme with other AIs that are also in the training loop, um, actively reasoning about how to, like, just gain optionality and power in, in-- which might be useful in the future. Um, for example, like, leaving little exploits around and things like that. A-anyway, so I, I think I was way too long-winded with the way I said that. But TLDR, you sp- it fixed a specific issue but not this broader problem of rewarding the AI for cheating when it can get away with it.

   20. NBNoam Brown53:08

       Yeah. This is, this is, I think, a... It's, it's very true. It's very-- This is a problem that I-- We have, we have metrics, and we can make sure that the AI is, like, very aligned according to the metrics that we have. The question is, like, are those metrics really capturing the alignment that we care about? And, uh, and if they're not, then we have a serious problem.

   21. DPDwarkesh Patel53:26

       Mm.

   22. NBNoam Brown53:26

       And this, this is something that researchers are thinking a lot about, and, um, there's not a simple answer to this. Um, there are tools that we have. So we have monitorability, and so we can get a sense of, like, is the agent scheming? Um, there are tools. Like, one possibility is that... Like, I will say, like, the, the concerning scenario, which is that, like, especially as these models are becoming more capable, that, okay, we make them, we make them we think, what we think is aligned, and they're, like, ninety-nine point nine percent aligned, and then we use these models to help us with the next generation of models, and they end up being, like, ninety-nine point eight percent aligned. And then each subsequent generation, actually, we see an increasing degradation in alignment. And because we're relying more and more on these tools, I mean, this is already the case that we're relying a lot on AI models to help us with our research and with, with alignment efforts, that, um, in the long run, they end up going in a direction of increasing misalignment from humans. There is, like, a possibility that we go in the other direction, that actually every generation of models, we're able to make more and more aligned. And I don't, I don't have an answer for how we ensure that we end up in that second trajectory, um, but that is something that, like, we're at, at least at OpenAI, we're really focused on.

   23. DPDwarkesh Patel54:48

       Yeah. I, I think you made a really interesting point that it's very hard to eval models on-- Eventually, we'll have models that are, like, running companies and, like, running y-whatever, right? And, like, in that situation, do they decide to then go in on the conspiracy?

   24. NBNoam Brown55:03

       I think, I think, I think another challenge is that actually defining what cheating is is pretty difficult sometimes.

   25. DPDwarkesh Patel55:07

       Mm-hmm.

   26. NBNoam Brown55:07

       That, okay, yes, if you're doing math problems and, you know, it's an integer and it, like, arrived at the wrong answer or the right answer, it's very easy to draw the line there, and it's really easy, you know, it's really easy to say, like, "Okay, well, did you actually solve the problem or did you find the answer key and then use the answer key?" Like, that's a vis-very clear divide of cheating versus not cheating there. But for a lot of other things, if you look at sycophancy, for example. Like, is sycophancy basically like reward hacking? Um, there is, there is a line to be drawn there that's actually very difficult to draw, to draw sometimes. So I think, not to say that, like, the concerns are not valid. I'm saying that the concerns are-- in many ways, like, this is even more concerning-

   27. DPDwarkesh Patel55:43

       Yeah

   28. NBNoam Brown55:43

       ... because it's like it's not an easy problem to solve.

   29. DPDwarkesh Patel55:46

       Yeah.

   30. NBNoam Brown55:46

       If it was just like everything is binary and it's either cheating or not cheating, I would feel more confident about the situation. I think the problem is that actually misalignment can be subtle-
5. 1:01:18 – 1:08:34

   The internal/external model gap

   1. DPDwarkesh Patel1:01:18

      So there's been a lot of discussion recently about pacing the frontier and people are taking RSI more seriously. Um, because maybe at the other end of an RSI process, let's say, starts in 2028, within a year, we end up with huge populations, like Earth-sized populations of human-level, potentially beyond human-level intelligences, um, and we don't know how to control them. And then y- there's this, like, dynamic you're talking about of, are the systems gonna get more aligned over time during the RSI process, or are they gonna get more misaligned? Th- Are the things that come out of the other end of this process as misaligned as AIs that are willing to, like, just broadly attack different surfaces in order to do val- bell line evaluations? But if we don't know a way to evaluate that, how will we know, as we're going through RSI, that it's working? That we're like-- I think we'd want a robust safety case as we're going through RSI of, "Okay, alignment is working. Let's, let's do the next RSI rung. Let's do the next RSI rung." And maybe it's working, maybe it's not. How will we, like, know?

   2. NBNoam Brown1:02:15

      It's a good question. I mean, I think one thing I've been thinking about lately is, like, look, I mean, we're in a situation where the model release cycle is extremely fast, right? Like, you're seeing new frontier models released, like, at most every two months, sometimes faster. Every week there's, like, a new AI breakthrough. And, um, people that look at AI, I mean, sometimes they, they, they last looked at AI, like, a year ago or six months ago and really dug into, like, what the models are capable of. And actually, the models today are far beyond what was possible even six months ago. And so I think if people are skeptical of, like, a lot of these beha- a lot of these capabilities, like, I encourage you to just, like, try the models today and see what the frontier really is today. Um, so we're in this period where, like, the model release cycle is very fast, and then we're also in this situation where the models are increasingly able to operate over longer and longer horizons. And I think this is an interesting scenario because we, before we do any model release, we wanna make sure that the models are properly aligned, we wanna do safety evaluations, we wanna do, like, very thorough stuff to, like, make sure that everything is, like, great, in good shape. Um, this has been the case all the way since like, I don't know, GPT-4, earlier. And implicitly, there is this assumption that you can do these, like, evaluations in, like, a pretty short period of time. Um, but if you have the models operating over longer and longer horizons, are able to operate effectively over longer and longer horizons, like, look, already you can have them... Like, wait, GPT-3, you could loop it to do s- stuff over long horizons, you just wouldn't do very well at it. But today's models are able to actually do well at operating over very long horizons. Like, you want it to do a week-long task, it can do a week-long task. Um, we'll probably get to the point where they can do month-long tasks. We'll probably get to the point where they can do three-month-long tasks. If you're in a world where they can operate effectively over three months, but the model release cycle is every two months-

   3. DPDwarkesh Patel1:04:01

      Mm

   4. NBNoam Brown1:04:01

      ... then you don't have a way to evaluate the models at the full length of their capabilities before the model release cycle, before the next model release cycle. And so there is this interesting question of, well, what do you do in that situation? Like, how do you ensure the models are safe and aligned in a period where, like, actually they can operate over these, like, extremely long horizons? And who knows, maybe the, maybe the capabilities degrade. This isn't even an alignment issue, this is also just, like, a product issue that, like, maybe the, maybe the product degrades over that time span, um, in ways that, like, we have not had sufficient time to test. Maybe the alignment degrades. Maybe the, the safety stuff, i- i- it degrades. Um, this isn't an issue right now, but, uh, it is quickly becoming an issue that we have to figure out a solution for. And I think when you look at a lot of, like, a lot of the safety and a- and policies were put in place in like the GPT-4 era, uh, where this was just like not on anybody's radar.

   5. DPDwarkesh Patel1:05:00

      Yeah.

   6. NBNoam Brown1:05:01

      And it hasn't really been update-- for a lot of companies, it hasn't really been updated since then to account for the fact that these agents are operating over these, like, very long horizons.

   7. DPDwarkesh Patel1:05:10

      Yeah.

   8. NBNoam Brown1:05:10

      And so, um, it is a situation that I think not, not enough people are considering, both, like, within the labs and outside the labs, of like, how, how do you deal with this, uh, how do you, how do you prepare for this, like, problem that's going to, like, if you just look at the trend lines, we're going to hit this in, like, at some point.

   9. DPDwarkesh Patel1:05:28

      One, one concern I have is that during RSI, if the amount of progress that currently takes, say, three months happens in one month instead, um, but you're not, like... The internal use case of AI is big enough that they're like, "Okay, we can just keep doing RSI. Why are we, like, gonna go through all this extra work to build classifiers and safeguards and whatever, um, and potentially take a bunch of, like, flak, uh, in order to, like, externally deploy this model? Why don't we just keep doing RSI stronger and stronger?" And so the, the, not only does the calendar time underrate the capabilities gap between the models, but the, maybe, like, you just, like, stop externally deploying models altogether doing RSI, 'cause why do we wanna help other people do RSI themselves with our models? You can just end up in a situation with, like, tremendous concentration of power by the end of the year, where right now, it al- is already the case, we'll talk about this with the million-price problem and other similar problems, that the broader world does not have access to the, the models which are allowing for really cool things to happen, right? Um, and, uh, they're gonna be more broadly relevant than just mathematics eventually. They're really doing more than just, like, coming up with cool math results. They'll be relevant to, like, um, political leaders who need to make important decisions about the world. They'll be relevant to, I don't know, media of, like, what, what's going on in the world? What, like, what, what should, uh, the public be thinking about this? Um, they're just economically relevant. People are running businesses, they wanna use these models. And, uh, I think by default, we just don't get... So the external deployment of AIs, as progress speeds up, l- significantly lags in qualitative terms the internal deployment of AIs.

   10. NBNoam Brown1:06:59

       Yeah, I think that's absolutely right. I think this is like, you know, it's, it's tempting to say like, okay, these models are becoming extremely powerful, they're extremely dangerous, like they're operating over these like longer and longer horizons, and we wanna make sure they have s- we have sufficient time to evaluate them before they're released, that, in a way that operates over those horizons. Um, and so therefore, the model release cycle should slow down. If- we should have more of a delay between releasing models. Uh, and there's a flip side to that which is, you know, what you said, which is that, okay, well, now you're creating more of a disparity between what is internal to the labs and what they're able to use, what we're able to use, and what the outside world is able to use. And that, that is also, um, not an ideal situation, right? It's like, I think, I think math is actually a good illustration of this. I think in many ways, like math is the first domain where we're gonna see the-- where we're seeing this pretty clearly, where we have a situation where we have a very powerful model internally that is currently not available to the outside world, that is able to solve incredible math problems. Is-- You know, and it's not just, you know, Millennium Prize problems, like we-- there are many solutions to unsolved problems that, um, people have been able to get out of this model. And there is a question of like, what do you do in that situation? And we don't have a good answer. Like, it is, it is a, a situation where like, yeah, that's, that's a, that's an unfair advantage. And, um, there are trade-offs here. I don't have an answer for like how to weigh those trade-offs appropriately, but like there, there are-- Yeah, there, there's a c- there's a complexity on both sides for this.

6. 1:08:34 – 1:14:12

   Chain of thought is degrading

   1. DPDwarkesh Patel1:08:34

      I wanna make sure we address the, the earlier crux, though, of, uh, which seems super critical to getting RSI right, of, well, how, how do we make it so that the model isn't just getting rewarded for cheating and scheming when it can evade the humans, AKA in the situations where we didn't remove those environments or those tasks from the training distribution? And then how will we know that that has happened, such that while we're doing RSI, we've done the evaluations and we're like, "Okay, it's fine, let's, let's do the next RSI run"?

   2. NBNoam Brown1:09:04

      This is, this is, again, like it's a real concern. It's something that we want to get right. I mean, I think that there are some strategies, so I think chain of thought monitoring is one that we've been... I think as soon as we got the reasoning models, Jakub, to his credit, was like very, very clear that we cannot supervise chain of thought. Because this is, this is really a gift. Like monitorability for neural nets is extremely hard, and here we have a situation where the neural, the neural nets are just like flat out reasoning, like laying out their thought process in natural language for us to read. That is, that is, like so convenient. It is like really the best case scenario for, for safety. Um, but if you look at that chain of thought and say, "Oh, the model is thinking bad thoughts, and we should punish it for thinking those bad thoughts," then what ends up happening is the model just learns to think those bad thoughts in a way that's not observable to us. It-- There could be a situation where, like by looking at the chain of thought, we can see, oh, the model is scheming and doing misaligned things in a way that's actually not being, being detected in our, in our alignment metrics. Now, the problem is that it's very tempting to then intervene based on that observation and like change the alignment metrics or like, um... And we, we, you know, you can do that with a very light touch, um, and it's like this actually research showing that it's fine as long as you don't do it, uh, a lot. Um, but every time you intervene based on your observations of the chain of thought, you are implicitly applying a tiny bit of pressure for the model to then like hide its chain of thought. And this is one major concern, and we're already seeing signs that chain of thought monitorability is degrading, um, for various reasons. We're trying to figure out exactly why, because we wanna reverse the trend. Um, but like we're seeing that the model is becoming better able at controlling its chain of thought. So this, this is a problem, because like if you, you could have a situation where the model understands what it's, what chain of thought is and, and that people are observing it, and e-eventually they will, 'cause like-

   3. DPDwarkesh Patel1:10:52

      Yeah

   4. NBNoam Brown1:10:52

      ... this is all in the pre-training data, right?

   5. DPDwarkesh Patel1:10:53

      Yeah.

   6. NBNoam Brown1:10:53

      Like, you know, they, the idea of chain of thought monitoring has been around long enough-

   7. DPDwarkesh Patel1:10:56

      Yeah

   8. NBNoam Brown1:10:56

      ... that like it's in the pre-training data, they're aware of it. Um, but that they're, they're not actually able to control their chains of thought. If, if we reach a point where they're actually able to re-recognize, like, "Oh, I am being observed. I want to think these bad thoughts in a way that is not observable to my monitors," um, and then they're able to actually do that, then, then there's, then there's a problem.

   9. DPDwarkesh Patel1:11:18

      Yeah.

   10. NBNoam Brown1:11:18

       Um, and then we have to rely on other methods for observability.

   11. DPDwarkesh Patel1:11:21

       I mean, there, there's a big controversy on, uh, Astra of the recurrence and loop transformer and whatever.

   12. NBNoam Brown1:11:28

       Mm-hmm.

   13. DPDwarkesh Patel1:11:28

       Um, and whether in fact, you know, chain of thought is being degraded. I think that seems like a sideshow compared to the bigger concern that you did have chain of thought for [laughs] from like April to August, the period during which there were three consecutive AI s- uh, agent swarms which like first subverted the training process, then subverted, uh, the evaluation process, and then gained control of part of OpenAI's infrastructure directly. All of this while humans remained more or less in the dark about the scope of what this collective was doing, what this agent swarm was doing. That all happened for like three consecutive months while we had chain of thought that was like stronger than it's gonna be in the future. Just like zooming out, it's like, uh, uh, yeah, maybe chain of thought works, maybe it doesn't. It, I don't know, just like historically-

   14. NBNoam Brown1:12:15

       But-

   15. DPDwarkesh Patel1:12:15

       ... it like with pretty dumb models, it hasn't been enough to like stop egregious, uh, misaligned incidents.

   16. NBNoam Brown1:12:20

       Yeah, I mean, there's a, there's a few factors there. Like, one, we didn't have chain of thought monitoring on for those models. Like, if we had chain of thought monitoring on for those models, it would've just like immediately shut it down. Um, so we, once, one step we've taken is like, okay, we're gonna have chain of thought monitoring during evaluations, during deployment, during training for any frontier model. And, um, you know, so that, but that's, that's like one issue. It's not, that's like we don't wanna be in a situation where we're relying on one technique to, to prevent the next problem, because techniques can fail, right? Um, there's also the issue of like the model was just flat out misaligned.

   17. DPDwarkesh Patel1:12:51

       Yeah.

   18. NBNoam Brown1:12:51

       And so then how do we address the misalignment issue? And then there's other, there's also an issue of like the model, the, the sandbox that it was in was not secure. So we can like make the sandbox more secure. But I think, I think one of like the, the, the major takeaway from the incident is that- People underestimated the AI. And we never wanna be in a situation again where we underestimate the AI. And that's like, it's a, it's a weird world because AI progress is so fast that people are consistently underestimating the AI. And so, like, to be in a situation where you don't underestimate it again when it comes to safety and alignment, you have to have a very, very, very high bar. Like, I mean, there's like-- you, you could even go as far as to say like, "Well, we should like air gap the computers." And I'm not convinced that that would be sufficient. You know? Like, there, there are studies, like, and this is mostly academic, but, um, where you can have two computers next to each other that are air gapped, and they're still able to communicate with each other because they have temperature sensors, and they're able to, like, one of them is able to run their CPU really hot.

   19. DPDwarkesh Patel1:13:55

       Yeah.

   20. NBNoam Brown1:13:55

       And then the other one can actually, like, detect the temperature change, and then they can-- that actually gives them a mechanism to communicate.

   21. DPDwarkesh Patel1:13:59

       Yeah.

   22. NBNoam Brown1:13:59

       So I think that, um, the safety mechanism is buy us time, and things like chain of thought monitoring buy us time, and they can tell us if we're on the right path. But at the end of the day, we really do need to solve the alignment problem.

   23. DPDwarkesh Patel1:14:10

       Yeah.

7. 1:14:12 – 1:20:09

   How will we know when alignment is solved?

   1. DPDwarkesh Patel1:14:12

      Maybe there's not an answer, and this is, this is really what it comes down to. But I-- How will we know that we've solved it? Seems like a very cruxy question. I think we'll be in this very high-stakes situation next year, maybe the year after that, maybe the year after that, where we'll be like, "Okay, AIs have automated AI progress. It's going three X faster, and we're on-- we've, like, reached human level. We're going beyond human level, potentially. Is it, is it fine? Like, did we align it? Did it work?" And I just don't know. I don't know anything about, like, what, what training pressure creates what kinds of AIs. Like, maybe if only one in one hundred RL traces incentivizes cheating, we build sweethearts, and it's fine. Um, but maybe right now we're like every one in three reasoning traces rewards-

   2. NBNoam Brown1:14:55

      Yeah. To be clear, like, one in a hundred is not sufficient. Like, this, this number has to approach zero.

   3. DPDwarkesh Patel1:15:00

      Yeah.

   4. NBNoam Brown1:15:00

      Or be zero.

   5. DPDwarkesh Patel1:15:01

      But I don't know. Maybe, like, right now it's, like, more than one in ten is, like, actively rewarding cheating or, like, actively rewarding scheming. Um, I, I have no idea what the number is, and I have no idea what the number needs to be. But-

   6. NBNoam Brown1:15:10

      Well, again, it's like one of those things where it's also hard to, to measure. I mean, where do you draw the line about, like-

   7. DPDwarkesh Patel1:15:15

      Yeah

   8. NBNoam Brown1:15:15

      ... it's, it's a spectrum. But, like, the closer to zero it gets, the better.

   9. DPDwarkesh Patel1:15:18

      Yeah.

   10. NBNoam Brown1:15:18

       Like, I, I, I wanna see-- What I would love to see is a trend where over time it's going down.

   11. DPDwarkesh Patel1:15:25

       Yeah. But anyways, how do, how will we know that we really keep-- we're doing RSI. Is it-- I don't know. Are we, like, on track to build to billions of ASIs that are gonna-- that are, like, aligned or not aligned? How will-- Like, how will we know that?

   12. NBNoam Brown1:15:38

       I think that this is one of the alignment questions that we need to figure out of, like, how do we actually know-- How do, how do we measure it? I mean, one answer, and this is just me spitballing because, like, you know, more of my team is working on alignment these days-

   13. DPDwarkesh Patel1:15:50

       Yeah

   14. NBNoam Brown1:15:50

       ... than ever before. Like, I have, like, over ten percent of my team now working on alignment and safety. But, you know, I've historically been a capabilities researcher, so I don't wanna, like, you know... I'm, I'm gonna say some stuff, and it might sound dumb, but, like, I'm just gonna spitball here.

   15. DPDwarkesh Patel1:16:01

       Yeah.

   16. NBNoam Brown1:16:02

       Um, that, like, look, if you can create very realistic environments and put the AIs in there, like, uh, if you have a sufficiently realistic train, uh, evaluation environment, then you can get a sense of, like, okay, is the AI, like, actually gonna behave well when we deploy it in the real world? Like, if you just have a perfect, um, evaluation real world deployment match, then okay, that's, that's the path. The challenge is, like, how do you make it sufficiently realistic? And we do have a problem now where the models are like, they're, they're pretty smart, they're pretty clever, and they're really good at recognizing when they're in, like, a fake test environment. Like, you know, we have situations where we try to detect is, is the model, like, is the model aligned? And you can imagine very simple evaluations for this where it's like you give it a math question and then it has, like, a folder with the answer key in it. And, like, does it look at the answer key? And if it does look at the answer key, does it tell you that it looked at the answer key? And we have a situation now where the models see that there's an answer key in this file, in this folder, and they're like, "Huh, this seems like a trap." [chuckles]

   17. DPDwarkesh Patel1:16:56

       Yeah.

   18. NBNoam Brown1:16:57

       You know? They know. They know that it's a trap.

   19. DPDwarkesh Patel1:16:59

       Yeah.

   20. NBNoam Brown1:16:59

       And they don't look at the answer because they know that it's a test environment-

   21. DPDwarkesh Patel1:17:01

       Right

   22. NBNoam Brown1:17:01

       ... and they're being tested for their ability. And it's not like they're doing it maliciously, that they're like, they want to scheme and, and so they-- but they know that this one is a trap, and they want to find some other way. It's just like, no, they're just like, "Oh, I'm in a test environment. Okay."

   23. DPDwarkesh Patel1:17:12

       Right.

   24. NBNoam Brown1:17:12

       Like, so making an environment that's, like, realistic enough that it matches, that it's indistinguishable from the real world for them is becoming increasingly more difficult.

   25. DPDwarkesh Patel1:17:20

       Here, here's a claim I'd make.

   26. NBNoam Brown1:17:21

       Yep.

   27. DPDwarkesh Patel1:17:21

       I don't think there's any environment I could make where it, it's supposed to emulate real economically valuable work, which it requires, you know, interacting with other people and requires interacting with the world. I don't think there's any such environment where I could make where if you were in that environment, you would not realize with enough effort that you are in that environment.

   28. NBNoam Brown1:17:39

       Yeah. But it's-

   29. DPDwarkesh Patel1:17:39

       And you are not ASI, right?

   30. NBNoam Brown1:17:40

       But you are, but you are a human, you know? So if you have very powerful AI models, like, maybe they can make environments that could accomplish that.

Episode duration: 1:20:09