Dario Amodei on Dwarkesh Patel: Why the Exponential Ends

1. 0:00 – 12:36

   What exactly are we scaling?

   1. DPDwarkesh Patel0:00

      So we talked three years ago. I'm curious, in your view, what has been the biggest update over the last three years? What has been the biggest difference between what it felt like the last three years versus now?

   2. DADario Amodei0:08

      Yeah. I would say, actually, the underlying technology, like the exponential of the technology has, has gone, broadly speaking, I would say about, about as I expected it to go. I mean, there's like plus or minus, you know, a, a couple-- there's plus or minus a year or two here, there's plus or minus a year or two there. I don't know that I would've predicted the specific direction of code, um, but, but actually when I look at the exponential, it, it is roughly what I expected in terms of the march of the models from like, you know, smart high school student to smart college student to like, you know, beginning to do PhD and professional stuff, and in the case of code, reaching beyond that. So, you know, the frontier is a little bit uneven. It's roughly what I expected. I will tell you, though, what the most surprising thing has been. The most surprising thing has been the lack of public recognition of how close we are to the end of the exponential. To me, it is absolutely wild that, you know, you have peop- you know, within the bubble and outside the bubble, you know, but, but you have people talking about these, these, you know, just the same tired, old hot button political issues and like, you know, ar-ar-around us we're like [chuckles] near the end of the exponential.

   3. DPDwarkesh Patel1:21

      I, I wanna understand w-what that exponential looks like right now because the first question I asked you when we recorded three years ago was, you know, "What's up with scaling? How, how does it work?" Um, I have a similar question now, but I feel like it's a more complicated question because at least from the public's point of view-

   4. DADario Amodei1:37

      Yes

   5. DPDwarkesh Patel1:37

      ... three years ago, there were these, you know, well-known public trends where across many orders of magnitude of compute you could see how the loss improves. And now we have RL scaling, and there's no publicly known scaling law for it. It's not even clear what exactly the story is of is this supposed to be teaching the model skills? Is it supposed to be teaching meta learning? Um, what is the scaling hypothesis at this point?

   6. DADario Amodei1:58

      Yeah. So, so I have actually the same hypothesis that I had even all the way back in 2017. So in 2017, I think I talked about it last time, but I wrote a doc called the, The Big Blob of Compute Hypothesis. And, a-and, you know, it, it wasn't about the scaling of language models in particular. When I, when I wrote it, GPT-1 had, had just come out, right? So that was, you know, one among many things, right? There was-- Back in those days, there was robotics. People tried to work on reasoning as a separate thing from language models. There was scaling of the kind of RL that happened, it, it, that, that, you know, kind of happened in AlphaGo and, uh, you know, that, that happened at Dota at OpenAI and, um, you know, people remember StarCraft at DeepMind, you know, the AlphaStar. Um, so, uh, it was written as a more general document, and, and the specific thing I said was the following: That, and, you know, it's, it's very, you know, Rich Sutton put out The Bi-Bitter Lesson a couple years later, um, uh, but, you know, the, the hypothesis is, is basically the same. So, so what it says is all the cleverness, all the techniques, all, all the kind of we need a new method to, to do something like that doesn't matter very much. There are only a few things that matter, and I think I listed seven of them. One is, like, how much raw compute you have. The other is the quantity of data that you have. Then the third is kind of the quality and distribution of data, right? It needs to be a broad, broad distribution of data. The fourth is, I think, how long you train for. Um, the fifth is you need an objective function that can scale to the moon. So the pre-training objective function is one such objective function, right? An-another objective function is, you know, the, the kind of RL objective function that says, like, you have a goal, you're gonna go out and reach the goal. Within that, of course, there's objective rewards like, you know, like you see in math and coding, and there's more subjective rewards like you see in RL from human feedback or kind of higher order, higher order versions of that. And, and then the sixth and seventh were things around kind of like normalization or conditioning, like, you know, just getting the numerical stability so that kind of the big blob of compute flows in this laminar way instead of, instead of running into problems. So that was the hypothesis, and it's a hypothesis I still hold. I, I don't think I've seen very much that is not in line with that hypothesis. And so the pre-trained scaling laws were one example of what, of, of, of, of, of kind of what we see there. And indeed, those have continued going. Like, you know, uh, you know, I think, I think now it's been, it's been widely reported, like, you know, we feel good about pre-training. Like pre-training is continuing to give us gains. What has changed is that now we're also seeing the same thing for RL, right? So we're seeing a pre-training phase, and then we're seeing like an RL phase on top of that. Um, and with RL, it's, it's actually just the same. Like, you know, e-even, even other companies have, have published, um, uh, um, like, um, you know, in some of their, in some of their releases have published things that say, "Look, you know, we train the model on math contests, you know, AIME or, or the kind of other things, and, you know, how well, how well the model does is log linear in how long we've trained it." And we see that as well, and it's not just math contests. It's a wide variety of RL tasks. And so we're seeing the same scaling in RL that we saw for pre-training.

   7. DPDwarkesh Patel5:26

      Um, you mentioned Rich, Richard Sutton and The Bitter Lesson.

   8. DADario Amodei5:29

      Yeah.

   9. DPDwarkesh Patel5:29

      I interviewed him, uh, last year, and he is actually very non-LLM pilled. And if I'm, if I, I don't know if this is his perspective, but one way to paraphrase this objection is something like, look, something which possesses the true core of human learning would not require all these billions of dollars of data and compute and these bespoke environments to learn how to use Excel or how does an, you know, how to, how to use PowerPoint, how to navigate a web browser. And the fact that we have to build in these skills using these RL environments hints that we're actually lacking this-... core human learning algorithm. Uh, and so we're scaling the wrong thing. And so yeah, that, that does raise the question, why are we doing all this RL scaling if we do think there's something that's gonna be human-like in its ability to learn on the fly?

   10. DADario Amodei6:19

       Yeah. Yeah. So I think, I think this kind of puts together several things that should be kind of thought of, thought of differently.

   11. DPDwarkesh Patel6:26

       Yeah.

   12. DADario Amodei6:26

       I think there is a genuine puzzle here, but it, it may not matter. Um, uh, in fact, I would guess it probably, it probably doesn't matter. So l- let's take the RL out of it for a second, 'cause I actually think RL, and i- it's a red herring to say that RL is any different from pre-training in this matter. Um, so if we, if we look at pre-training scaling, um, it, it was very interesting. Back in, you know, 2017 when Alec Radford was doing GPT-1, if you look at the models before GPT-1, they were trained on these data sets that didn't represent a wide, you know, distribution of text, right? You had like, you know, these very standard, you know, kinda language modeling benchmarks, and GPT-1 itself was trained on a bunch of, I think it was fan fiction, actually.

   13. DPDwarkesh Patel7:13

       Hmm.

   14. DADario Amodei7:13

       Um, but, you know, it was, it was like literary, you know, it was like literary text, which is a very small fraction of the text that you get. And what we found with that, you know, and, and in those days it was like a billion words or something, so small data sets and represented a pretty narrow distribution, right? Like a narrow distribution of kind of what, what you can see, what you can see in the world, and it didn't generalize well. If you did better on, um, you know, the, the, the, I, you know, I, I forget what it, but it's some, some kind of fan fiction corpus, um, it wouldn't generalize that well to kind of the other ta- You know, we had all these measures of like, you know, how well does a, how well does a model do at predicting all of these other kinds of texts? You really didn't see the generalization. It was only when you trained over all the tasks on the, uh, you know, you know, the internet, when you, when you kind of did a general internet scrape, right, from something like, you know, Common Crawl or scraping links on Reddit, which is what we did for GPT-2. It's only wh- when you do that that you kind of started to get generalization. Um, and I think we're seeing the same thing on RL, that we're starting with first very simple RL tasks, like training on math competitions. Then we're kind of moving to, you know, kind of broader, broader training that involves things like code as a task, and now we're moving to do kind of many, many other tasks, and then I think we're going to increasingly get generalization. So that, that kind of takes out the RL versus the pre-training side of it. But I think there is a puzzle here either way, which is that on pre-training, when we train the model on pre-training, you know, we, we use like trillions of tokens, right? And, and humans don't see trillions of words, so there is an actual sample efficiency difference here. There, there is actually something different that's, that's happening here, which is that the models start from scratch and, you know, they have to get much more, much more training. But we also see that once they're trained, if we give them a long context length, the only thing blocking a long context length is like inference, but if we give them like a context length of a million, they're very good at learning and adapting within that context length. And, and so I don't know the full answer to this, but, but I think there's something going on that pre-training, it's, it's not like the process of humans learning. It's somewhere between the process of humans learning and the process of human evolution. It's like it's somewhere between... Like, we get many of our priors from evolution. Our brain isn't just a blank slate, right?

   15. DPDwarkesh Patel9:41

       Right.

   16. DADario Amodei9:41

       Whole books have been written about... I think the language models, they're much more blank slates. They literally start as like random weights, whereas the human brain starts with all these regions. It's connected to all these inputs and outputs. Um, and, and so maybe we should think of pre-training, and for that matter, RL as well, as, as being something that exists in the middle space between human evolution and, you know, kind of human on, on the spot learning and as the in-context learning that the models do as, as something between long-term human learning and short-term human learning. So, you know, there, there's this hierarchy of like there's evolution, there's long-term learning, there's short-term learning, and there's just human reaction. And the LLM phases exist along this spectrum, but not necessarily exactly at the same points, that there's no analog to some of the human modes of learning, that LLMs are kind of falling between the points.

   17. DPDwarkesh Patel10:38

       Hmm.

   18. DADario Amodei10:38

       Does that make sense?

   19. DPDwarkesh Patel10:39

       Um, yes, although the, some things are still a bit confusing. For example, if the analogy is that this is like evolution, so it's fine that it's not that sample efficient, then like, well, if we're gonna get the kind of super sample efficient agent from in-context learning, why are we bothering to build in, you know, there's RL environment companies which are, it seems like what they're doing is they're teaching it how to use this API, how to use Slack, how to use whatever. It's confusing to me why there's so much emphasis on that if the kind of agent that can just learn on the fly is emerging or is gonna soon emerge or is, has already emerged.

   20. DADario Amodei11:10

       Yeah, yeah. So I, I, I mean, I can't speak for the emphasis of anyone else. I can, I can only talk about how we, how we think about it. I think the way we think about it is the goal is not to teach the model every possible skill within RL, just as we don't do that within pre-training, right? Within pre-training, we're not trying to expose the model to, you know, every, every possible, uh, you know, way that words could be put together, right? You know, we're, it's, it's rather that the model trains on a lot of things and then, and then it reaches generalization across pre-training, right? That was, that was the transition from GPT-1 to GPT-2 that I saw up close, which is like, you know, the m- the model reaches a point. You know, I, I, I, I, like, had these moments where I was like, "Oh yeah, you just give the model, like, you just give the model a list of numbers that's like, you know, um, you know, this is the cost of the house, this is the square feet of the house," and the model completes the pattern and does linear regression. Like, not great, but it does it, but it's never seen that exact thing before. And, and so to, you know, to, to the extent that we are building these RL environments, the, the goal is, is very similar to what is be-- you know, to what was done five or 10 years ago with pre-training with we're trying to get a wh- we're trying to get a whole bunch of data not because we wanna cover a specific document or a specific skill, but because we wanna generalize.

   21. DPDwarkesh Patel12:35

       I mean, I-I,

2. 12:36 – 29:42

   Is diffusion cope?

   1. DPDwarkesh Patel12:36

      I think the framework you're laying down obviously makes sense, like we're making progress towards AGI. I think the crux is something like nobody at this point disagrees that we're gonna achieve AGI in this century, and the crux is you say we're hitting the end of the exponential, um, and somebody else looks at this and says, "Oh, yeah, we're m- we're making progress. We've been making progress since 2012, and then 2035 we'll have a human-like agent." And so I wanna understand what it is that you're seeing which makes you think, um, yeah, obviously we're seeing the kinds of things that evolution did or that human, within human lifetime learning is like in th- these models and why think that it's one year away and not 10 years away.

   2. DADario Amodei13:14

      I, I-

   3. DPDwarkesh Patel13:14

      Yeah.

   4. DADario Amodei13:14

      I actually think of it as like two-- there's kind of two cases to be made here or like two, two claims you could make, one of which is like stronger and the other of which is weaker. So I think starting, starting with the weaker claim, you know, when, when I first saw the scaling back in like, you know, 2019, um, you know, I wasn't sure. You know, this was-- the, the whole business was kind of a fifty-fifty thing, right? I thought I saw something that was, you know-- And, and my claim was this is much more likely than anyone thinks it is. Like this is wild. No one else would even consider this. Maybe there's a fifty percent chance this happens. Um, on the basic hypothesis of, uh, you know, as you put it, within ten years we'll get to, you know, you know, what I call kinda country of geniuses-

   5. DPDwarkesh Patel14:00

      Right

   6. DADario Amodei14:01

      ... in a data center, I'm at like ninety percent on that. Um, and it's hard to go much higher than ninety percent 'cause the world is so unpredictable.

   7. DPDwarkesh Patel14:08

      Yeah.

   8. DADario Amodei14:09

      Um, maybe the irreducible uncertainty would be if we were at ninety-five percent where you get to things like, I don't know, may- maybe multi, you know, multiple companies have, you know, kind of internal turmoil and nothing happens, and then Taiwan gets invaded and like all the, all the fabs get blown up by missiles and, and, you know, and then-

   9. DPDwarkesh Patel14:27

      Now you're getting into scenario. [laughs]

   10. DADario Amodei14:28

       Yeah, yeah, yeah. You, you know, just i- you could construct a, a scenario where there's like a five percent chance that it, it, y- or, you know, you, you can construct a five percent world-

   11. DPDwarkesh Patel14:36

       Yeah

   12. DADario Amodei14:36

       ... where like things, things get delayed for t- for, for, for, for, for, for 10 years. That's maybe five percent. There's another five percent which is that I'm very confident on tasks that can be verified, so I think, I think with coding I'm just except for that irreducible uncertainty there's just, there's-- I mean, I think we'll be there in one or two years. There's no way we will not be th- there in 10 years in terms of being able to do it end-to-end coding. My one little bit, the one little bit of, of f- fundamental uncertainty even on long timescales is this thing about tasks that aren't verifiable, like planning a mission to Mars, like, uh, you know, doing some fundamental scientific discovery like, like CRISPR, like, you know, writing a, writing a novel. Hard to, hard to verify those tasks. I am almost certain that w- we have a reliable path to get there, but like if there was a little bit uncertainty, it's there. So, so, so, so, so on the 10 years I'm like, you know, ninety percent which is about as certain as you can be. Like I think it's, I think it's crazy to say that this won't happen by, by, by 2035. Like in some sane world it would be outside the mainstream.

   13. DPDwarkesh Patel15:48

       But, but the emphasis on verification hints to me as a lack of, a lack of, uh, belief that these models will generalize. If you think about humans-

   14. DADario Amodei16:00

       Yes

   15. DPDwarkesh Patel16:00

       ... we are good at things that both which we get verifiable reward and things which we don't. You're like you have a good start-

   16. DADario Amodei16:05

       We, we, uh, no, no, no, this is, this is why I'm almost sure. We already see substantial generalization from things that, that verify to things that don't veri- we're already seeing that.

   17. DPDwarkesh Patel16:13

       Right. But, but it seems like you were emphasizing this as a spectrum which will, uh, split apart which domains-

   18. DADario Amodei16:20

       Yeah, yeah

   19. DPDwarkesh Patel16:20

       ... we see more progress, and I'm like, but that's, that doesn't seem like how humans get better.

   20. DADario Amodei16:23

       But the world in which we don't make it or, or, or the world in which we don't get there is the world in which we do, we do all the things that are, that are verifiable and then they like, you know, many of them generalize but what we kinda don't get fully there. We don't, we don't, we don't fully, you know, col- we don't fully color in this side of the box. It's, it's, it's not a, it's not a binary thing.

   21. DPDwarkesh Patel16:43

       But, but it also seems to me as even if, if even if we're in the world where generalization is weak when you only see it in verifiable domains, it's not clear to me in such a world you could automate software engineering because software en- like in some sense you are, quote unquote, "a software engineer."

   22. DADario Amodei16:56

       Yeah. Yeah.

   23. DPDwarkesh Patel16:56

       But, yeah, part of being a software engineer for you involves writing these like long memos about your grand vision about-

   24. DADario Amodei17:00

       That's right

   25. DPDwarkesh Patel17:00

       ... different things, and so-

   26. DADario Amodei17:01

       Well, I don't think that's part of the job of SWE. That's part, that's part of the job of the company.

   27. DPDwarkesh Patel17:04

       That's right.

   28. DADario Amodei17:05

       But I do think SWE involves like design documents-

   29. DPDwarkesh Patel17:07

       Right

   30. DADario Amodei17:07

       ... and other things like that, um, which by the way the, the models are not bad. They're already pretty good at writing comments. Uh, and so with, with a- again, I, again, I'm making like much weaker claims here than I believe to like, you know, to, to, to, to, to kinda set up a, a, a, you know, to, to distinguish between two things. Like we're, we're already almost there for software engineering. We are already almost there.
3. 29:42 – 46:20

   Is continual learning necessary?

   1. DPDwarkesh Patel29:42

      Coming back to concrete predictions because I think because there's so many different things to disambu-ambiguate, it can be easy to talk past each other when we're talking about capabilities. So for example, when I interviewed you three years ago, I asked you a prediction about what we should we expect three years from now. I think you were right. So you said we should expect systems which if you talk to them for the course of an hour, it's hard to tell them apart from-

   2. DADario Amodei30:04

      Yes

   3. DPDwarkesh Patel30:04

      ... a generally well-educated human.

   4. DADario Amodei30:05

      Yes.

   5. DPDwarkesh Patel30:05

      I think you were right about that, and I think spiritually I feel unsatisfied because my internal expectation was, was that such a system could automate large parts of white-collar work. And so it might be more productive to talk about the actual end capabilities you want such a system-

   6. DADario Amodei30:20

      So, so I will, I will, I will basically tell you what, what, you know, wh-where, where I think we are. So-

   7. DPDwarkesh Patel30:26

      But let me, let me ask it in a very specific question so that we can figure out exactly what kinds of capabilities we should expect soon. So maybe I'll ask about it in the context of a job I understand well, not because it's the most relevant job, but, um, just 'cause I can evaluate the claims about it. Um-Take video editors, right? I have video editors, and part of their invo- job involves learning about our audience's preferences, learning about my preferences and tastes and the different trade-offs we have and how, just over the course of many months, building up this understanding of context. And so the skill and ability they have six months into the job, a model that can pick up that skill on the job, on the fly, when should we e-expect such an AI system?

   8. DADario Amodei31:03

      Yeah. So I guess what you're talking about is, like, you know, we've-- we're, we're doing this interview for three hours, and then, like, you know, someone's gonna come in, someone's gonna edit it. They're gonna be like, "Oh, you know, you know, I don't know, Dario, like, you know, scratched his head, and, you know, we could, we could edit that out," or, you know-

   9. DPDwarkesh Patel31:19

      Magnify that. [laughs]

   10. DADario Amodei31:19

       There was this, like, long, there was this, like, long discussion that, like, is less interesting to people, and then, then, you know, then there's the other thing that's, like, more interesting to people, so, you know, let's, let's co- let's kind of make this, this edit. So, you know, I think the country of geniuses in a data center wi-will be able to do that. The, the way it will be able to do that is, you know, it will have general control of a computer screen, right? Like, it, you know, and, and, and you'll be able to feed this in, and it'll be able to also use the computer screen to, like, go on the web, look at all your previous, look at all your previous interviews, like, look at what people are saying on Twitter in response to your interviews, like, talk to you, ask you questions, talk to your staff, look at the history of kind of edits, edits that you did, and from that, like, do the job.

   11. DPDwarkesh Patel32:01

       Yeah.

   12. DADario Amodei32:02

       Um, so I think that's dependent on several things. One that's dependent, and, and, and, and I think this is one of the things that's actually blocking deployment, um, getting to the point on computer use, where the models are really masters at using the computer, right? And, you know, we've seen this climb in, in benchmarks, and benchmarks are always, you know, imperfect measures, but, like, you know, OS world is, you know, went from, you know, like, five percent occa-- you know, like, uh, I think when we first re-released, you know, uh, computer use like a, a year and a quarter ago, it was, like, maybe fifteen percent, or I don't remember exactly. But we've climbed from that to, like, sixty-five or seventy percent. Um, and, and, you know, there may be harder measures as well, but, but I think computer use has to pass a point of reliability.

   13. DPDwarkesh Patel32:46

       Can I just ask a follow-up on that-

   14. DADario Amodei32:47

       Yeah.

   15. DPDwarkesh Patel32:47

       -before we move on to the next point? Um, I often-- for years, I've been trying to build different internal LLM tools for myself, and I oft-often I have these text in, text out tasks which should be dead center in the repertoire of these models, and yet I still hire humans to do them just because it's, if it's something like make cl- for-- identify what the best clips would be in this transcript, and maybe they'll do, like, a seven out of ten job at them. But there's not this ongoing way I can engage with them to help them get better at the job the way I could with a human employee. And so that missing ability, even if you solve computer use, would still block my ability to, like, offload an actual job to them.

   16. DADario Amodei33:23

       Again, there's, there's s- this gets back to what w- to, to kind of, to kind of what, what we were talking about-

   17. DPDwarkesh Patel33:27

       Yeah.

   18. DADario Amodei33:28

       -before with learning on the job, where it's, it's very interesting. You know, I think, I think with the coding agents, like, I don't think people would say that learning on the job is what, is what is, you know, preventing the coding agents from, like, you know, doing everything end-to-end. Like, they keep, they keep getting better. We have engineers at Anthropic who, like, don't write any code. And when I look at the productivity, to your, to your previous question, you know, we have folks who say, "This, this GPU kernel, this chip, I used to write it myself. I just have Claude do it." And so there's this, there's this enormous improvement in productivity, and I don't know. Like, when I see Claude code, like, familiarity with the code base or, like, it w- you know, or, or a feeling that the model hasn't worked at the company for, for a year, that's not high up on the list of complaints I see. And so I think what I'm saying is we're, we're, like, we're kind of taking a different path.

   19. DPDwarkesh Patel34:22

       But don't, don't, don't you think with coding that's because there is an external scaffold of memory which exists instantiated in the code base, which I don't know how many other jobs have-- Coding made fast progress preci-precisely because it has this unique, um, advantage that-

   20. DADario Amodei34:35

       But, but-

   21. DPDwarkesh Patel34:35

       -other economic activity doesn't.

   22. DADario Amodei34:37

       But, but when you say that, what you're, what you're implying is that by reading the code base into the context, I have everything that the human needed to learn on the job. So that would be an example of whether it's written or not, whether it's available or not, a case where everything you needed to know you got from the context window, right? And that, and that what we think of as learning, like, "Oh man, I started this job. It's gonna take me six months to understand the code base," the model just did it in the context.

   23. DPDwarkesh Patel35:07

       Yeah. I, I honestly don't know how to think about this because there, there are people who qualitatively report what you're saying. Um, there was a meter study I'm sure you saw last year-

   24. DADario Amodei35:16

       Yes.

   25. DPDwarkesh Patel35:17

       -where they had experienced developers try to close a pull request in repositories that they were familiar with, and those developers reported an uplift. They, they reported that they felt more productive with the use of these models. But in fact, if you look at their output and how much was actually merged back in, there's a twenty percent down lift. They were less productive as a result of using the models. And so I'm trying to square the qualitative feeling that people feel with these models versus, um, one, in a macro level, where are all the, where is this, like, renaissance of software? And then two, when people do these independent evaluations, w-why are we not seeing the, uh-

   26. DADario Amodei35:51

       Yeah. So-

   27. DPDwarkesh Patel35:52

       -productivity benefits that we would expect?

   28. DADario Amodei35:53

       Within Anthropic, this is just really unambiguous, right? We're under an incredible amount of commercial pressure and make it even hard-harder for ourselves because we have all this safety stuff we do that I think we do more than, than, than other companies. So, like, the, the, the pressure to survive economically while also keeping our values is, is just incredible, right? We're trying to keep this 10X revenue curve going. There's, like, there is zero time for bullshit. There is zero time for feeling like we're productive when we're not. Like, these tools make us a lot more productive. Like, why, why do you think we're concerned about competitors using the tools? Because we think we're ahead of the competitors, and, like, w-we don't, we don't want to excel. We, we, we, we wouldn't be going through all this trouble if this was secretly reducing, reducing our productivity. Like, we see the end productivity every few months in the form of model launches. Like, there's no kidding yourself about this. Like, the models make you more productive.

   29. DPDwarkesh Patel36:56

       Um-One, that i- people feeling like they're more productive is qualitatively predicted by studies like this. But two, if I just look at the end output, obviously, you guys are making fast progress. But the fact... You know, the, the, the, the idea was supposed to be w-with recursive self-improvement is that you make a better AI, the AI helps you build a be-better next AI, et cetera, et cetera.

   30. DADario Amodei37:16

       Yes.
4. 46:20 – 58:49

   If AGI is imminent, why not buy more compute?

   1. DADario Amodei46:20

      ones.

   2. DPDwarkesh Patel46:20

      Exactly. So you're, you're predicting that within one to three years. Um, and then generally, Anthropic has predicted that by late '26, early '27, we will have AI systems that are, quote, um, "Have the ability to navigate interfaces available to humans doing digital work today, intellectual capabilities matching or exceeding that of Nobel Prize winners, and the ability to interface with the physical world." And then you gave an interview two months ago with DealBook where you were emphasizing your, um, your company's more responsible compute scaling as compared to your competitors. And I'm trying to square these two views where if you really believe that we're gonna have a country of geniuses, you, you want as big a data center as you can get. There's no reason to slow down. The TAM of a Nobel Prize winner that is actually-- can do everything a Nobel Prize winner can do is, like, trillions of dollars. And so I'm trying to square this conservatism, uh, which seems rational if you have more moderate timelines, with your stated views about AI progress.

   3. DADario Amodei47:16

      Yeah. So, so it actually all fits together. And, and we go back to this fast but not infinitely fast diffusion. So, like, let's say that we're making progress at this rate. Um, you know, the, the, the technology is making progress this fast. Again, I have, you know, very high conviction that, like, it's going, you know, the, the te- the [chuckles] you know, we're, we're, we're gonna get there within, within a few years. I have a hunch that we're gonna get there within a year or two. So a s- a little uncertainty on the technical side, but like, you know, pretty, pretty strong confidence that it won't be off by much. What I'm less certain about is, again, the economic diffusion side. Like, I really do believe that we could have models that are a country of geniuses, a hund- country of geniuses in a data center in one to two years. One question is, how many years after that do the trillions in, you know, do, do the, do the trillions in revenue start rolling in? Um, I don't think it's guaranteed that it's going to be immediate. Um, you know, I think it could be, um, one year. It could be two years. I could even stretch it to five years, although I'm, like, I'm skeptical of that. And so we have this uncertainty, which is even if the technology goes as fast as I suspect that it willWe, we don't know exactly how fast it's gonna drive revenue. We, we know it's coming, but with the way you buy these data centers, if you're off by a couple years, that can be ruinous. It is just like how I wrote, you know, in Machines of Loving Grace, I said, "Look, I think we might get this powerful AI, this country of geniuses in the data center." That description you gave comes from the Machines of Loving Grace. I said, "We'll get that twenty twenty-six, maybe twenty twenty-seven." Again, that is, that is my hunch. Wouldn't be surprised if I'm off by a year or two, but, like, that is my hunch. Let's say that happens. That's the starting gun. How long does it take to cure all the diseases, right? That's, that's one of the ways that, like, drives a huge amount of, of, of, of economic value, right? Like, you cure, you cure every disease. You know, there's a question of how much of that goes to the pharmaceutical company, to the AI company, but there's an enormous consumer surplus because everyone-- you know, every- uh, assuming we can get access for everyone, which I care about greatly, we, you know, we, we cure all of these diseases. How long does it take? You have to do the biological discovery. You have to, you know, g- you have to, you know, m-m-manufacture the new drug. You have to, you know, go through the regulatory proc- I mean, we saw this with, like, vaccines and COVID, right? Like, it, th-th-there's just this we, we got the vaccine out to everyone, but it, it took a year and a half, right? And, and so my question is, how long does it take to get the cure for everything, which AI is the genius that can, in theory, invent out to everyone? How long from when that AI first exists in the lab to when diseases have actually been cured for everyone, right? And, and, w-w-you know, we've had a polio vaccine for fifty years. We're still trying to eradicate it in the most remote corners of Africa. And, you know, the Gates Foundation is trying as hard as they can. Others are trying as hard as they can. But, you know, that's difficult. A-again, I, you know, I don't expect most of the economic diffusion to be as difficult as that, right? That's, like, the most difficult case. But, but there's a, there's a real dilemma here, and, and where I've settled o-on it is it will be f- it will be a-- it will be faster than anything we've seen in the world, but it, it still has its limits. And, and so then when we go to buying data centers, you know, you-- Again, again, the curve I'm looking at is, okay, we, you know, we've had a ten X a year increase every year. So beginning of this year, we're looking at ten billion in, in, in annual, in, you know, rate of annualized revenue at the beginning of the year. We have to decide how much compute to buy. Um, and, you know, it takes a year or two to actually build out the data centers, to reserve the data centers. So basically, I'm saying, like, in, uh, twenty twenty-seven, how much compute do I get? Well, I could assume, um, uh, that, uh, the, uh, revenue will continue growing ten X a year, so it'll be, uh, one, uh, one, uh, hundred billion at the end of twenty twenty-six and one trillion at the end of twenty twenty-seven. And so I could buy a trillion dollars. Actually, it would be, like, five trillion dollars of compute because it would be a trillion dollar a year for, for five years, right? I could buy a trillion dollars of compute that starts at the end of twenty twenty-seven, and if my, if my revenue is not a trillion dollars, if it's even eight hundred billion, there's no force on Earth, there's, there's no hedge on Earth that could stop me from going bankrupt if I, if I buy that much compute. And, and so even though a part of my brain wonders if it's gonna keep growing ten X, I, I can't buy a trillion dollars a year of compute in, in, in, in, in, in, in, in, in, in, in, in, in, twenty twenty-seven. Uh, if I'm just off by a year in that rate of growth or if the, the growth rate is five X a year instead of ten X a year, then, then, you know, then [laughs] then you go bankrupt. Um, and, a-a-and, and so you end up in a world where, you know, you're supporting hundreds of billions, not trillions, and you accept s- you accept some risk that there's so much demand that you can't support the revenue, and you accept still some risk that, you know, y-you got it wrong and it still slow. And so when I talked about behaving responsibly, what I meant actually was not the absolute amount. That, that actually was not, um, you know... I, I think it is true we're spending somewhat less than some of the other players. It's actually the other things, like have we been thoughtful about it or are we YOLOing and saying, "Oh, we're gonna do a hundred billion dollars here or a hundred billion dollars there"? I kinda get the impression that, you know, some of the other companies have not written down the spreadsheet, that they don't really understand the risks they're taking. They're just kinda doing stuff 'cause it sounds cool. Um, uh, and, and we've thought carefully about it, right? We're an enterprise business, therefore, you know, we can rely more on revenue. It's less fickle than consumer. We have better margins, which is the buffer between buying too much and buying too little. And so I think we bought an amount that allows us to capture pretty strong upside worlds. It won't capture the full ten X a year. Um, and things would have to go pretty badly for us to be, for us to be in financial trouble. So I think we've thought carefully and we've made that balance and, and that's what I mean when I say that we're being responsible.

   4. DPDwarkesh Patel53:50

      Okay, so it seems like, um, it's possible that we're a- we actually just have different definitions of country of a genius in a data center. Because when I think of, like, actual human geniuses, an actual country of human geniuses in a data center, I'm like, I, I would happily buy five trillion dollars worth of compute to run a actual country of human geniuses in a data center. So let's say JPMorgan or Moderna or whatever doesn't wanna use them. Also, the, I've got a country of geniuses. Let they'll, they'll start their own company, and if, like, they, they can't start their own company and they're bottlenecked by clinical trials, it is worth stating with clinical trials, like, most clinical trials fail because the drug doesn't work. There's no efficacy, right?

   5. DADario Amodei54:23

      And, and I make exactly that point in-

   6. DPDwarkesh Patel54:25

      Right

   7. DADario Amodei54:26

      ... in Machines of Loving Grace. I say the clinical trials are gonna go much faster than we're used to-

   8. DPDwarkesh Patel54:30

      But-

   9. DADario Amodei54:31

      ... but not, not instant, not infinitely fast.

   10. DPDwarkesh Patel54:33

       A-a-and then suppose it takes a year to, uh, for the clinical trials to work out so that you're getting revenue from that and can make more drugs. Okay, well, you've got a country of geniuses and you're an AI lab, and you have [laughs] you could use, uh, many more AI researchers. Um, you also think that there's these, like, self-reinforcing gains from, you know, smart people working on AI tech. So like, okay, you can have the-

   11. DADario Amodei54:54

       That's right, but-

   12. DPDwarkesh Patel54:55

       You can have the data center working on, you know, like AI progress.

   13. DADario Amodei54:57

       Is there more gains from buyingLike substantially more gains from buying a trillion dollars a year of compute versus three hundred billion dollars a year of compute

   14. DPDwarkesh Patel55:07

       If your competitor's buying a trillion, yes there is. [chuckles]

   15. DADario Amodei55:09

       Well, yeah, no, there's some gain, but then, but ag-again, there, there's this chance that they go bankrupt before, uh, you know, befo- a-a-again, if you're off by only a year, you destroy yourselves. That's the, that's the balance. We're buying a lot. We're buying a hell of a lot. Like, we're not, we're, we're, you know, we're buying a-an amount that's comparable to that, that, you know, the, the, the, the, the, the biggest players in the game are buying. Um, but, but if you're asking me, why don't, why haven't we signed, you know, ten, ten trillion of compute starting in, starting in mid twenty twenty-seven? First of all, it can't be produced. There isn't that much in the world. Um, uh, but, but second, um, what if the country of geniuses comes, but it comes in mid twenty twenty-eight instead of mid twenty twenty-seven? You go bankrupt.

   16. DPDwarkesh Patel55:56

       So if your projection is one to three years, it seems like you should want ten trillion dollars of compute by, um, twenty twenty-nine.

   17. DADario Amodei56:03

       Twenty twenty, maybe twenty twenty-

   18. DPDwarkesh Patel56:05

       I mean-

   19. DADario Amodei56:05

       ... by latest

   20. DPDwarkesh Patel56:05

       ... like, I mean, you know, you, you- But like, are you enter-- Like, it, it seems like even in your l- the longest version of the timelines you state, the compute you are ramping up to build doesn't seem-

   21. DADario Amodei56:15

       What, what, what-

   22. DPDwarkesh Patel56:15

       ... in accordance

   23. DADario Amodei56:16

       ... what, what makes you think that?

   24. DPDwarkesh Patel56:18

       Well, you, you, as you said, you w-you want the ten trillion. Like, human wages, let's say, are, um, o-on the order of fifty trillion a year.

   25. DADario Amodei56:23

       If, if you look at-

   26. DPDwarkesh Patel56:24

       And then, like-

   27. DADario Amodei56:24

       So, so I won't, I won't talk about Anthropic in particular, but if you talk about the industry, like, um, the amount of compute the industry ha- You know, the, the, the, the amount of compute the industry's building this year is probably in the, you know, I don't know, very low tens of m- m- you know, call it ten, fifteen gigawatts. Next year, I, you know, it, it goes up by roughly three X a year, so, like, next year's thirty or forty gigawatts, and, um, twenty twenty-eight might be a hundred. Twenty twenty-nine might be, like, three, three hundred gigawatts. And, like, each gigawatt costs, like, um, maybe ten-- I mean, I'm doing the math in my head, but each gigawatt costs maybe ten billion dollar, you know, o-or the order of ten to fifteen billion dollars a year. So, you know, y-you kind of, y-y-you know, you put that all together and, and you're getting about, about what you described. You're getting multiple trillions a year by twenty twenty-eight or twenty twenty-nine. So you're, you're, you're getting exactly that. You're getting, you're getting exactly what you predict.

   28. DPDwarkesh Patel57:22

       Um, that's for the industry.

   29. DADario Amodei57:24

       That, that's for the industry. That's right.

   30. DPDwarkesh Patel57:26

       So suppose Anthropic's compute keeps three X-ing a year, and then by, like, twenty-seven you have, uh, or twenty-seven, twenty-eight, you have ten gigawatts. And, uh, l-like, multiply that by, as you say, um, ten billion, so then it's, like, a hundred billion a year. But then you're saying the TAM by twenty twenty-eight, twenty twenty-nine-
5. 58:49 – 1:31:19

   How will AI labs actually make profit?

   1. DPDwarkesh Patel58:49

      You've told investors that you plan to be profitable starting in '28, and this is the year where we're, like, potentially getting the country of geniuses as a data center. And w-w-you know, this is, like, gonna now unlock all this, uh, progress and, uh, medicine and, uh, health and et cetera, et cetera, and new technologies. Wouldn't this be particular- the, the, exactly the time where you'd, like, want to reinvest in the business and build bigger countries so they can-

   2. DADario Amodei59:14

      Yeah

   3. DPDwarkesh Patel59:14

      ... make more discoveries?

   4. DADario Amodei59:15

      So, so, I mean, profit-profitability is this kind of, like, weird thing in this field. I, I, like, like, I don't think, I, I don't think in this field profitability is actually a measure of, uh, um, you know, kind of spending down versus investing in the business. Like, let's, let's just, let's just take a model of this. I actually think profitability happens when you underestimated the amount of demand you were gonna get, and loss happens when you overestimated the amount of demand you were going to get, um, because you're buying the data centers ahead of time. So think about it this way. Um, ideally, you would like, uh, and again, these are stylized facts. These numbers are not exact for Anthro-- I'm just trying to make a toy model here. Let's say half of your compute is for training and half of your compute is for inference. Um, and you know, the inference has some gross margin that's, like, more than fifty percent. Um, and so what that means is that if you were in steady state, you build a data center. If you knew exactly the, exactly, exactly the demand you were getting, you would, um, uh, uh, uh, you know, y-you would, you would, you, you would, you would get a certain amount of revenue. Say, I don't know, uh, uh, let's say you pay a hundred billion dollars a year for compute, and on fifty billion dollars a year you support a hundred and fifty billion dollars on, of, of, of, of, of, of, of revenue, and the other fifty billion, the other fifty billion are used for training. Um, so basically you're profitable. You make f- you make f- you make fifty billion dollars of profit. Those are the economics of the industry today. Or, or sorry, not today, but, like, that's where we're, wh-where we're projecting forward in a year or two. The only thing that makes that not the case is if you getLess demand than fifty billion, um, then you have more than fifty percent of your, your data center for research, and you're not profitable. So you, you know, you train stronger models, but you're, like, not profitable. Um, if you, uh, get more demand than you thought, then your research gets squeezed, um, but, uh, you know, you're, you're, you're kind of able to support more inference, and you're more profitable. So it's... Uh, maybe I'm not explaining it well, but, but the thing I'm trying to say is you decide the amount of compute first, and then you have some target desire of, of inference versus, versus training. But that gets determined by demand. It doesn't get determined by you.

   5. DPDwarkesh Patel1:01:29

      But what, what I'm hearing is the reason you're predicting profit is that you are systematically underestimate-- uh, underinvesting in compute, right? Because if you actually like-

   6. DADario Amodei1:01:36

      No, no, no. I'm saying, I'm saying it's hard to predict. So, so these things about twenty twenty-eight and when it will happen, that's our, that's our attempt to do the best we can with investors. All of this stuff is really uncertain because of the cone of uncertainty. Like, we could be profitable in twenty twenty-six if the, if the revenue grows fast enough and then, and then, um, uh, you know, if we, if we overestimate or underestimate the next year, that could swing wildly. Like, I, I, I... What I'm trying to get at is you have a model in your head of, like, the, the business invests, invests, invests, invests, gets scale, and, and, and, and kind of then becomes profitable. There's a single point at which things turn around. I don't think the economics of this industry work that way.

   7. DPDwarkesh Patel1:02:19

      I see. So if I'm understanding correctly, you're saying because of the discrepancy between the amount of compute we should have gotten and the amount of compute we got, we, we were, like, sort of forced to make profit. But that, that doesn't mean we're gonna continue making profit. We're gonna, like, reinvest the money because, well, now AI's made so much progress, and we want the bigger country of geniuses, and so then back into, uh-

   8. DADario Amodei1:02:40

      No, no. If we-

   9. DPDwarkesh Patel1:02:40

      Revenue is high, but losses are also high

   10. DADario Amodei1:02:42

       ... if we, if we predict, if every year we predict exactly what the demand is going to be, we'll be profitable every year because gro- because spending, spending fifty percent of your compute on, on, um, fifty percent of your compute on research, roughly, um, plus a gross margin that's higher than fifty percent and, and correct demand prediction leads to profit. That's the prof- that's, that's the profitable business model that I think is kind of, like, there but, like, obsc- obscured by these, like, building ahead and prediction errors.

   11. DPDwarkesh Patel1:03:13

       I, I guess you're treating the fifty percent as a, uh, as a sort of like, uh, you know, just like a given constant.

   12. DADario Amodei1:03:19

       Yes. Yes.

   13. DPDwarkesh Patel1:03:19

       Whereas you-- In fact, if you, if AI progress is fast and you can increase the progress by scaling up more, you should just have more than-

   14. DADario Amodei1:03:24

       Well-

   15. DPDwarkesh Patel1:03:24

       ... fifty percent and not make profit

   16. DADario Amodei1:03:25

       ... here's what I'll say. You might wanna scale up it more. You might wanna scale it up more, but, but, but, you know, remember the log returns to scale, right? If, if seventy percent would get you a, a, a, a very little bit of a smaller model through a factor of, of one point four X, right, like, that extra twenty billion dollars is, is, is, is, you know, that each, each dollar there is worth much less to you because of, because, because the log linear setup. And so you might find that it's better to invest that, that, that, that... It's better to invest that twenty billion dollars in, you know, in, in serving inference or in hiring engineers who are, who are, who are kinda better, who are, who are kinda better, who are kinda better at what they're doing. So the, the reason I said fifty percent, that's not, that's not exactly our target. It's not exactly gonna be fifty percent. It'll probably vary, vary over time. What, what I'm saying is the, the, the, the, the, like, log linear return, what it leads to is you spend of order one fraction of the business, right? Like, not five percent, not ninety-five percent, and then it, then it, then, you know, then, then you get diminishing returns because of the, because of the log-

   17. DPDwarkesh Patel1:04:30

       Everyone will think that I'm, like, convincing Dario-

   18. DADario Amodei1:04:32

       But, but-

   19. DPDwarkesh Patel1:04:32

       ... to, like, believe in AI progress or something. But, like, uh, you... Okay, you, you don't invest in research because it has diminishing returns, but you invest in the other things you mentioned.

   20. DADario Amodei1:04:39

       Again, again, we're talking about diminishing returns a- after you're spending fifty billion a year, right? Like-

   21. DPDwarkesh Patel1:04:46

       This is a point I, I'm sure you would make, but, like, diminishing returns on a genius is, could be quite high. And more generally, like, what is profit in a market economy? Profit is basically saying the-

   22. DADario Amodei1:04:57

       Well-

   23. DPDwarkesh Patel1:04:58

       ... other companies in the market can, like, do m- more things with this money-

   24. DADario Amodei1:05:00

       So-

   25. DPDwarkesh Patel1:05:00

       ... that I can't

   26. DADario Amodei1:05:01

       ... yeah, I mean, put aside Anthropic. I'm just trying to, like, 'c- 'cause I, you know, I don't wanna give information about Anthropic is why I'm giving these stylized numbers. But, like, let's just derive the equilibrium of the industry, right? I think the eq- So, so, so why doesn't everyone spend one hundred percent of their, um, uh, you know, one hundred percent of their compute on training and not serve any customers, right? It's because if they didn't get any revenue, they couldn't raise money, they couldn't do compute deals, they couldn't buy more compute the next year. So there's gonna be an equilibrium where every, every company spends less than one hundred percent on, on, on, on, on training and certainly less than one hundred percent on inference. It should be clear why you don't just serve the current models and, and, you know, and, and, and, and, and never train another model because then you don't have any demand because you'll, because you'll fall behind. So there's some equilibrium. It's, it's not gonna be ten percent. It's not gonna be ninety percent. Let's just say as a stylized fact it's fifty percent. That's what I'm getting at. And, and, and I think we're gonna be in a position where that equilibrium of how much you spend on training is less than the gross margins that, that you're, that, that, that you're able to get on compute. And so the, the, the, the underlying economics are profitable. The problem is you have this, this hellish demand prediction problem when you're, when you're buying the next year of compute, and you might guess under and be very profitable, but have no compute for research, or you might guess over and, you know, you're, you're, you're, um, uh, you, you are not profitable, and you have all the compute, compute for research in the world. Does, does, does that make sense just as a dynamic model of the industry?

   27. DPDwarkesh Patel1:06:38

       Yeah. May- maybe stepping back, I'm like, uh, I, I, I'm not saying I, I think the country of genius is gonna come in two years, and therefore you should buy this compute. Um, to me, what you're saying, the end conclusion you're arriving at makes a lot of sense, but-Uh, that's because like, oh, it seems like country of geniuses is hard, and there's a long way to go. And so the-- stepping back, the thing I'm trying to get at is more like it seems like your worldview is compatible with somebody who says, "Uh, we're like 10 years away from-

   28. DADario Amodei1:07:06

       Yeah

   29. DPDwarkesh Patel1:07:06

       ... a world in which like we're generating trillions of dollars worth."

   30. DADario Amodei1:07:08

       And that's, and that's, and that's just, that's just not my view.
6. 1:31:19 – 1:47:41

   Will regulations destroy the boons of AGI?

   1. DPDwarkesh Patel1:31:19

      let me ask you about, uh, now, um, making AI go well. Um, it seems like whatever vision we have about how AI goes well has to be compatible with two things. One is the ability to build and run AIs is diffusing extremely rapidly, and two is that the population of AIs, the amount we have and their intelligence, will also increase very rapidly. And that means that lots of people will be able to build huge populations of misaligned AIs or, uh, AIs which are just, like, companies which are trying to increase their, uh, footprint or have weird psyches like Sydney Bing, but now they're superhuman. What is a vision for a world in which-We have an equilibrium that is compatible with lots of different AIs, some of which are misaligned running around.

   2. DADario Amodei1:32:05

      Yeah. Yeah. So I think, you know, in the adolescence of technology, I was kind of, you know, skeptical of, like, the balance of power. But I w- I think I was particularly skeptical of-- Or sp- the thing I was specifically skeptical of is you have, like, three or four of these companies, like, kind of all building models that are kind of derived, you know, sort of, sort of, um, uh, uh, like, derived from the-- like, derived from the same thing and, uh, you know, that, that these would check each other. Or, or even that kinda, you know, any number of them would, would, would, uh, would, would check each other. Like, we might live in a offense-dominant world where, you know, like, one person or one AI model is, like, smart enough to do something that, like, causes damage for everything else. Um, I think in the w-- I mean, in the short run, we have a limited number of players now, so we can start by, within the limited number of players, we, uh, you know, w-we kind of, you know, we, we need to put in place the, you know, the safeguards. We need to make sure everyone does the right alignment work. We need to make sure everyone has bio classifiers. Like, you know, those are, those are kind of the immediate things we need to do. I agree that, you know, that, that doesn't solve the problem in the long run, particularly if the ability of AI models to make other AI models proliferates, then, you know, y- the, the whole thing can kind of, um, you know, uh, can become harder to solve. I, you know, I th- I think in the long run, we need some architecture of governance, right? Some ar- some architecture of governance that preserves human freedom but, but kind of also allows us to, like, you know, govern the, the very large number of kind of, um, you know, uh, uh, uh, human systems, AI systems, [chuckles] hybrid, hybrid human, human, um, you know, h-hybrid h-hy-hybrid human AI, like, you know, companies or, or like, or like, or like economic units. So, you know, we're g- we're gonna need to think about, like, you know, how do we, how do we protect the world against, you know, bioterrorism? How do we protect the world against, like, you know, against, like, against, like, mirror life? Like, you know, probably, probably we're gonna need to, you know, need some kind of, like, AI monitoring system that, like, moni- you know, kind of monitors for, for all of these things. But then we need to build this in a way that, like, you know, preserves civil liberties and, like, our constitutional rights. So I think just, just as, as, a-as is anything else, like, it's, it's like a new security landscape with a new set of, you know, a new set of tools and a new set of vulnerabilities. And I, I think my worry is if we had 100 years for this to happen all very slowly, we'd get used to it. You know, like, we've gotten used to, like, you know, the presence of, you know, the presence of explosives in society or, like, the, you know, the presence of various, um, you know, like, new weapons or the, you know, the pre- the pre- the presence of video cameras. Um, we would get used to it over, over, over, over 100... And we'd develop governance mechanisms. We'd make our mistakes. My, my worry is just that this is happening all so fast.

   3. DPDwarkesh Patel1:35:07

      Mm-hmm.

   4. DADario Amodei1:35:07

      And so I think maybe we need to do our thinking faster about how to make these governance mechanisms work.

   5. DPDwarkesh Patel1:35:12

      Yeah. It seems like in a offense-dominant world, over the course of the next century, so the idea is that AI is making the progress that would happen over the next century happen in some period of five to 10 years. But we would still need the same mechanisms, or balance of power would be similarly intractable even if humans were the only game in town. Um, and so I guess we have the advice of AI. We-- It, it, it fundamentally doesn't seem like a totally different ball game here. If checks and balances were gonna work, they would work with humans as well. If they aren't gonna work, they wouldn't work with AIs as well. Um, and so maybe this just dooms human checks and balances as well, but-

   6. DADario Amodei1:35:50

      Yeah, a-aga-again, I think there's some way to... I think there's some way to make this happen. Like, it, you know, it, it, it just, it just, you know, the governments of the world may have to work together to make it happen. Like, you know, we may have to-- you may have to talk to AIs about kind of, you know, building societal structures in such a way that, like, these, these defenses are possible.

   7. DPDwarkesh Patel1:36:11

      Yeah.

   8. DADario Amodei1:36:11

      I, I, I don't know. I mean, this is so-- This is, you know... I, I don't wanna say so far ahead in time, but, like, so far ahead in te-technological ability that may happen over a short period of time that it's hard for us to anticipate it in advance.

   9. DPDwarkesh Patel1:36:23

      Um, speaking of governments getting involved, on December 26, the Tennessee legislature introduced a bill which, uh, said, quote, um, "It would be an offense for a person to knowingly train artificial intelligence to provide emotional support, including through open-ended conversations with a user." And of course, one of the things that Claude attempts to do is be a, a thoughtful, um, a thoughtful friend, a thoughtful, knowledgeable friend. And in general, it seems like we're gonna have this patchwork of state laws. A lot of the benefits that normal people could experience as a result of AI are going to be curtailed, especially when we get into the kinds of things you discuss in "Machines of Loving Grace," biological freedom, mental health improvements, et cetera, et cetera. It seems easy to imagine worlds in which these get whack-a-moled away by different laws, um, whereas bills like this don't seem to address the actual existential threats that you're concerned about. So I'm curious about, to understand, in the context of things like this, your Anthropic's position against the federal moratorium-

   10. DADario Amodei1:37:22

       Yeah.

   11. DPDwarkesh Patel1:37:22

       -on state AI laws.

   12. DADario Amodei1:37:23

       Yes. So I don't know. There's, there's many different things going on at, at once, right? I think, I think that, that-- I think that particular law is, is dumb. Like, you know, I think it was, it was clearly made by legislators who just probably had little idea what AI models could do and not do. They're like, "AI models serving as... That just sounds scary. Like, I don't want, I don't want that to happen." So, you know, we're, we're, we're not, we're not in favor of that, right? But, but, but that, you know, that, that wasn't the thing that was being voted on. The thing that was being voted on is we're going to ban all state regulation of AI for 10 years with no apparent plan to, to do any federal regulation of AI, which would take Congress to pass, which is a very high bar. Um, so, you know, the idea that we'd ban states from doing anything for 10 yearsAnd people said they had a plan for federal government, but, you know, there was no actual-- there was no proposal on the table. There was no actual attempt. Um, given the serious dangers that I lay out in adolescence of technology around things like the, you know, kind of biological weapons and bioterrorism, autonomy risk, and the timelines we've been talking about, like, 10 years is an eternity. Like, that's, that's a, that's a-- I, I think that's a crazy thing to do. So if, if that's the choice, if that's what you force us to choose, then, then we're gonna, we're gonna choose not to have that moratorium. And, you know, uh, the-- I think the, the, the benefits of that position exceed the costs, but it's, it's not a perfect position if that's the choice. Now, I think the thing that we should do, the thing that I would support is the federal government should step in, not saying, "States, you can't regulate," but "Here's what we're gonna do, and, and states, you can't differ from this." Right? Like, I think preemption is fine in the sense of saying the federal government says, "Here's our standard. This applies to everyone. States can't do something different." That would be something I would support if it would be done in the right way. What-- Um, but, but this idea of states, you can't do anything, and we're not doing anything either, that, that struck, that struck us as, you know, very much not making sense and I think will not age well. It's already starting to not age well with, with all the, um, backlash that, that you've seen. Now, in terms of, in terms of what we would want, I mean, you know, the things we've talked about are, are starting with transparency standards, um, uh, uh, you know, in order to monitor some of these autonomy risks and bioterrorism risks. As the risks become more serious, um, as we, as we get more evidence for them, then I think we could be more aggressive in some targeted ways and, and say, "Hey, AI bioterrorism is really a threat. Let's, let's pass a law that kind of forces people to have classifiers." And I could even imagine... It, it depends. It depends how serious a threat it ends up being. We don't know for sure. Then we need to pursue this in an intellectually honest way where we say ahead of time, "The risk has not emerged yet." But I could certainly imagine with the pace that things are going that, you know, I could imagine a world where later this year we say, "Hey, this, this AI bioterrorism stuff is really serious. We should do something about it. We should put it in a federal-- We should, you know, put it in a federal standard, and if the federal government won't act, we should put it in a state, state standard." I could totally see that.

   13. DPDwarkesh Patel1:40:36

       I, I, I'm concerned about a world where if you just consider the l- the pace of progress you're expecting, the life cycle of, of legislation, you know, the, the benefits are, as you say, because of diffusion lag, the benefits are slow enough that I really do think this patchwork of-- on the current trajectory, this patchwork of state laws would prohibit... I mean, having an emotional chatbot friend is something that freaks people out, then just imagine the kinds of actual benefits from AI we want, uh, normal people to be able to experience from improvements in health and health span and improvements in mental health and so forth. Whereas at the same time, uh, it seems like you think the dangers are already on the horizon, and I just don't see that much, um... It, it seems like it would be especially injurious to the benefits of AI, uh, as compared to the, the dangers of AI, and so that, that's maybe the, where the cost-benefit, uh, makes less sense to me.

   14. DADario Amodei1:41:27

       So, so, so there's a few things here, right? I mean, people talk about there being thousands of these state laws. First of all, the vast mass majority of them do not pass. Um, and, you know, the, the, the, the, the, you know, the world works a certain way in theory, but, like, just because a law's been passed doesn't mean it's really enforced, right? The people, the people, you know, implementing it may be like, "Oh, my God, this is stupid. It would mean shutting off, like, you know, everything that's ever been built and everything that's ever been built in Tennessee." So, you know, very often laws are interpreted in, like, you know, a way that makes them, that, that, that makes them not as dangerous or not as harmful. On, on the same side, of course, you have to worry if you're passing a law to stop a bad thing, you had this, you had this problem as well.

   15. DPDwarkesh Patel1:42:08

       Yeah.

   16. DADario Amodei1:42:09

       Um, uh, look, my, my... Look, I mean, my basic view is, you know, if, if, if, you know, we could decide, you know, what laws were passed and how things were done, which, you know, we're only one small input, input into that, you know, I would deregulate a lot of the stuff around the health benefits of AI. Um, I think, you know, I, I, I don't worry as much about the, like, the, the, the, the kind of chatbot laws. I, I actually worry more about the drug approval process, where I think AI models are going to greatly accelerate, um, the rate at which we discover drugs, and just the, the pipeline will get jammed up. Like, the pipeline will not be prepared to, like, process all, all the stuff that's going through it. So, um, you know, I, I, I think, I think reform of the regulatory process to buy us more towards, we have a lot of things coming where the safety and the efficacy is actually gonna be really crisp and clear. Like, I mean, a, a beautiful thing, re-really, really crisp and clear a-and, like, really, really effective. But, you know, and, and, and maybe we don't need all this, all this, um, uh, uh, like, um, a-all this superstructure around it that was designed around an era of drugs that barely work and often have serious side effects. Um, but at the same time, I think we should be ramping up quite significantly the, um, uh, you know, this, this kind of safety and security legislation. And, you know, like I've said, um, you know, starting with transparency is, is my view of trying not to hamper the industry, right? Trying to find the right balance. I'm worried about it. Some people criticize my essay for saying that's too slow. The dangers of AI will come too soon if we do that. Well, basically, I kinda think, like, the last six months and maybe the next few months are gonna be about transparency. And then if these ri- if these risks emerge when we're more certain of them, which I think we might be as soon as la- as later this year, then I think we need to act very fast in the areas that we've actually seen the risk. Like, I think the only way to do this is to be nimble. Now, the legislative process is normally not nim-nimble, but we, we need to emphasizeTo everyone involved, the urgency of this. That's why I'm sending this message of urgency, right? That's why I wrote Adolescence of Technology. I wanted policymakers to read it. I wanted economists to read it. I want national security professionals to read it. You know, I want decision-makers to read it so that they have some hope of acting faster than they would have otherwise.

   17. DPDwarkesh Patel1:44:39

       Is there anything you can do or advocate that would make it more certain that the benefits of AI are, um, are better instantiated? Where I feel like you have worked with legislatures to be like, "Okay, we're gonna prevent bioterrorism here a way, we're gonna increase transparency, we're gonna increase whistleblower protection." And I just think by default, the actual be-- like, the things we're looking forward to here, it just seems very easy. They seem very fragile to, uh, different kinds of moral panics or political economy problems.

   18. DADario Amodei1:45:09

       Yeah. I don't actually-- So, so I don't actually agree that much in the developed world. I feel like, you know, in the developed world, like markets function pretty well, and when there's, when there's like a lot of money to be made on something, and it's clearly the best available alternative-

   19. DPDwarkesh Patel1:45:26

       Mm.

   20. DADario Amodei1:45:26

       -it's actually hard for the regulatory system to stop it. You know, we're s- we're seeing that in AI itself, right? I, you know, like a thing I've been trying to fight for is export controls on chips to China, right? And, like, that's in the national security interests of the US, like, you know, that's like square within the, you know, the, the policy beliefs of, you know, every-- almost everyone in Congress of both parties. But a-and, you know, I think the case is very clear. The counterarguments against it are, I'll politely call them fishy. Um, uh, and yet it doesn't happen, and we sell the chips because there's, there's so much money. There's so much money riding on it. Um, and, you know, the, the-- that money wants to be made and, and in that case, in my opinion, that's a bad thing. Um, and but, but it also, it also applies when, when it's a good thing. And, and so I, I don't think that if we're talking about drugs and benefits of the technology, I, I, I, I am not as worried about those benefits being hampered in the developed world. I am a little worried about them going a, too slow, and I, as I said, I do think we should work to speed the approval process in the FDA. I do think we should fight against these chatbot bills that you're describing, right? Described individually, I'm against them. I think they're stupid. Um, but I actually think the bigger worry is the developing world, um, where we don't have functioning markets, where, um, you know, we often can't build on the technology that, that we've had. I worry more that those folks will get left behind. And I worry that even if the cures are developed, you know, maybe there's someone in rural Mississippi who, who doesn't get it as well, right? That's a, that's a, that's a kinda smaller version of the thing, the concern we have in the, in the developing world. And so the things we've been doing are, you know, you know, we work with, you know, we work with, you know, philanthropists, right? You know, we work with folks, um, who, you know, who, you know, deliver, you know, medicine and health interventions to, you know, to, to developing world, to sub-Saharan Africa, you know, India, Latin America, you know, o-o-o-you know, other, other developing parts of the world. That's the thing I think that won't happen on its own.

   21. DPDwarkesh Patel1:47:40

       Hmm.

7. 1:47:41 – 2:05:46

   Why can’t China and America both have a country of geniuses in a datacenter?

   1. DPDwarkesh Patel1:47:41

      You mentioned export controls.

   2. DADario Amodei1:47:42

      Yeah.

   3. DPDwarkesh Patel1:47:42

      Why can't US and China both have a country of geniuses-

   4. DADario Amodei1:47:46

      Why can't-

   5. DPDwarkesh Patel1:47:46

      -on a data center?

   6. DADario Amodei1:47:47

      Why can't, you know... Why won't it happen, or why shouldn't it happen?

   7. DPDwarkesh Patel1:47:49

      No, like, why, why shouldn't it happen?

   8. DADario Amodei1:47:51

      Why shouldn't it happen? Um, you know, I think, I think if this does happen, um, you know, then, then we kind of have a-- Well, w-we could have a few situ-- If we have, like, an offense-dominant situation, we could have a situation like nuclear weapons, but, like, more dangerous, right? Where it's like, um, you know, kind of, kind of either side could, could easily destroy everything. Um, we could also have a world where it's kind of, it's unstable. Like, the nuclear equilibrium is stable, right? Because it's, you know, it's like deterrence. But let's say there were uncertainty about, like, if the two AIs fought, which AI would win. Um, that could create instability, right? You, you often have conflict when the two sides have a different assessment of their likelihood of winning, right? If one side is like, "Oh, yeah, there's a ninety percent chance I'll win," and the other side's like, "There's a ninety percent chance I'll win," then, then, then a fight is much more likely. Um, they can't both be right, but they can both think that.

   9. DPDwarkesh Patel1:48:45

      But this seems like a g- fully general argument against the diffusion of AI technology, which it may, it, which-

   10. DADario Amodei1:48:50

       Well, so-

   11. DPDwarkesh Patel1:48:50

       That's the implication of this world. Um-

   12. DADario Amodei1:48:51

       Let me, let, let me just go on 'cause I think we will get diffusion eventually. The other concern I have is that people, the governments will oppress their own people with AI. And, and, and so, um, you know, I'm, I'm just, I'm worried about some world where you have a country that's already, uh, you know, kind of a, uh, you know, uh, uh, you know, there's, there's a government that kind of, kind of already, um, you know, is, is kind of, kind of building a, you know, a tech, a high-tech authoritarian state. Um, and to be clear, this is about the government. This is not about the people. Like, people, we need to find a way for people everywhere to benefit. Um, my worry here is about governments. Um, so yeah, my, you know, my, my worry is if the world gets carved up into two pieces, one of those two pieces could be authoritarian or totalitarian in a way that's very difficult to displace. Um, now, will, will governments eventually get powerful AI and, and, you know, there's risk of authoritarianism? Yes. Will governments eventually get powerful AI and there's risk of, um, uh, you know, of, of kind of bad, bad, bad equilibria? Yes, I think both things, but the initial conditions matter, right? You know, at, at, at some point, we're need, we're gonna need to set up the rules of the road. I'm not saying that one country, either the United States or a coalition of democracies, which I think is a, would be a better setup, although it requires more international cooperation than we currently seem to wanna make. Um, but, you know, I don't, I don't think a coalition of democracies or, or certainly one country should just say, "These are the rules of the road." There's gonna be some negotiation, right? The world is gonna have tograpple with this. And what I would like is that the, the, the, you know, the democratic nations of the world, those with, you know, who, who are clo-- whose governments have, represent closer to pro-human values are, are holding the stronger hand then, have, have more leverage when the rules of the road are set. And, and so I'm, I'm very concerned about that initial condition.

   13. DPDwarkesh Patel1:50:47

       W- I, um, I was re-listening to an interview from three years ago, and one of the ways it aged poorly is that I kept asking questions assuming there was gonna be some key fulcrum moment two to three years from now, when in fact, being that far out, it just seems like progress continues, AI improves, AI is more diffused, and people will use it for more things. It seems like you're imagining a world in the future where the countries get together, and here's the rules of the road, and here's the leverage we have, here's the leverage you have, when it seems like on current trajectory, everybody will have more AI. Um, some of that AI will be used by authoritarian countries. Some of that within the authoritarian countries will be used by private actors versus state actors. It's not clear who b- will benefit more. It's always unpredictable to tell, tell in advance. You know, it seems like the internet privileged authoritarian countries more than you would've expected. Um, and maybe the AI will be the opposite way around. Um, so I, I wanna better understand what you're imagining here.

   14. DADario Amodei1:51:40

       Yeah. Yeah. So, so just to be precise about it, I think the exponential of the underlying technology will continue as it has before, right? The models get smarter and smarter, even when they get to country of geniuses in a data center. P- you know, I, I, I think you can continue to make the model smarter. There's a question of, like, getting diminishing returns on their value in the world, right? How much does it matter after you've already solved human biology or, you know... Eh, eh, you know, at some point you can do harder math, you can do more abstruse math problems, but nothing after that matters. But putting that aside, I do think the, the exponential will continue, but there will be certain distinguished points on the exponential, and companies, individuals, countries will reach those points at different times. Um, and, and so, you know, there's, there's, you know, could there be so- You know what? You know, I talk about is nuclear deterrent still in adolescence of technology? Is nuclear deterrent still stable, uh, in the world of, of, of AI? I don't know, but that's, that's an example of, like, one thing we've taken for granted that, like, the technology could reach such a level that it's no longer, like... You know, we can no longer be certain of it, at least. Um, uh, you know, think of, think of others. You know, there, there, there, you know, there, there are kind of points where if you re- if you reach a certain point, you maybe you have offensive cyber dominance, and, like, every, every computer system is transparent to you after that, um, uh, un- unless the other side has a, has a kind of equivalent defense. So I don't know what the critical moment is or if there's a single critical moment, but I think there will be either a critical moment, a small number of critical moments, or some critical window where it's like AI is, AI confers some large advantage from the perspective of national security, and one country or coalition has reached it before others. That, that, you know, that, that, that, you know, I'm not advocating that they're just like, "Okay, we're in charge now," or tha- that's not, that's not how, that's not how I think about it, you know. The, there's always the, the other side is catching up. There's extreme actions you're not willing to take, and, and, and it's not right to take, you know, to take complete, um, to take complete control anyway. But, but at, at the point that that happens, I think people are gonna understand that the world has changed, and there, there's gonna be some negotiation, implicit or implicit, about what, what is the, what does the post-AI world order look like? And, and I think my interest is in, you know, making that neg- negotiation be one in which, you know, classical liberal democracy has, you know, has a strong hand.

   15. DPDwarkesh Patel1:54:23

       But w- I wanna understand what that better means 'cause you say in the essay, quote, "Autocracy is simply not a form of government that people can accept in the post-powerful AI age." And that sounds like you're saying the CCP as an institution cannot exist after we get AGI. Um, and that seems like a, a, like, a s- very strong demand, and it seems to imply a world where the leading lab or the leading country will be able to and, by that language, should get to determine how the world is governed or what kinds of governments are allowed and not allowed.

   16. DADario Amodei1:54:59

       Yeah. So w- when I, when I, um, I, I believe w- th- that paragraph was-- I think I said something like, "You could take it even further and say X." So I wasn't, I wasn't necessarily endorsing that, that, that-- I wasn't necessarily endorsing that view. I, you know, I was saying, like, "Here's, if first, you know, here, here's a weaker thing that I believe." But, you know, I thi- I, you know, I think I said, you know, "We have to worry a lot about authoritarians-

   17. DPDwarkesh Patel1:55:23

       Right

   18. DADario Amodei1:55:23

       ... and, you know, we should try and, you know, kind of, kind of check them and limit their power." Like, you could take this kind of further, much more interventionist view that says, like, authoritarian countries with AI are these, you know, the, the, the, you know, the- these kind of self-fulfilling cycles that you, that you can't-- that are very hard to displace, and so you just need to get rid of them fro- from the beginning. That, that has exactly all the problems you say, which is, you know, uh, you know, if you were to make a commitment to overthrowing every authoritarian country, I mean, they, then they would take a bunch of actions now that, like-

   19. DPDwarkesh Patel1:55:54

       Right

   20. DADario Amodei1:55:54

       ... you know, that, that, that could, could lead to instability. So that, that may or, you know, that, that, that just, that just may not be possible. But the point I was making that I do endorse is that it is, it is quite possible that, you know, today, you know, the view, or at least my view, or the view in most of the Western world is, is democracy is a better form of government than authoritarianism. But it's not like if a country's authoritarian, we don't react the way we reacted if they committed a genocide or something, right? And, and I'm-- W- I guess what I'm saying is I'm a little worried that in the age of AGI, authoritarianism will have a different meaning. It will be a graver thing. Um, and, and we have to decide one way or another how to, how, how, w- how, how, how to deal with that. And the interventionist view is one possible view. I was exploring such views. Um, you know, uh, uh-It may end up being the right view, it, it may end up being too extreme to be the right view, but I do have hope. A-and one piece of hope I have is there, there is-- we have seen that as new technologies are invented, forms of government become obsolete. I, I mentioned this in Adolescence of Technology, where I said, you know, like, feudalism was basically, you know, like, a form of government, right? And, and then when, when we invented industrialization, feudalism was no longer sustainable, no longer made sense.

   21. DPDwarkesh Patel1:57:21

       Why is that hope? Why-- W- Couldn't that imply that democracy is no longer gonna be-

   22. DADario Amodei1:57:24

       Well-

   23. DPDwarkesh Patel1:57:25

       ... a competitive system?

   24. DADario Amodei1:57:26

       It, it cou- Right. It, it could go, it could go either way, right? But, but I actually-- So I-- These problems with authoritarianism, right? That the problems with authoritarianism get deeper. I just-- I wonder if that's an indicator of other problems that authoritarianism will have, right? Another words, people become-- Because authoritarianism becomes worse, people are more afraid of authoritarianism. They work harder to stop it. It's, it's more of a c- like, you have to think in terms of total equilibrium, right? Um, I just wonder if it will motivate new ways of thinking about, with the, with, with the new technology, how to preserve and protect freedom.

   25. DPDwarkesh Patel1:58:12

       Mm.

   26. DADario Amodei1:58:12

       And, and, uh, even more optimistically, will it lead to a collective reckoning and, you know, a, a, a kind of a, a more emphatic realization of how important some of the things we take as individual rights are, right? A, a more emphatic realization that we just, we really can't give these away. There's, there-- We've seen, you-- There's no other way to live that actually works. Um, I, I, I, I am actually, I am actually hopeful that-- I, I guess one way to say it, it sounds too idealistic, but I actually believe it could be the case, is that, is that, is that dictatorships become morally obsolete. They become morally unworkable forms of government, um, and that, and that, and that the, the, the, the crisis that that creates is, is, is sufficient to force us to find another way. Um.

   27. DPDwarkesh Patel1:59:03

       I, I think there is genuinely a tough question here, which I'm not sure how you resolve. For, uh-- And we've had to come out one way or another on it through history, right? So with China in the '70s and '80s, we decided, even though it's an authoritarian system, we will engage with it. And I think in retrospect, that was the right call, because in a state authoritarian system, but a billion-plus people are much wealthier and better off than they would've otherwise been. Um, and it's not clear that it would've stopped being an authoritarian country otherwise. You can just look at North Korea, uh, as an example of that, right? And I don't know if that t- if that mu- m- that much intelligence to remain an authoritarian country that continues to coalesce its own power, as you can just imagine a North Korea with an AI that's much worse than everybody else's, but still enough to keep power. And, and, and then, and then-- So in general, it seems like should we just have this attitude of the benefits of AI will, in the form of all these empowerments of humanity and health and so forth, will be big, and, and historically we have decided it's good to spread the benefits of technology widely, even with, even to people whose governments are authoritarian. And I think, I guess it is a tough question w- how to think about it with AI, but, um, historically we have said, "Yes, there's, there's a positive sum world, and it's still worth diffusing the technology."

   28. DADario Amodei2:00:14

       Yeah. So, so there are a number of choices we have. I, you know, I think framing this as a kind of government-to-government decision and, and, you know, in, in national security terms, that's, like, one lens, but there are a lot of other lenses. Like, you could imagine a world where, you know, we produce all these cures to diseases and, like, the, you know, the, the, the cures to diseases are fine to sell to authoritarian countries. The data centers just aren't, right? The chips and the data centers just aren't. Um, and, and the, the AI industry itself. Um, uh, you know, uh, like, like, another possibility is, and, and I think folks should think about this, like, you know, could there be developments we can make either that naturally happen as a result of AI or that we could make happen by building technology on AI? Could we create an equilibrium where, where it becomes infeasible for authoritarian countries to deny their people kind of private use of the benefits of the technology? Um, uh, you know, are there, are there, are there, are there equilibria where we can kind of give everyone in an authoritarian country their own AI model that kind of, you, you know, like, defends themselves from surveillance, and there isn't a way for the authoritarian country to, like, crack, crack down on this while, while retaining power? I don't know. That, that sounds to me like if that went far enough, it would be, it would be a reason why authoritarian countries would disintegrate from the inside. Um, but, but maybe there's a middle world where, like, there, there's an equilibrium where if they wanna hold onto power, the authoritarians can't deny kind of individualized access, access to the technology. But I actually do have a hope for the, for the, um, for the, for the more radical version, which is, you know, i- is it possible that the technology might inherently have properties, or that by building on it in certain ways, we could create properties, um, that, that, that, that have this kind of dissolving effect on authoritarian structures? Now, we, we hoped originally, right? If we think back to the beginning of the Obama administration, we thought originally that, that, you know, social media and, and the internet would have that property, and it turns out not to. But, but I, I don't know. What, what if we could, uh, what if we could try again with, with the knowledge of how many things could go wrong and that this is a different technology? I don't know that it would work, but it's worth a try.

   29. DPDwarkesh Patel2:02:25

       Yeah. I, I, I think it just, it, it's very unpredictable. Like, there, there's first principles reasons why authoritarianism might be privileged, but-

   30. DADario Amodei2:02:31

       It's, it's all very unpredictable.
8. 2:05:46 – 2:22:19

   Claude's constitution

   1. DPDwarkesh Patel2:05:46

      You guys recently announced Claude is gonna have a constitution that's aligned to a set of values and not necessarily just to the end user. And there's a world I can imagine where if it is aligned to the end user, it preserves the balance of power we have in the world today because everybody gets to have their own AI that's advocating for them. And so the ratio of bad actors to good actors stays constant. It seems to work out for our world today. Um, why is it better not to do that but to have a specific set of values that the AI should carry forward?

   2. DADario Amodei2:06:14

      Uh, yeah. So I, I'm not sure I'd quite draw the distinction in that way.

   3. DPDwarkesh Patel2:06:18

      Yeah.

   4. DADario Amodei2:06:18

      Um, there, there may be two relevant distinctions here, which are, uh, I think you're talking about a mix of the two. Like, one is should we give the model a set of instructions about do this and d- versus don't do this?

   5. DPDwarkesh Patel2:06:30

      Yeah.

   6. DADario Amodei2:06:31

      And the other, you know, versus should we give the model a set of principles for, you know, for kinda how to act? Um, and, and, and there, it's, it's, you know, it's i-i-i-i-i-you know, it's, it's just pu- it's, it's kind of purely a practical and empirical thing that we've observed that by teaching the model principles, getting it to learn from principles, its behavior is more consistent, it's easier to cover edge cases, and the model is more likely to do what people want it to do. In other words, if, you know, if you're like, you know, "Don't tell people how to hotwire a car. Don't speak in Korean. Don't," you know, d-uh, you know, just, you know, if you give it a, a list of rules, it doesn't really understand the rules, and it's kind of hard to generalize from them, um, you know, if, if it's just kind of a, like, you know, list of do, do's and don'ts. Whereas if you give it principles, and then it, you know, it has some hard guardrails like, "Don't make biological weapons," but overall, you're trying to understand what it should be aiming to do, h-how it should be aiming to operate. So just from a practical perspective, that turns out to be just a more effective way to train the model. That's one piece of it. So that, you know, that's the kind of rules versus principles trade-off. Then there's another thing you're talking about, which is kind of like the corrigibility versus, um, like, you know, I would say kind of intrin, you know, intrinsic motivation trade-off, which is like how much should the model be a kind of, I don't know, like a, a, a skin suit or something where, you know, you know, i-i-i-you know, you just kind of, you know, it, it just kind of fo- directly follows the instructions that are given to it by whoever is giving it those instructions, um, versus how much should the model have an inherent set of values and, and go off and do things on its own. Um, and, and, and, and, and there, I, I would actually say e-everything about the model is actually closer to the direction of, of, like, you know, it should mostly do what people want. It should mostly follow the ins-- We're not trying to build something that, like, you know, goes off and runs the world on its own. We're actually pretty far on the corrigible side. Now, now, what we do say is there are certain things that the model won't do, right? That it's like, you know, that, that, that, and, and I think we say it in various ways in the Constitution, that under normal circumstances, if someone asks the model to do a task, it sh-should do that task. That, that should be the default. Um, but if you've asked it to do something dangerous or if you've, you know, if you've, um, asked it to, um...You know, ah, ah, ah, to kind of harm someone else, um, then the model is unwilling to do that. So I, I actually think of it as like a mostly, a mostly corrigible model that has some limits, but those limits are based on principles.

   7. DPDwarkesh Patel2:09:10

      Yeah, I mean, then the fundamental question is how are those principles determined? And this is not a special question for Anthropic, this would be a question for any AI-

   8. DADario Amodei2:09:16

      Yeah

   9. DPDwarkesh Patel2:09:16

      ... company. But, um, ah, because you have been the ones to actually write down the principles, I get to ask you this question. Uh, normally a constitution is like you write it down, it's set in stone, and there's a process of updating it and changing it and so forth. In this case, it seems like a document that people at Anthropic write that can be changed at any time that guides the behavior of systems that are gonna be the basis of a lot of economic activity. What is the... H- How, how do you think about w- how, how those principles should be set?

   10. DADario Amodei2:09:49

       Yes. Um, so I think there's, there's two, there's maybe three, three kind of sizes of loop here-

   11. DPDwarkesh Patel2:09:56

       Right

   12. DADario Amodei2:09:56

       ... like three, three ways to iterate. One is you can iterate, we iterate within Anthropic, we train the model, we're not happy with it, and we kinda change the constitution.

   13. DPDwarkesh Patel2:10:03

       Yeah.

   14. DADario Amodei2:10:03

       And I think that's good to do. Um, and, you know, putting out publicly, you know, making updates to the constitution every once in a while saying, "Here's a new constitution."

   15. DPDwarkesh Patel2:10:11

       Right.

   16. DADario Amodei2:10:11

       I think that's good to do 'cause people can comment on it. The second level of loop is different companies will have different constitutions. Um, and you know, I think it's useful for like Anthropic puts out a constitution and, you know, you, the Gemini model puts out a constitution and, you know, other companies put out a constitution, and then, then they can kind of look at them, compare. Outside observers can critique and say this, this, "I like this one, this thing from this constitution and this thing from that constitution." And, and then kind of that, that creates some kind of, you know, soft incentive and feedback for all the companies to like take the best of each elements and improve. Then I think there's a third loop which is, you know, society beyond the AI companies and beyond just those who kind of, you know, who, who comment on the constitutions without hard power. And, and there, you know, we've done some experiments like, you know, a couple years ago we did an experiment with I think it was called the Collective Intelligence Project to like, um, you know, to, to basically poll people and ask them what should be in our AI constitution. Um, ah, and, and, you know, yeah, I think at the time we incorporated some of those changes. And so you could imagine with the new approach we've taken to the constitution doing something like that. It's a little harder because it's like that was actually an easier approach to take when the constitution was like a list of dos and don'ts. Um, at the level of principles, it has to have a certain amount of coherence. Um, but, but you could, you could s- still imagine getting views from a wide variety of people. And I think you could also imagine, and this is like a crazy idea, but hey, you know, this whole interview is about crazy, crazy ideas, right?

   17. DPDwarkesh Patel2:11:43

       [laughs]

   18. DADario Amodei2:11:43

       So, um, uh, you know, you could even imagine systems of, of kind of representative government having, having input, right? Like, you know, I, I wouldn't, I wouldn't do this today because the legislative process is so slow, like this is exactly why I think we should be careful about the legislative process in AI regulation. But there's no reason you couldn't in principle say like, you know, all AI, uh, you know, all AI models have to have a constitution that starts with like these things and then like you can append, you can append other things after it, but like there has to be this special section that like takes precedence. I wouldn't do that. That's too rigid. That, that sounds, um, you know, that, that, that, that sounds kind of overly prescriptive in a way that I think overly aggressive legislation is. But like that is a thing you could, you know, like, like that is a, that is a thing you could try to do. Is, is there some much less heavy-handed version of that? Maybe.

   19. DPDwarkesh Patel2:12:33

       I, I, I really like control loop too, um, where obviously this is not how constitutions of actual governments do or should work, where there, there's not this vague sense in which the Supreme Court will feel out how people are feeling and what are the vibes-

   20. DADario Amodei2:12:48

       Yeah

   21. DPDwarkesh Patel2:12:48

       ... and then update the, update the constitution accordingly. So there's-

   22. DADario Amodei2:12:50

       Yeah

   23. DPDwarkesh Patel2:12:50

       ... it, with actual governments there's a more procedural pro-

   24. DADario Amodei2:12:53

       More formal process, yeah

   25. DPDwarkesh Patel2:12:54

       ... process. Yeah, exactly. But what, you actually have a vision w- of competition between constitutions which is actually very reminiscent of how, um, some libertarian charter cities people used to talk about-

   26. DADario Amodei2:13:06

       Yes

   27. DPDwarkesh Patel2:13:06

       ... what an archipelago of different kinds of governments could look like.

   28. DADario Amodei2:13:08

       Yes, yes.

   29. DPDwarkesh Patel2:13:08

       And then there would be selection among them of who could operate the most effectively-

   30. DADario Amodei2:13:12

       Yes

Episode duration: 2:22:19