AI 2027: month-by-month model of intelligence explosion — Scott Alexander & Daniel Kokotajlo

1. 0:00 – 7:45

   AI 2027

   1. SAScott Alexander0:00

      Other countries, especially China, will be coming up with superintelligence around the same time. People both in Beijing and Washington are going to be thinking, "Well, if we start integrating this with the economy sooner, we're going to get a big leap over our competitors."

   2. DKDaniel Kokotajlo0:13

      We made the guess that early in 2027, the company would basically be like, "We are going to deliberately wake up the president and, like, scare the president with all of these demos of crazy stuff that could happen, and then use that to lobby the president to help us grow faster and to cut red tape."

   3. SAScott Alexander0:26

      I know this sounds crazy, because if you read our document, all sorts of bizarre things happen. It's probably the weirdest couple of years that have ever been. But we're trying to take almost a conservative position where the trends don't change. I think it might be useful to think of our timelines as being like 2070, 2100, it's just that the last 50 to 70 years of that all happened during the year 2027-

   4. DKDaniel Kokotajlo0:48

      (laughs)

   5. SAScott Alexander0:48

      ... to 2028.

   6. DPDwarkesh Patel0:50

      Today, I have the great pleasure of chatting with Scott Alexander and Daniel Coccotello. Scott is, of course, the author of the blog Slate Star Codex, Astral Codex, 10Now. Um, it's actually been a, as you know, a big bucket list item of mine to get you on the podcast. So, this is also the first podcast we've ever done, right?

   7. DKDaniel Kokotajlo1:08

      Yes.

   8. DPDwarkesh Patel1:09

      And then Daniel is the director of the AI Futures project. And you have both just launched today something called AI 2027. So, what is this?

   9. SAScott Alexander1:20

      Yeah. AI 2027 is a scenario trying to forecast the next few years of AI progress. Um, we're trying to do two things here. First of all is we just want to have a concrete scenario at all. So, you have all these people, Sam Altman, Dario Amodei, uh, Elon Musk, saying gonna have AGI in three years, superintelligence in five years. And people just think that's crazy, because right now, we have chatbots that's able to do like a Google search, not much more than that in a lot of ways. Um, and so people ask, "How is it going to be AGI in three years?" Um, what we wanted to do is provide a story, provide the transitional fossils. So, start right now, go up to 2027 when there's AGI, 2028 when there's potentially superintelligence, show on a month-by-month level what happened. Um, kind of in fiction writing terms, make it feel earned. So, that's the easy part. The hard part is we also want to be right.

   10. DPDwarkesh Patel2:16

       (laughs)

   11. SAScott Alexander2:17

       Um, so w- we're trying to forecast how things are going to go, what speed they're going to go at. We know that, in general, the median outcome for a forecast like this is being totally humiliated when everything goes completely differently. And if you read our scenario, you're definitely not going to expect us to be the exception to that trend.

   12. DPDwarkesh Patel2:37

       (laughs)

   13. SAScott Alexander2:37

       Um, the thing that gives me optimism is Daniel, uh, back in 2021, wrote kind of the prequel to this scenario, uh, called What 2026 Looks Like, it's his forecast for the next five years of AI progress, and he got it almost exactly right. Like, you should stop this podcast right now-

   14. DPDwarkesh Patel2:55

       (laughs)

   15. DKDaniel Kokotajlo2:55

       (laughs)

   16. SAScott Alexander2:55

       ... and you should go and read this document. It's amazing. Kind of looks like you asked ChatGPT, "Summarize the past five years of AI progress," and you got something with like a couple of hallucinations, but basically well-intentioned-

   17. DPDwarkesh Patel3:07

       Yeah.

   18. SAScott Alexander3:07

       ... and correct. So, when Daniel said he was doing this sequel, I was very excited, um, really wanted to see where it was going. It goes to some pretty crazy places, and I'm excited to talk about it more today.

   19. DKDaniel Kokotajlo3:21

       I think you're hyping it up a little bit too much.

   20. DPDwarkesh Patel3:23

       (laughs)

   21. DKDaniel Kokotajlo3:23

       Yes, I do recommend people, uh, go read the old thing I did, which was a blog post. Um, I think it got a bunch of stuff right, a bunch of stuff wrong, but overall held up pretty well and inspired me to like try again and do a better version of it.

   22. SAScott Alexander3:34

       I think read the document and decide which of us is right.

   23. DKDaniel Kokotajlo3:38

       Another related thing too is that, um, it was going to... The original thing was not supposed to end in 2026, it was supposed to go all the way through the exciting stuff, right? 'Cause everyone's talking about like, "What about AGI? What about superintelligence?"

   24. DPDwarkesh Patel3:50

       Yeah.

   25. DKDaniel Kokotajlo3:50

       Like, what would that even look like? So, I was trying to sort of like step-by-step work my way from where we were at the time until things happen and then see what they look like. Uh, but I basically chickened out-

   26. DPDwarkesh Patel4:01

       (laughs)

   27. DKDaniel Kokotajlo4:01

       ... when it got to 2027 because things were starting to happen and the-

   28. DPDwarkesh Patel4:04

       That's right.

   29. DKDaniel Kokotajlo4:05

       ... automation loop was starting to take off, and it was just so confusing and there was so much uncertainty. So, I basically just deleted, I- I just, uh, deleted the last chapter and published what I had up until that point, and that was the blog post.

   30. DPDwarkesh Patel4:17

       Okay. And then Scott, how did you get involved in this project?
2. 7:45 – 15:30

   Forecasting 2025 and 2026

   1. DPDwarkesh Patel7:45

      okay, n- now let's talk about this new forecast. Let's start, because w- you do a month by month analysis of what's gonna happen from here. So, what is it that you expect in mid-2025 and end of 2025 in this forecast?

   2. SAScott Alexander7:57

      So, beginning of the forecast, um, mostly focuses on agents. So, we think they're going to, uh, start with agency training, expand the time horizons, get coding going well. Um, our theory is that they are, to some degree consciously, to some degree accidentally, working towards this intelligence explosion, where the AIs themselves can start taking over some of the AI research, move faster. So, 2025, slightly better coding, 2026, slightly better agents, slightly better coding. Um, and then we focus on, and we name the scenario after 2027, uh, because that is when this starts to pay off. Um, the intelligence explosion gets into full swing. The agents become good enough to help with, at the beginning not really do, but help with some of the AI research. So, we s- we introduced this idea called the R&D progress multiplier. So, how many months of progress without the AIs do you get in one month of progress with all of these new AIs helping with the intelligence explosion? So, 2027, we start with... I can't remember if it's literally start with, or by March or something, a five times multiplier for algorithmic progress.

   3. DKDaniel Kokotajlo9:10

      I mean, we have it, we ha- so we have, like, the, the stats tracked on the side of the story. Part of why we did it as a website is so that you can have these cool, uh, gadgets and widgets. And so, as you read the story, uh, the stats on the side automatically update, and so one of those stats is like the progress multiplier. Another answer to the same question you asked is basically 2026, nothing super interesting, or 2025, nothing super interesting happened. Uh, more or less similar trends to what we're seeing.

   4. DPDwarkesh Patel9:34

      Computer use is totally solved? Partially solved? How good is computer use by the end of 2025?

   5. DKDaniel Kokotajlo9:39

      My guess is that they won't be making basic mouse click errors by the end of 2025, like they sometimes currently do.

   6. DPDwarkesh Patel9:44

      Yeah.

   7. DKDaniel Kokotajlo9:44

      Like, if you watch Claude play his Pokemon, which he totally should, uh, it seems like sometimes it's just, like, failing to parse what's on the screen, and like-

   8. DPDwarkesh Patel9:52

      Yeah.

   9. DKDaniel Kokotajlo9:52

      ... it, it mis-... it, like, thinks that its own player character is an NPC and, like, gets confu-... Like, my guess is that that sort of thing will mostly be gone, uh, by the end of this year, but that they still won't be able to, like, autonomously operate for many... for, for long periods on their own, because-

   10. DPDwarkesh Patel10:09

       But, but like, wha- by 2025, when you say it won't be able to act coherently for long periods of time in computer use, if I want to organize a happy hour in my office, I don't know, that- that's like, what, a 30-minute task? What fraction of that is... it's gotta invite the right people, it's gotta book the right DoorDash or something. What fraction of that is it able to do?

   11. DKDaniel Kokotajlo10:27

       My guess is that by the end of this year, there'll be something that can sort of, like, kind of do that, but unreliably.

   12. DPDwarkesh Patel10:33

       Mm-hmm.

   13. DKDaniel Kokotajlo10:33

       And that if you actually, like, tried to use that to run your life, it would make some hilarious mistakes that would appear on Twitter and go viral. But that, like, the MVP of it will probably exist by this year. Like, there'll be, like, some Twitter thread about someone being like, "I plugged in this agent to, like, run my party, and it worked."

   14. DPDwarkesh Patel10:48

       Yeah.

   15. SAScott Alexander10:48

       Our scenario focuses on coding in particular, because we think coding is what starts the intelligence explosion.

   16. DPDwarkesh Patel10:54

       Mm-hmm.

   17. SAScott Alexander10:55

       So, we are less interested in questions of, like, how do you mop up the last few things that are uniquely human, compared to when can you start coding in a way that helps the human AI researchers speed up their AI research? And then, if you've helped them speed up the AI research enough, is that enough to, with some ridiculous speed multiplier, 10 times, 100 times, mop up all of these other things?

   18. DPDwarkesh Patel11:18

       Mm-hmm. One observation I have is, you could have told the story in 2021, once ChatGPT comes out. I think... I had friends who were, like, you know, credible AI thinkers who were like, "Look, you've got the, you've got the coding agent now. It's been cracked. Now the GPT-4 will go around and they'll do all this engineering and we do this all on top. We can totally scale up the system 100x." And every single layer of this has been much harder than the strongest optimist expected. It seems like there have been significant difficulties in increasing the pre-training size, um, at least from rumors about failed training runs or underwhelming training runs at labs. It seems like building up these RL... and I, I'm, like, total outside view. I know nothing about the actual engineering involved here. But just from the outside view, it seems like building up the 01-like RL clearly took...... much, b- at least two years after GPT-4 was released, and with these things are also their economic impact and the kinds of things you would immediately expect based on benchmarks for them to be especially capable at, isn't overwhelming. Like, the call center workers are, haven't been fired yet. (laughs) Um, uh, so th- y- why not just say that, like, "Look, at- at higher scale, it'll probably get even more difficult"?

   19. SAScott Alexander12:33

       Wait a second. I'm a little confused to hear you say that, because when I have seen people predicting AI milestones, like Cut Your Grace's expert surveys-

   20. DPDwarkesh Patel12:42

       Yeah.

   21. SAScott Alexander12:43

       ... um, they have almost always been too pessimistic from a point of view of how-

   22. DPDwarkesh Patel12:47

       Mm-hmm.

   23. SAScott Alexander12:47

       ... fast AI will advance. So like, I think the 2022 survey, um, they ac- I mean, they actually said that things that had already happened would take, like, 10 years to happen.

   24. DPDwarkesh Patel12:59

       (laughs)

   25. SAScott Alexander12:59

       But then when they... The, the survey, it may, it might have been 2023. It was, like, six months before GPT-3, GPT-4 came out, and there were things that GPT-3 or 4 or whichever one of them w- was did that it did in six months th- that they were still predicting, like, five or 10 years from. Right? I, I'm sure Daniel is gonna have, um, a more detailed answer, but I absolutely reject the premise that everybody has always been too optimistic.

   26. DKDaniel Kokotajlo13:23

       Yeah, I think in general, most people following the field have been, uh, have underestimated the pace of AI progress and underestimated the pace of AI diffusion into the world. For example, Robin Hanson famously made a bet about less than a billion dollars of revenue, I think by 2025, from-

   27. DPDwarkesh Patel13:37

       I agree. Robin Hanson in particular has been too (laughs) too pessimistic.

   28. DKDaniel Kokotajlo13:39

       But, but he's a smart guy, you know? And s- so-

   29. DPDwarkesh Patel13:41

       Yeah.

   30. DKDaniel Kokotajlo13:41

       ... I think that the aggregate opinion has been under- underestimating the pace of, of both-
3. 15:30 – 25:22

   Why LLMs aren't making discoveries

   1. DKDaniel Kokotajlo15:30

   2. DPDwarkesh Patel15:30

      Isn't this a good opportunity to discuss, uh, a certain question I asked, uh, Da- Dario that you responded to? (laughs)

   3. SAScott Alexander15:36

      What are you thinking of?

   4. DPDwarkesh Patel15:37

      Well, I asked this question where, as you say, they know all this stuff. I, I, I don't know if you saw this. I, I, uh, um, I asked this question where I said, "Look, these models know all this stuff, and if a human knew every single thing a human has ever written down on the internet, they'd be able to make all these interesting connections between different ideas and maybe even find medical cures or scientific discoveries as a result." Um, there was some guy who noticed that magnesium deficiency causes something in the brain that is similar to what happens when you get a migraine, and so he just said, "Give people magnesium supplements." It cured a lot of migraines. So why aren't LLMs able to leverage this enormous asymmetric advantage they have to make a single new discovery like this?

   5. SAScott Alexander16:18

      Yeah, and then the example I gave was that humans also can't do this. So for me, the most salient example is etymology of words.

   6. DPDwarkesh Patel16:27

      Yeah.

   7. SAScott Alexander16:28

      You have all of these words in English that are very similar, like happy versus hapless, happen, perhaps, and we never think about them unless you read an etymology dictionary, and then-

   8. DPDwarkesh Patel16:38

      (laughs)

   9. SAScott Alexander16:38

      ... like, "Oh, obviously these all come from some old root that has to mean luck or occurrence or something like that."

   10. DPDwarkesh Patel16:43

       Yeah.

   11. SAScott Alexander16:43

       So, like, it, it's kind of about figuring out versus checking. If I tell you those, you're like, "This seems plausible," and of course in etymology there are also a lot of false friends where they seem plausible but aren't connected. Um, but you really do have to have somebody shove it in your face before you start thinking about it and make all of those connections.

   12. DPDwarkesh Patel17:01

       I will actually disagree with this. Um, we know that humans can do... Like, we, we have examples of humans doing this. I, I agree that we don't have logical omniscience, uh, because there is a commentarial explosion, but, um, we are able to leverage our intelligence to... Oh, actually, one of my favorite examples of this is David Anthony, the guy who wrote The Horse, The Wheel, and Language, he made this super im- im- impressive discovery before we had the genetic evidence for it, like a decade before, where he said, "Look, if I look at all these, um, languages in India and Europe, they all share the same etymology." I mean, literally what you're talking about. The same etymology for words like wheel and cart and horse, and these are technologies that have only been around for the last 6,000 years, which must mean that there was some group, uh, that these groups are all at least linguistically descended from. And now we have g- genetic evidence for the Yamnaya, which we believe is this group. Um, you have a blog where you do this. (laughs) This is your job, Scott. So, um, why shouldn't we hold the fact that language models can't do this more against them?

   13. SAScott Alexander18:05

       Yeah, so to me, it doesn't seem like he is just kind of sitting there being logically omniscient and getting the answer.

   14. DPDwarkesh Patel18:11

       (laughs)

   15. SAScott Alexander18:11

       It seems like...He's a genius. He's thought about this for years, probably at some point, like, he heard a couple of Indian words and a couple of European words-

   16. DPDwarkesh Patel18:20

       Yeah.

   17. SAScott Alexander18:20

       ... at the same time, and they kind of connected, and the light bulb came on. So, this isn't about having all the information in your memory so much as the normal process of discovery, which is kind of mysterious, that seems to come from just kind of having good heuristics and throwing them at things until you kind of get a lucky strike. My guess is if we had really good AI agents, and we applied them to this task, it would look something like a scaffold, where it's like, think of every combination of words that you know of, compare them. If they sound very similar, write it on this scratch pad here. If there's a combina- if a lot of words of the same type show up on this scratch pad, that's pretty strange, do some kind of thinking around it. And I just don't think we've even tried that. And I think right now, if we tried it, we would run into the combinatorial explosion. We would need better heuristics.

   18. DPDwarkesh Patel19:09

       Mm-hmm.

   19. SAScott Alexander19:09

       Humans have such good heuristics that probably most of the things that show up even in our conscious mind, rather than happening on the level of some kind of unconscious processing, are at least the kind of things that could be true.

   20. DPDwarkesh Patel19:20

       Mm-hmm.

   21. SAScott Alexander19:20

       Um, like, I think you could think of this as, like, a chess engine. You have some unbelievable number of possible next moves. You have some heuristics for picking out which of those are going to be the right ones. And then gradually, you kind of have the chess engine think about it, go through it, come up with a better or worse move. Then at some point, you potentially become better than humans. I think if you were to force the AI to do this in a reasonable way, or you were to train the AI such that it itself could come up with a plan of going through this in some kind of heuristic-laden way, you could potentially equal humans.

   22. DKDaniel Kokotajlo19:52

       I'll add some more things to that. So, um, I think there's a long and sordid history of people looking at some limitation of the current LLMs and then making grand claims about how the whole paradigm is doomed-

   23. SAScott Alexander20:04

       Mm-hmm.

   24. DKDaniel Kokotajlo20:04

       ... because they'll never overcome this limitation. And then, like, a year or two later, uh, the new LLMs overcome that limitation.

   25. SAScott Alexander20:11

       Yep.

   26. DKDaniel Kokotajlo20:11

       Um, and I would say that, like, with respect to this thing of, like, why haven't they made these interesting scientific discoveries by combining the knowledge they already have and, like, noticing interesting connections? I would say, first of all, have we seriously tried to build scaffolding to make them do this? And I think the answer is mostly no.

   27. DPDwarkesh Patel20:27

       I think Google DeepMind tried this, right?

   28. DKDaniel Kokotajlo20:28

       Maybe, may- so maybe. Second thing, um, have you tried making the model bigger? Uh, they've made it a bit bigger over the last couple years, and it hasn't worked so far. Maybe if they make it even bigger still, they'll notice more of these connections. And then third thing, and here's, here's, I think, the, the special one, have you tried training the model to do the thing? You know, just because, like, the, the, you know, the pre-training, um, the pre-training process doesn't strongly incentivize this type of connection-making, right? Um, in general, I think it's a helpful heuristic that I use to ask the question of, like, r- remind, remind oneself, what was the AI trained to do? What was its training environment like?

   29. DPDwarkesh Patel21:05

       Right.

   30. DKDaniel Kokotajlo21:05

       And if you're wondering, why hasn't the AI done this, ask yourself, like, did the training environment train it to do this? And often the answer is no. And often, I think that's a good explanation for why the AI is not good at it.
4. 25:22 – 50:34

   Debating intelligence explosion

   1. DPDwarkesh Patel25:22

      So, if I look back at AI progress in the past, if we were back in, say, um, 2017, yeah, how, I suppose we had the superhuman coders in 2017. The amount of progress we've made since then, so what we, where we are currently in 2025, by when could we have had that instead?

   2. DKDaniel Kokotajlo25:39

      Great question. We still have to, like, stumble through all the discoveries that we've made since 2017.

   3. DPDwarkesh Patel25:43

      Yeah.

   4. DKDaniel Kokotajlo25:43

      We still have to, like, figure out that language models are a thing. We still have to, like, figure out that you can fine-tune them with RL. Like, like-

   5. DPDwarkesh Patel25:50

      Yeah.

   6. DKDaniel Kokotajlo25:50

      ... so all those things would still have to happen. How much faster would they happen? Maybe 5X faster because (laughs) , uh, because a lot of the, like, small-scale experiments that these people do in order to, like, test out ideas really quickly before they do the big training runs would happen much faster because they're just, like, lickety-split being spit out. Um, I'm not very confident in that 5X number. It could be lower, it could be higher, but that was sort of, like, roughly what we were guessing. Our 5X, by the way, is for the algorithmic progress part, not for the overall thing.

   7. DPDwarkesh Patel26:15

      Oh, okay, got it.

   8. DKDaniel Kokotajlo26:16

      So, so in this hypothetical, according to me, um, basically things would be going, like, 2.5X faster, where the algorithms would be advancing at 5X speed, but the compute is still stuck at the usual speed.

   9. DPDwarkesh Patel26:26

      That seems plausible to me. Um, you have a 5X point and then dot, dot, dot, you have 1,000X AI progress within the matter of a year. Maybe that's the part I'm like, "Wait, how did that happen exactly?"

   10. DKDaniel Kokotajlo26:39

       Yeah.

   11. DPDwarkesh Patel26:39

       Um, so what's the story there?

   12. DKDaniel Kokotajlo26:40

       The way that we did our takeout forecast was basically by breaking down how we think the intelligence explosion would go into a series of milestones. First, you automate the coding, then you automate the whole research process, but in a very similar way to how humans do it, with, like, you know, teams of agents that are about human level. Uh, then you get to a superhuman level and so forth. And so we, we broke it down into these milestones, you know, the superhuman coder, superhuman AI researcher, and then super intelligent AI researcher. And the way we did our forecast was we basically... Well, for each of these milestones, we were like, "What is it gonna take to make an AI that has, that, that achieves that milestone?" Um, and then once you do achieve that milestone, how much is your overall speedup? And then what's it gonna take to achieve the next milestone? Combine that with the overall speedup and that gets you your clock time distance until that happens. And then, okay, now you're at that milestone, what's your overall speedup, assuming that you have that milestone? Also, what's the next one? How long does it take to get to the next one? So we sort of, like, worked through it bit by bit. And at each stage, we're just making our best guesses. Um, so quantitatively, we were thinking something like 5X speedup, uh, to algorithmic progress from the superhuman coder, and then something like a 25X speedup to algorithmic progress from the superhuman AI researcher because at that point, you've got the whole stack automated, which I think is substantially more useful than just automating the coding. Um, and then, uh, I think we... I forget what we say for super intelligent AI researcher, but off the t- top of my head, it's probably something like in the hundreds or maybe, like, 1,000X overall speedup.

   13. DPDwarkesh Patel28:12

       So maybe the big picture, uh, thing I have with the intelligence explosion is, uh, we can go through the specific arguments about how much will the automated coder be able to do and how much will the superhuman AI coder be able to do, but on prior, it's just, like, such a wild thing to expect. And so before we get into all the specific arguments, maybe you can just address this idea that, like, why w- why not just h- like, start off with, like, .01% chance this thing might happen? Then you need extremely, extremely strong evidence that it will before, um, making that your mortal view. I think that it's a question of, like, what is your default option or what are you comparing it to? I think that, naively, people think, like, "Well, every particular thing is potentially wrong, so let's just have a default path where nothing ever happens."

   14. DKDaniel Kokotajlo29:00

       (laughs)

   15. DPDwarkesh Patel29:00

       And I think that-

   16. DKDaniel Kokotajlo29:02

       (laughs)

   17. DPDwarkesh Patel29:02

       ... that has been the most consistently wrong prediction of all.

   18. DKDaniel Kokotajlo29:05

       Yeah.

   19. DPDwarkesh Patel29:05

       Like, I think in order to have nothing ever happen, you actually need a lot to happen. Like, you need, suddenly, AI progress that has been going at this constant rate for so long stops. Why does it stop? Well, we don't know. Whatever claim you're making about that is something where you would expect there to be a lot of out-of-model error, is where you would expect the- like, somebody must be making a pretty definite claim that you wanna challenge. So, I don't think there's a neutral position where you can just say, "Well, given that out-of-model error is really high and we don't know anything, let's just choose that." I think we are trying to take... I know this sounds crazy, because if you read our document, all sorts of bizarre things happen. It's probably the weirdest c- couple of years that have ever been. But we're trying to take, almost in some sense, a conservative position where the trends don't change, um, nobody does an insane thing, nothing that we have no evidence to think will happen happens. And the way that the AI intelligence explosion dynamics work are just so weird that in order to have nothing happen, you need to have a lot of crazy things happen. Mm-hmm.

   20. DKDaniel Kokotajlo30:07

       One of my favorite, you know, meme images is this graph showing world GDP over time. You've probably seen.

   21. DPDwarkesh Patel30:15

       Yeah.

   22. DKDaniel Kokotajlo30:15

       It spikes up, and then there's, like, a little thought bubble.... at, like, the top of the spike in, like, 2020, you know, 2010 or something, and the thought bubble says, like, "My life is pretty normal. I have a good grasp of what's weird versus standard, and people thinking about, uh, different futures with, like, digital minds and space travel are just engaging in silly speculation." Like, the point of the graph is, like, actually, there's been amazing transformative changes in th- the course of history that would have seemed totally insane to people, you know, multiple times. We've gone through multiple such waves of those things.

   23. SAScott Alexander30:52

       Yeah. Everything we've talked about has happened before. Algorithmic progress already doubles, like, every year or so, um, so it's not insane to think that algorithmic progress can contribute to these compute things. In terms of general speedup, we're already at, like, 1,000 times research speedup multiplier compared to the Paleolithic or something. So like, from the point of view of anyone in most of history, we are going at a blindingly insane pace. And all that we're saying here is that it's not gonna stop, is that the same trend that has caused us to have a thousand times speedup multiplier relative to past eras, and not even the Paleolithic. Like, e- what happened in the century between, I don't know, 600 and 700 AD, I'm sure there are things, I'm sure historians could point them out. Then you look at the century between 1900 and 2000, and it's just completely qualitatively different. Uh, of course, there are models of whether that stagnated recently or what's going on here. We can talk about those. We can talk about why we expect for the intelligence explosion to be an antidote to that kind of stagnation. But nothing we're saying is that different from what has already happened. I mean, you are saying that this transition... these... the previous transitions have been smoother than the one you're anticipating. We're not sure about that, actually. So according to, like, one of these models is just a hyperbola. Everything is along the same curve. Another model is that there are these things like the literal Cambrian explosion, if you want to take this very far back, go full Ray Kurzweil, um, the literal Cambrian explosion, the Agricultural Revolution, the Industrial Revolution as phase changes. When I look at the economic modeling of this, my impression is the economists think that we don't have good enough data to be sure whether this is all one smooth process or whether it's a series of phase changes. When it is one smooth process, the smooth process is often a hyperbola that shoots to infinity in weird ways. Um, we don't think it's gonna shoot to infinity. We think it's going to hit bottleneck- Yeah, you guys are the conservative crowd, you know? (laughs)

   24. DKDaniel Kokotajlo32:48

       (laughs)

   25. SAScott Alexander32:48

       Yeah. We think it's gonna hit bottlenecks the same as all these previous processes. The last time this hit a bottleneck, if you take the hyperbola view, is in, like, 1960 when humans stopped reproducing at the same rate they were reproducing before. We hit a population bottleneck, the usual population to ideas flywheel stopped working, and then we stagnated for a while. If you can create a country of geniuses in a data center, as I think Dario Amodei put it, then you no longer have this population bottleneck, and you're just expecting continuation of those pre-1960 trends. So, I- I realize all of these historical hyperolas are also kinda weird, also kind of theoretical, but I don't think we're saying anything that there isn't models for which have previously seemed to work for long historical periods.

   26. DKDaniel Kokotajlo33:32

       Another thing also is I think people equivocate between fast and... or between slow and continuous, right? So like, if you look at our scenario, there's this, like, continuous trend (laughs) that runs through the whole thing of this algorithmic progress multiplier, and we- we're not having discrete jumps from like 0 to 5x to 25x. We have this continuous improvement. Um, so I think continuous is not the crux. The crux is like, is it gonna be this fast, you know? And we don't know. Maybe it'll be slower, maybe it'll be faster, but like, we have our arguments for why we think maybe this fast.

   27. SAScott Alexander34:04

       Okay, let's... uh, now that we brought up the pos- the intelligence explosion, let's, let's just discuss that, because I'm kinda skeptical. Um, it doesn't really seem to me that a- a notable bottleneck to AI progress or the main bottleneck to AI progress is the amount of researchers, engineers who are doing this kind of research. It seems more li- more like compute or some- some other thing is a bottleneck. And the piece of evidence is that when I talk to my AI researcher friends at the labs, they say there's maybe 20 to 30 people on the core pre-training team that's discovering the- all these algorithmic breakthroughs. If the- if this- if the headcount here was so valuable, you would think that, for example, Google DeepMind would take not just everybody from... eh, all the smartest people not just from DeepMind, but for all of Google and just put them on pre-training, or RL, or whatever the big bottleneck was. You think OpenAI would hire every single Harvard math PhD, and in six months, you're all gonna be trained up on how to make, um, do AI research. The fa- they don't seem that... I mean, I know they're increasing headcount, but they don't seem to treat this as the kind of bottleneck that, um, it would have to be for, uh, millions of them in parallel to be rapidly speeding up AI research. And there just is this, you know, um... there's this quote that Napoleon... one Napoleon is worth 40,000 soldiers was a- commonly a thing that was said when he was fighting. But, um, ten Napoleons is not 400,000 soldiers, right? So, why think that this- these million AI researchers are netting you something that looks like an intelligence explosion?

   28. DKDaniel Kokotajlo35:39

       So, previously, I talked about sort of three stages of our takeoff model. First is like you get the superhuman coder. Second is when you're fully automated AI R&D, but it's still at, like, basically human level, like it's- it's because you're best humans. And then third is, like, now you're in superintelligence territory and it's qualitatively better. In our, like, guesstimates of how much faster algorithmic progress would be going, the- the progress multiplier-

   29. SAScott Alexander36:00

       Yeah.

   30. DKDaniel Kokotajlo36:00

       ... for the middle level, we basically do assume that, like, you get massive diminishing returns to-
5. 50:34 – 1:17:43

   Can superintelligence actually transform science?

   1. DKDaniel Kokotajlo50:34

   2. DPDwarkesh Patel50:34

      The other notable thing about your model is, once you've g- y- so you got this, like, superhuman thing at the end of it, and then it seems to just go through the tech tree of, like, mirror life and nanobots and whatever crazy stuff. And maybe that part I'm espec- also really skeptical of. It just looks like if you look at the history of invention, it just seems like you, people are just, like, trying different random stuff. You often, even before the theories about how that industry works or how the relevant machinery works, is developed. Like, the steam engine was developed before the theory of thermodynamics. The Wright Brothers did some things. They were just experimenting with airplanes. Um, uh, and is often influenced by breakthroughs in totally different fields, which is why you have this pattern of parallel innovation because y- the r- you know, the background level of tech is at a point at which you can do this experiment. I mean, machine learning itself is a place where this happened, right? Where people had these ideas about how to do deep learning or something, but it just took a totally unrelated industry, um, of gaming to make the relevant progress to get the whole s- you know, the ... Basically the economy as a whole advanced enough that, like, deep learning, like Ge- Geoffrey Hinton's ideas could work. So I know we're accelerating way into the future here, but-

   3. DKDaniel Kokotajlo51:49

      Yeah, yeah.

   4. DPDwarkesh Patel51:49

      ... I just want to get to this crux.

   5. DKDaniel Kokotajlo51:50

      So, so we ... So again, we have that, like, three-part division of, like, the superhuman coder, then, like, the complete AI researcher, and then, like, the superintelligence.

   6. DPDwarkesh Patel51:56

      Yeah.

   7. DKDaniel Kokotajlo51:56

      You're now jumping ahead to that one. Um, there I would say, uh ... So now we're imagining systems that are like true superintelligence.

   8. DPDwarkesh Patel52:04

      Yeah.

   9. DKDaniel Kokotajlo52:05

      They are just, like, better than the best humans at everything.

   10. DPDwarkesh Patel52:07

       Yeah.

   11. DKDaniel Kokotajlo52:08

       Including being better at data efficiency and better at learning on the job and stuff like that. Now, our scenario does depict a world in which they're bottlenecked on real world experience and r- and, uh, that sort of thing. I think that, like, you know, if you want to contrast, some people in the past, uh, have proposed much faster scenarios where they, like, email some cloud lab and start building nanotech, you know, right away-

   12. DPDwarkesh Patel52:32

       Yeah.

   13. DKDaniel Kokotajlo52:33

       ... by just using their brains to figure out, like, appropriate protein folding and stuff like that. We are not depicting that in our scenario. In our scenario, they are in fact bottlenecked on lots of real world experience to, like, build these actual practical technologies. But the way they get that is they just actually get that experience, and it happens faster than humans would. And the way they do that is, you know, they've ... They're already superintelligent. They're already buddy-buddy with the government. The, the government deploys them heavily in order to, uh, beat China and so forth. And so all these existing US companies and factories and military, even pr- procurement providers and so forth, are all, like, chatting with (laughs) the superintelligences and taking orders from them about, like, how to build the new widget and test it. And like, they're, like, downloading superintelligent designs and, you know, manufacturing them and then testing them and, and so forth. And then the question is like, okay, so th- they are getting this experience. They're learning on the job. Quantitatively, how fast does this go? Like does it t- is it taking years or is it taking months or is it taking days, right? In our story, it takes like about a year. Um, and we're uncertain about this. Maybe it's gonna take several years. Maybe it's going to take less than a year, right? Um, here are some factors to consider for why it's plausible that it could take a year. Um, one, you're gonna have something like a million of them. And quantitatively, that's like comparable in size to the existing scientific industry, I would say. Like maybe it's a bit smaller, but it's not like dramatically smaller. Two, they're thinking a lot faster. They're thinking like 50 times speed or like 100 times speed. That I think counts for a lot. And then three, which is the biggest thing, uh, they're just qualitatively better as well. So not only are they ... there are lots of them and they're thinking very fast, but they are better at learning from each experiment than the best human would be at learning from that experience.

   14. DPDwarkesh Patel54:17

       Yeah, um, I think the fact that there's a million of them, or s- th- the, the fact that they're comparable to maybe the size of the s- key researcher population of the world or something. I don't think a million is ... I think the re- s- there's more than a million researchers in the world, but um-

   15. DKDaniel Kokotajlo54:33

       Well, but it's very heavy tailed. Like a lot of-

   16. DPDwarkesh Patel54:34

       Sure.

   17. DKDaniel Kokotajlo54:34

       ... the research actually comes from like the best ones, you know?

   18. DPDwarkesh Patel54:36

       That's right. But, uh, it's not clear to me that m- most of the new stuff that is developed is a result of this researcher population. I mean, there's just like so many examples in the history of science where, uh ...... a lot of growth or productive improvements is just the result of, you know, h- how do you count, like, the guy at the TSMC process who figures out a different way to-

   19. SAScott Alexander54:57

       So, I actually argued with Daniel about this recently.

   20. DPDwarkesh Patel55:00

       Yeah.

   21. SAScott Alexander55:01

       About one interesting case that I can go over, um, is we have an estimate that about a year after the super intelligences start wanting robots, they're producing a million units of robots per month.

   22. DPDwarkesh Patel55:11

       Mm.

   23. SAScott Alexander55:12

       So, like, I think that's pretty relevant because you have, I think it's Wright's law, which is that, um, your ability to improve efficiency on a process is proportional to, um, doubling the amount of copies produced. If you're producing a million of something, you're probably getting very, very good at it.

   24. DPDwarkesh Patel55:28

       Mm.

   25. SAScott Alexander55:28

       So the question we were arguing about is, can you produce a million units a month after a year? And for context, I think Tesla produces, like, a quarter of that in terms of cars or something. This is an amazing scale up in a year.

   26. DPDwarkesh Patel55:40

       Yeah.

   27. NANarrator55:40

       Only 4x.

   28. SAScott Alexander55:41

       Yeah.

   29. NANarrator55:41

       Also just for Tesla.

   30. SAScott Alexander55:42

       Yeah. And the argument that we went through was something like, so it's gotta first, um, get factories. OpenAI is already worth more than all of the car companies in the US except Tesla combined. So if OpenAI today wanted to buy all the car factories in the US except Tesla, start using them to produce humanoid robots, they could. Obviously not a good value proposition today, but it's just obvious and over-determined that in the future when they have super intelligence and they want them, they can start buying up a lot of factories. How fast can they, uh, convert these car factories to robot factories? So, fastest conversion we were able to find in history was World War II. Um, they suddenly wanted a lot of bombers. So they bought up, in some cases, bought up, in other cases, got the car companies to produce new factories. Um, but they bought up the car factories, converted them to bomber factories. That took about three years from the time when they first decided to start this process to the time when the factories were producing a bomber an hour. Um, we think it will potentially take less with super intelligence because first of all, if you look at the history of this process, despite this being the fastest anybody has ever done this, it was actually kind of a comedy of errors. They made a bunch of really silly mistakes in this process. If you actually have something that even just doesn't have the normal human bureaucratic problems, and we do think that this will be done in the middle of an arms race with China, so the government will be kind of moving things through. Um, and then the super intelligences will be good at the logistical issues navigating bureaucracies. So we estimated maybe if everything goes right, we can do this three times faster than the bomber conversions in World War II. So that's about a year.

Episode duration: 3:05:16