He Risked Everything To Warn You: No One Is Ready For What's Coming, And The AI Companies Know It!

1. 0:00 – 2:34

   Intro

   1. DKDaniel Kokotajlo0:00

      The scary open secret in the AI industry right now is that it's possible that we'll end up essentially creating a new species that ends up ruling the world with a 70% chance that this goes horribly wrong, like human extinction. That's one possibility. There's many more.

   2. SBSteven Bartlett0:12

      It's quite chilling what you're saying.

   3. DKDaniel Kokotajlo0:13

      Yeah, it's, uh, it gets me down sometimes. I basically told my wife, like, "Let's not have any more kids. It's too uncertain. I don't think they'll ever join the workforce." Everybody should be afraid that their jobs are gonna be lost, and I know this because I went to OpenAI in 2022. What I did there was forecasting as to what the next couple years might look like, and unfortunately, most of the world is kind of asleep at the wheel and doesn't really realize what's going on with AI, so I resigned.

   4. SBSteven Bartlett0:37

      I read it somewhere that you lost $2 million for not signing an anti-disparagement clause, meaning you couldn't criticize the company.

   5. DKDaniel Kokotajlo0:44

      Yes, for reasons I'm happy to get into. But the main thing I've learned is when I go talk to people at Anthropic and OpenAI about forecasting, they're like, "It's not gonna take that long. You need to shorten them again. Get them back to 2027 or 2028." Because these powerful CEOs, Dario or Sam or Elon, are racing each other to be in control of the most powerful AIs and are literally afraid that if the other guy gets there first, he might become dictator. I mean, Anthropic is on track to be the entire economy by 2030, but none of these people should be trusted with that much power. So this is the most important thing happening in our lifetimes, probably in all of history, in fact, and it's very important that it go well. So I think that there's a lot we can do to, like, steer things in a better direction. There's loads of benefits that we could get from AI if we do it right, and if we do solve the problems, then things could be absolutely amazing for everyone.

   6. SBSteven Bartlett1:26

      Well, this report here in 2021, it was remarkably accurate, and then you just published this one.

   7. DKDaniel Kokotajlo1:31

      Yeah. So this is our new scenario.

   8. SBSteven Bartlett1:32

      So let's go through these slowly and one at a time.

   9. DKDaniel Kokotajlo1:34

      I would be incredibly happy if all my predictions turn out to be wrong.

   10. SBSteven Bartlett1:40

       This is super interesting to me. My team give me this report to show me how many of you that watch this show subscribe, and some of you have told us, according to this, that you are unsubscribed from the channel randomly. So favor to ask all of you, please could you check right now if you've hit the subscribe button, if you are a regular viewer of the show and you like what we do here. We're approaching quite a significant landmark on this show in terms of a subscriber number. So if there was one simple free thing that you could do to help us, my team, everyone here, to keep this show free, to keep it improving year over year and week over week, it is just to hit that subscribe button and to double-check if you've hit it. Only thing I'll ever ask of you. Do we have a deal? If you do it, I'll tell you what I'll do. I'll make sure every single week, every single month, we fight harder and harder and harder and harder to bring you the guests and conversations that you wanna hear. I've stayed true to that promise since the very beginning of The Diary of a CEO, and I will not let you down. Please help us. Really appreciate it. Let's get on with the show. [upbeat music]

2. 2:34 – 4:21

   Why This AI Mission Could Affect Everyone

   1. SBSteven Bartlett2:34

      Daniel Kokotajlo, at the very heart of what you do, um, what is your mission and why?

   2. DKDaniel Kokotajlo2:41

      So what would you do if you thought that superintelligence was coming in a few years?

   3. SBSteven Bartlett2:46

      I guess it depends what the consequences were.

   4. DKDaniel Kokotajlo2:51

      Well, let's talk about it. So superintelligence, AIs that are better than the best humans at everything, while also being faster and cheaper, also able to operate robots that can do everything in the physical world that humans can do, but better, faster and cheaper. If that really is coming in a few years, then we need to prepare, and we need to think about how to make it go well instead of poorly. So that's sort of my answer is, like, I'm doing that to the best of my ability.

   5. SBSteven Bartlett3:15

      So you believe it's coming in a few years?

   6. DKDaniel Kokotajlo3:16

      Yes.

   7. SBSteven Bartlett3:17

      How could you be so sure?

   8. DKDaniel Kokotajlo3:19

      I spend a lot of time trying to forecast this sort of thing. My sort of median estimate, 50% chance, is currently in 2029. Maybe it'll slip to 2028. It's possible that it'll take significantly longer, like maybe 10 years or something like that. But, uh, you know, for reasons I'm happy to get into, seems to me like it's probably happening by the end of the decade. What's less important is the, the sense of how close we are. What's more important is the pace of the trends. Anthropic this time last year was making something like a billion dollars a year, and now they're making something like $60 billion a year. So that's 60 X growth in one year, which is extremely impressive even for very small startups, but for a company of their size, it might be the fastest growth in history. Um, we expect that rate of growth to slow down, but even if it slows down quite a lot, they're still on track to be, you know, the entire economy by 2030 or so.

3. 4:21 – 8:29

   Why the Average Person Should Care About AI

   1. SBSteven Bartlett4:21

      Why should the average person care?

   2. DKDaniel Kokotajlo4:22

      High level thing is absolutely everything's going to change for the whole world, and including therefore for them and their families. Um, could change for the better, could change for the worse, depending on the details of how it's done. So, for example, everyone could die. You know? Um, this is the classic loss of control scenario or one version of it. If we do build these superintelligences and we use them to automate all the jobs and we put them in the military and we, you know, have them giving advice to politicians and so forth, they will eventually have accumulated enough real world power that they don't need humans anymore, and they're smarter than us, they're more strategic, et cetera. At that point, we sort of have to hope that they are virtuous, that they have, you know, the goals that we wanted them to have, the values that we want them to have, et cetera. And the sort of scary open secret in the AI industry right now is that right now that is kind of just a hope. It's not something that we can be at all confident in, and in fact, there's lots of evidence and arguments that it- we're not on track to achieve that. So there's lots of reasons... Like current AIs, for example, will often lie, uh, to people, or they will, like, you tell them to do something, and they go do something else and then pretend that they did it, right? So it's an inherently difficult problem to make something that's superintelligent and also has the values and virtues that you want it to have, and it doesn't seem like we're on track to solve that problem. Also, it seems like the sort of problem that you could think you've solved when you haven't actually solved it, right? Uh, that's a big reason why this is scary. So for all those reasons, it's possible that we'll end up essentially creating a new species that ends up ruling the world instead of us, and then maybe we go the way of other extinct species in the past that were out-competed by humans. That's one possibility. There's many more. Even if you're not worried about that and you think that the AIs will be totally controlled, there's the question of who controls the AIs, right? When there's a couple corporations that have made these superintelligences and are using them to automate all the jobs, well, that's a lot of power, [chuckles] you know. That's a lot of money. It's a lot of political power. They'll have the best strategists, the best advisors. You know, th- they'll think faster. Militarily, uh, the countries that has these AIs will be able to absolutely wipe the floor with all the other countries. The AIs themselves, it's, it's kind of a single point of failure, like central, uh, control system, where, you know... The CEO of Anthropic, Dario, he coined this phrase, "The country of geniuses in the data, data center." That was his phrase to describe what they're trying to build, you know? I think that's a little bit misleading. I think it would be more accurate to describe it as army of geniuses in the data center, because it's not like it's a bunch of diverse, different AIs, you know, living in their different parts of the data center. They're all copies of the same big model, and they're owned by the company, and so they all follow the orders given by the company, right? People should be asking questions of, like, who controls this army or these armies, and what are they gonna be doing with them? I think that we could very easily end up in a sort of, uh, a situation where some tiny group of people are essentially oligarchs or dictators. And ironically, both of these risks, the loss of control and the constitution of power, are things that people in the industry have been thinking about for decades. Um, even before the AI industry existed, you know, people thinking about AI were talking and writing about these things. And then part of the founding narrative or the founding myth of DeepMind and OpenAI and Anthropic is, "These problems are real, so we need to get there first [chuckles] so that we can handle it responsibly." Those are, I think, the big two reasons, but then I can go on. There's lots more reasons as well. So one thing is, you know, World War III, geopolitical conflict. Um, if AI does in fact get incredibly powerful, that's going to change the balance of power between nations. That's going to disrupt a lot of things. That puts us at increased risk of crisis more generally, right? Another one, what about those jobs? [chuckles] You, you, you're gonna lose your taxi job, but not just the taxi job, for everybody pretty much. Um, there might be a few exceptions, like people whose jobs, for legal reasons, are only allowed to be done by humans. But for the most part, everybody should be afraid that their jobs are gonna be lost, even if we manage to avoid all the other problems, right?

4. 8:29 – 10:06

   Are AI Experts Overreacting or Sounding the Alarm?

   1. SBSteven Bartlett8:29

      This narrative has started to emerge, and I've had several interviews on this show where I've interviewed people who are very, very scared and anxious about AI, and these are people that have worked in the industry for sometimes decades.

   2. DKDaniel Kokotajlo8:38

      Yeah.

   3. SBSteven Bartlett8:39

      Um, the counter-narrative coming over the hill is that this is doomerism, that these people are, for whatever reason, just trying to scare people, and that they don't really understand what they're talking about. How do you respond to that sort of counter-narrative? And you must have seen this emerging yourself, especially from people who stand to benefit, dare I say.

   4. DKDaniel Kokotajlo8:55

      Yeah, exactly. This counter-narrative is fairly recent, and it's been pushed by the people who stand to benefit, um, from it, and it's not true. Like, these, these concerns have been around for decades, since before the AI industry existed. They're actually pretty reasonable concerns. Like, if you take the companies at their word and imagine that they are, in fact, going to build superintelligence, well, it raises a lot of questions, like, "Who's gonna control it? Will anybody control it? What about the jobs?" [chuckles] You know, like, th- these are just kind of obvious implications to be thinking about and worrying about.

   5. SBSteven Bartlett9:24

      Who are you, and what's your story?

   6. DKDaniel Kokotajlo9:26

      My name is Daniel Kokotajlo. Um, I currently run the AI Futures Project, which is a small nonprofit that mostly focuses on forecasting the future of AI. Before that, I worked at OpenAI.

   7. SBSteven Bartlett9:40

      AI forecasting?

   8. DKDaniel Kokotajlo9:42

      Yeah. So think about how, like, you know, industry analysts who work for hedge funds and stuff will make these forecasts of, like, "Here's, you know, how many cars Tesla will be s- selling five years from now," or, like, "Here's what the price of electricity will be in two years," right? That's forecasting. I was doing that, but specifically focused on AI. The reason I was doing it is because it's incredibly important to, to see where

5. 10:06 – 13:24

   Why He Joined OpenAI—and What He Saw Inside

   1. DKDaniel Kokotajlo10:06

      this is all headed.

   2. SBSteven Bartlett10:07

      Why did you go to OpenAI? What did you do there? What did you observe while you were there, and how did it change your perspective on the future of AI, but also, I guess, OpenAI as a company? And for anybody that doesn't know, OpenAI are the company that produce ChatGPT.

   3. DKDaniel Kokotajlo10:21

      Yeah. So I went to OpenAI in 2022. Uh, a large part of what I did there was more forecasting. AI 2027 is a scenario that you may have heard of. I did, like, smaller, you know, lower effort versions of them internally for just internal circulation of, like, "Here's some guesses as to what the next couple years might look like." I also worked on evaluations for dangerous capabilities, so you know, trying to measure the AI's cyber abilities or persuasion abilities or situational awareness. And I also briefly was on a, uh, a capabilities team doing reinforcement learning to create agents. AI is, in fact, getting, uh, a lot better, and I can ex- say more about why. You know, scaling laws, um, deeper neural nets, bigger, trained on more data, become more efficient, more competent at those things. I also became a bit more disillusioned with the AI industry. So OpenAI, Anthropic, and DeepMind all had these sort of founding narratives of, like, "Yes, these risks are real, but we've thought about them, and we're going to try to handle them responsibly, and that's why it's important for us to keep doing what we're doing." And I increasingly came to think that these were rationalizations to justify what they were doing rather than sort of, like, deeply guiding their actual behavior and that when push comes to shove, they'll follow their incentives rather than do what's actually good.

   4. SBSteven Bartlett11:43

      So you're inside OpenAI at the time, and you start to believe that they're following commercial incentives versus the, I guess, social ins- or societal incentives that they founded themselves on.

   5. DKDaniel Kokotajlo11:53

      Sort of. I mean, I would, I wouldn't actually describe it as commercial incentives. I think I would describe it as, um, power-seeking incentives. So-

   6. SBSteven Bartlett12:02

      Hmm

   7. DKDaniel Kokotajlo12:02

      ... like, it's true that the companies care a lot about making a lot of money, but especially at the very top of these companies, like the leaders They understand that this is about more than just money, you know? There are these emails that came up in, you know, the, the lawsuit between Musk and, and, um, OpenAI. A bunch of emails were surfaced in that lawsuit, which you can go read, and in some of them, the founders of OpenAI were talking back in, like, 2017 about how the reason why we made OpenAI was because we were worried that Demis Hassabis at Google was gonna become dictator with AGI. Even back then, they were-- This is obviously about more than just money. Like, these, these powerful CEOs are literally afraid that if the other guy gets there first, he might become dictator, and they don't trust each other, and so that's why they are racing as hard as they can so that they're the ones who get there first, so to speak.

   8. SBSteven Bartlett12:55

      Have you met Sam Altman?

   9. DKDaniel Kokotajlo12:57

      Yeah.

   10. SBSteven Bartlett12:58

       And did, did that shape your opinion of his incentives or what, why he's doing what he's doing? 'Cause there's a lot, you know, speculated about what his incentives are. I mean, his most recent narrative says, uh, for the good of humanity, I think that's what-

   11. DKDaniel Kokotajlo13:09

       Yeah, I mean, I think the main thing I've learned is don't pay attention to the narratives. [chuckles] You know, like, uh, what they say to one person is just different from what they can say to some other person at the same time, and what they say in public is a third thing entirely. I think you should judge people by their actions, not by their words.

6. 13:24 – 15:53

   Why He Left OpenAI

   1. SBSteven Bartlett13:25

      And why are you no longer at OpenAI?

   2. DKDaniel Kokotajlo13:27

      Largely for the reason that I mentioned, so I became gradually disillusioned with how the company was going to behave. For example, when I first joined in 2022, at least the people I talked to, my colleagues at the company, there was this general sense of like, "Of course, we wouldn't actually just build superintelligence as soon as possible. Once we started getting really close, like, once we started getting to AIs that could maybe automate the AI research process, we would pause and figure out how to make it safe. That's 'cause we're the good guys, and that's obviously the safe thing you should do rather than just going full speed ahead. But we're worried about other people who might not pause, you know, our competitors, Google, for example, and so that's why we need to be in the lead so that we have that room to do the safe stuff," right? That was sort of, like, a thing that seemed like maybe, like, the median position or something among the p- colleagues I talked to when I was there when I started, including people like Sam, you know, including the leadership. And then by the time I left, I was like, "Oh, man, they're really not gonna do that, are they?" Like, [chuckles] like they, they've sort of, you know, partly because this has become more politicized, and they've become bigger and been under more scrutiny, people have started asking, like, "Why are you doing this in the first place if it's so risky?" And so they've pivoted their narrative to being more like, "Actually, it's not that risky," you know? Um, and so yeah, I mean, it seems like they're just going to keep going roughly as fast as they can and hope that they can figure it out on the way.

   3. SBSteven Bartlett14:47

      How did your time at OpenAI come to an end?

   4. DKDaniel Kokotajlo14:50

      Uh, I resigned in 2024. Had a nice goodbye party.

   5. SBSteven Bartlett14:54

      What were the reasons you gave for quitting OpenAI?

   6. DKDaniel Kokotajlo14:57

      I thought that we were rationalizing too much, and that we needed to think more about what would actually be good for the world. Um, I wanted more freedom to publish. So at OpenAI, as it became a bigger company, it became more of a normal tech company with incentives and, you know, a PR department and things like that. And so it started becoming more difficult to, um, to publish the sort of research that I was doing. For example, those scenarios that I mentioned, couldn't, uh, couldn't publish those, right? They're just for internal use. I thought that that was a shame because right now, most of the world is kind of asleep at the wheel and doesn't really realize what's going on with AI and doesn't really realize what's coming in the pipeline a couple years from now, and the companies aren't really incentivized to tell people that much about it. I mean, they say some vague stuff in a sort of hype-y way, but, um, you know, well, they didn't want me to publish [chuckles] the scenario, for example, laying out, like, here's how things might

7. 15:53 – 17:16

   What It Was Like Inside OpenAI During ChatGPT's Launch

   1. DKDaniel Kokotajlo15:53

      actually look.

   2. SBSteven Bartlett15:54

      I'm just c- I'm super curious as to what it's like being in a company like that when they, you know, ChatGPT-3 is released. You were there at that time, right?

   3. DKDaniel Kokotajlo16:01

      Mm-hmm.

   4. SBSteven Bartlett16:02

      Um, which was a moment where I think the whole world stood up and realized that this technology was powerful.

   5. DKDaniel Kokotajlo16:08

      Yeah.

   6. SBSteven Bartlett16:08

      Um, and the conversation really began from a society level. Um, the company starts growing super quickly.

   7. DKDaniel Kokotajlo16:14

      Yeah.

   8. SBSteven Bartlett16:14

      Quicker than I think anybody could ever have imagined. And what, what was it like inside there? What did you see change, um, over, over that period of time?

   9. DKDaniel Kokotajlo16:23

      I remember one all-hands meeting where Ilya said something like-

   10. SBSteven Bartlett16:26

       Ilya being?

   11. DKDaniel Kokotajlo16:26

       Ilya Sutskever, who was, um, head of research at that time. He said something like, "Okay, now the world is starting to pay attention. Each of you is gonna be the most popular person at every party, uh, for the next year. Don't let it get to your head. Focus on the mission. Gotta build AGI." [chuckles] The company grew a lot. It already wasn't really feeling like a nonprofit when I joined, but it definitely didn't feel like a nonprofit by the time I left. Um, lots of new people came in. Ironically, the, like, amount of conversation about superintelligence and the implications of superintelligence arguably sort of went down over time due to this growth, right? So because r- the company would, like, double and then double again and then double again, all these new people were coming in from other parts of the tech industry who hadn't really been thinking about these things and were attracted by the

8. 17:16 – 19:30

   The $2 Million NDA Controversy Explained

   1. DKDaniel Kokotajlo17:16

      high salaries.

   2. SBSteven Bartlett17:16

      You lost $2 million for not signing an anti-disparagement clause, which would mean you could speak... You couldn't criticize the company.

   3. DKDaniel Kokotajlo17:25

      Uh, yes. Well, s- so, um, I got to keep the money.

   4. SBSteven Bartlett17:28

      Oh, you got to keep the money?

   5. DKDaniel Kokotajlo17:28

      Yeah. So what, what happened was after I had left, said my goodbyes, et cetera, um, I got the, the exit paperwork, and it included this clause that said you basically have to agree not to criticize the company again, um, and also a clause saying you can't tell anyone about this. And so I thought that was kind of rich coming from a nonprofit that's supposed to be, [chuckles] you know, for the benefit of all humanity. So I didn't sign it. And if you don't sign, you don't get to keep your equity. So your compensation, you know, what, what they pay you is a bunch of money and then also a bunch of stock, basically. But then they had this stuff in the contract that they get to yank back your, your stock if you don't sign this thing. Um, and my wife and I, you know, were- Uh, upset about this. We talked about it for, like, a month or two, consulted some lawyers, um, and then ultimately decided to just refuse to sign.

   6. SBSteven Bartlett18:22

      Which would mean you lost, you would have lost $2 million.

   7. DKDaniel Kokotajlo18:25

      That's right, which was, like, 80% of our net worth at the time. Um, fortunately, uh, it didn't go the way we expected. It blew up basically on the internet. Like, when people heard that, that we had done this and that we had said no, it became, like, this huge scandal. Employees at the company started, like, asking questions in Slack and, like, asking leadership, like, "Wait, what?" Like, "Why are you gonna take away our equity? What is this?" You know, 'cause a lot of people hadn't really noticed this before. It had been whispered about, but it hadn't been sort of like a thing that most employees knew about. Um, and so they backtracked, and they said, "Never mind, never mind. We'll change the paperwork. You can keep the equity. It's fine."

   8. SBSteven Bartlett19:02

      And Sam Altman came out and said he was embarrassed that he didn't realize this was going-

   9. DKDaniel Kokotajlo19:06

      He did, yeah. He had no idea, apparently.

   10. SBSteven Bartlett19:08

       You don't believe him?

   11. DKDaniel Kokotajlo19:09

       No. I think he probably knew. And if he didn't know, then people close to him probably did, such as his head lawyer.

   12. SBSteven Bartlett19:15

       Why did you decide not to take the $2 million?

   13. DKDaniel Kokotajlo19:19

       I mean-

   14. SBSteven Bartlett19:20

       Most people would have, I think.

   15. DKDaniel Kokotajlo19:22

       It's true, most people would have, and most people did. And, you know, money is nice, but, like, it's not the only thing, you know? Sometimes

9. 19:30 – 24:19

   Is Full AI Automation Coming Faster Than We Think?

   1. DKDaniel Kokotajlo19:30

      it's good to take a stand on principle. I, I keep mentioning superintelligence. Perhaps I should say more about, like, the, the sequence of events that the companies are planning to do. So right now they're focusing on automating coding. They're taking their AIs, they're making them bigger, they're training them for longer, and they're especially focusing the training on getting them to be good at autonomously writing and editing code because, uh, that will help the companies go faster, right? If they can automate the code, then they can do their own work better and faster and accelerate progress. The next step, which they've already begun, is to look at the rest of the research process as well, coming up with ideas, um, analyzing experiments, communicating those results. All the other parts of, of the research process, they're trying to figure out how to train AIs to be good at those as well so that they can have AIs do the entire thing autonomously.

   2. SBSteven Bartlett20:24

      When you say do the entire thing, what do you mean do the entire thing?

   3. DKDaniel Kokotajlo20:27

      So, like, Anthropic and OpenAI in particular are trying to automate themselves. Like, they're trying to make it the case that, um, they don't really need human employees anymore. Uh, they just have a giant army of AIs that's churning away, doing all this autonomous research to make better AIs, to train the new AIs, put them in charge so they can make even better AIs and so forth. And of course, not just, not all just happening internally, but also, like, interfacing with the world, right? Like going out and talking to people, collecting the data, setting up the training environments, doing the business deals and so forth. Like, they're, they're trying to automate all of that. The reason why they're doing this is because they're trying to get to a position where they have AIs that are superhuman at everything, superintelligence, and they're trying to get there before their competitors do. Needless to say, this is incredibly dangerous, [chuckles] I would say, you know? And in addition to being dangerous, it's a power grab, right? Like, if they actually succeed at this, then they'll be sitting on top of this army of superhuman AIs that will give them immense leverage over all sorts of other actors in the economy. Insofar as they can work out something with the presidents and, you know, integrate it into the military or whatever, then that would give the US immense hard power over all other countries, right? Obviously, nobody knows exactly when this is happening, but a very disquieting thing has happened over the last year to me, which is that when we published AI 2027, people were generally of the opinion that my timelines were too short and that, like, probably it would take more than 2027 until we got to the sort of events that I was just mentioning. You know, uh, recursive self-improvement, AIs automating the whole research process, superintelligence. These, these types of milestones, um, they happen in 2027 in AI 2027.

   4. SBSteven Bartlett22:18

      Which is this research paper you published.

   5. DKDaniel Kokotajlo22:19

      That's right. It's, it's a scenario forecast that sort of lays out, like, month by month a possible future trajectory. There was sort of like, at the time that we started writing, it was my best guess as to what would actually happen. Obviously, there's lots of uncertainty, but, you know, I, I, I thought it was valuable to make a concrete guess just to sort of see what it might look like. And at the time we were writing this, a lot of my friends in the AI industry and in nonprofits and so forth that work on AI, a lot of people were saying like, "Yeah, that stuff's gonna happen, but, like, it'll probably take a couple years longer than you think." And now it's more 50/50, especially when I go talk to people at Anthropic and OpenAI. They're often like, "Yeah, no, 2027, that's basically what's gonna happen, just like you wrote. Why did you, [chuckles] why did you become... Why did you update your timelines?" Oh, yeah. C- context for this is, um, after, after writing AI 2027, I shifted my timelines to be a little bit more conservative. So at the time that we published, my 50% mark was in 2028, not in 2027. And then after we published, progress just seemed like it was going a bit slower, and so I updated to 2030 mark, which is, you know, still could happen sooner, could happen later, 2030. Um, but now when I talk to people in, in the companies, they're like, "It's not gonna take that long." They're like, "Oh, you need to shorten them again. Like, get them back to 2027 or 2028," you know? Um, so that's a bit disquieting. Um, again, don't know how long it's gonna take, but this is the stated plans of the AI companies is to do this incredibly dangerous thing, and they think that they're just a few years away.

   6. SBSteven Bartlett23:53

      So you wrote this, um, report here, What 2026 Looks Like, and you wrote this in 2021.

   7. DKDaniel Kokotajlo23:59

      Mm-hmm.

   8. SBSteven Bartlett24:00

      And it was remarkably accurate, helped make a name for yourself amongst, um, amongst e- everybody in AI. And I... Which one was it that JD Vance, the vice president, read? I think it was this one, wasn't it?

   9. DKDaniel Kokotajlo24:10

      This one, yeah.

   10. SBSteven Bartlett24:10

       Yeah, this one.

   11. DKDaniel Kokotajlo24:11

       Yeah.

   12. SBSteven Bartlett24:11

       Um, and then, so then you published this one, AI 2027, and this was published, I believe, in 2025?

   13. DKDaniel Kokotajlo24:17

       Uh, yes, that's right. April

   14. SBSteven Bartlett24:18

       Yeah.

10. 24:19 – 26:33

    The AI 2027 Forecast That Changed the Conversation

    1. SBSteven Bartlett24:19

       What were you forecasting in here? What are, what are the key things that you said in here for people that haven't read it?

    2. DKDaniel Kokotajlo24:24

       The high level version of it is they automate the coding, then they automate the rest of the research process, then the pace of progress accelerates dramatically. They get to superintelligence. They're working with the government who, specifically the president, the executive branch naturally wants to control this technology and otherwise wants to use it to beat China and integrate it into military and so forth. By this point, it's sort of doing basically all the work itself. I mean, it's, it's superintelligent, so it's coming up with all these great ideas for how to integrate itself into everything and all these new technologies it's invented and so forth. And, uh, because of the race dynamics and because of the profit motive, they end up deploying it everywhere, and it builds robot factories that build more robots that build more robot factories, et cetera, transforms the world entirely. And then at some point it has enough power, it meaning the AIs have enough power that they don't have to pretend to, to be aligned anymore, right? Um, then they stop listening to orders. That's the race ending of AI 2027. We also wrote a sort of different branch, which was the slowdown ending, which was intended to sort of illustrate the concentration of power issues, um, that I mentioned previously. So what if hypothetically, the alignment issues get sorted out sufficiently quickly? Like, what if it turns out that like it's not too hard, with two months of s-slowdown, we can figure out how to make the AIs robustly do what we want, um, and have the values that we want them to have. So that's one possible branch, and in that branch, uh, it looks pretty similar. You know, they take the jobs, beat China, et cetera. Um, but instead of the AIs ultimately killing everyone, they create this sort of amazing utopia. But the amazing utopia is whatever the people who control the AIs wanted it to be, right? And so that would be a very small group of people, like the president, some CEOs, et cetera.

    3. SBSteven Bartlett26:16

       There should be a button just down below here, and if it says subscribed, you're already subscribed. If it says subscribe, that means you're not yet. And if you're not subscribed, please could you do us a favor and hit that button? It helps the show more than you know, and according to the algorithm, you're someone that watches our show, but you haven't yet hit that button. Thank you so much.

11. 26:33 – 27:17

    AGI vs. Superintelligence: The Difference That Matters

    1. SBSteven Bartlett26:33

       Is there any possibility, do you think, that we never get to this thing called AGI? And, a-and, and how do we distinguish AGI from this term superintelligence? What's the difference?

    2. DKDaniel Kokotajlo26:42

       Yeah. So the difference is that AGI is a more vague, uh, and weak term.

    3. SBSteven Bartlett26:46

       Okay.

    4. DKDaniel Kokotajlo26:46

       So superintelligence is a bit more precisely defined. It's better than the best humans at everything, faster and cheaper. Um, AGI is more like, it stands for artificial general intelligence, which means AIs that can do things in general rather than like some specific task.

    5. SBSteven Bartlett26:59

       Yeah.

    6. DKDaniel Kokotajlo26:59

       And so arguably, we've already achieved AGI, right? If you use Claude Code or something like that, it's like it can do a lot of stuff. It's, it's almost kind of like a little employee that you can like have go do stuff. So it's, it is quite general. It's not maximally general, though. Can't do everything. Whereas superintelligence by definition can do all the things that a human can do,

12. 27:17 – 30:23

    How Robots Could Soon Become Part of Everyday Life

    1. DKDaniel Kokotajlo27:17

       but better.

    2. SBSteven Bartlett27:17

       And how does this sort of overlap with robotics? Because obviously we're seeing this huge robotics boom at the moment. There are some real world things that humans can still do because these AIs are still stuck on my computer.

    3. DKDaniel Kokotajlo27:28

       The way that people talk about this is that they basically just say, "We've achieved superintelligence for cognitive tasks." Then you can talk about like full superintelligence that can do the physical stuff.

    4. SBSteven Bartlett27:38

       And are we gonna get there? Are we gonna get there with both?

    5. DKDaniel Kokotajlo27:40

       I think so. I mean, again, this is not something that we can be certain about. Um, you asked like, is it possible we'll never get there? Yes, it's possible we'll never get there. I don't think it's likely though. I think that there's nothing sort of like magical about the human brain. It's, you know, um, it's just a bunch of neurons. It is possible for a digital system to do similar functions in the same way that like, you know, a, a plane can fly just like a bird. Not in the same way as a bird necessarily. Like, it doesn't have, it's not flying in the same way that a bird flies, but it flies, you know?

    6. SBSteven Bartlett28:11

       Mm-hmm.

    7. DKDaniel Kokotajlo28:12

       Um, so, so it does seem like, yeah, like seems possible.

    8. SBSteven Bartlett28:16

       You've written all these, you know, these research reports. You're working on another one that'll be released, um, likely on the 7th of July. You have worked inside OpenAI. You then quit OpenAI because you were concerned about what was going on there and about the future of the industry. You know more than I do. Are you optimistic about the future or pessimistic? Are we heading to a bad place if things don't change, um, based on everything that you know?

    9. DKDaniel Kokotajlo28:40

       I think we are headed to a bad place if things don't change. Um, I'm not confident in that. I would say something like 70%. It's very, very hard to predict, of course. But yeah, it seems like the current default path is heading towards a very, very scary place.

    10. SBSteven Bartlett28:53

        How do you contend with that personally and emotionally?

    11. DKDaniel Kokotajlo28:55

        Um, it's rough. I mean, I think it, it's the sort of thing that like gets me down on a regular basis, but also I've been dealing with this for so many years now that I've sort of gotten used to it, if that makes sense. Um, yeah. Yeah. I, I'll put it this way. I would be incredibly happy if all my predictions turn out to be wrong and, uh, an AI hits a wall, for example.

    12. SBSteven Bartlett29:23

        It gets you down on a regular basis.

    13. DKDaniel Kokotajlo29:26

        I used to be known as a pretty chipper and optimistic person, but, um, in 2020, my AI timelines predictions started collapsing due to GPT-3 and the scaling laws papers and, um, the BioAnchors report, which I, I can talk about if you're interested. But basically, some events happened in 2020 that convinced me that actually this stuff was like quite plausibly coming by the end of the decade. And humanity is very obviously not ready for this, you know, in a whole bunch of different ways. And so that's obviously very scary.

    14. SBSteven Bartlett29:56

        And that's an extremely scary world because of all the things you've said, but, but again, because of this recursive self-improvement where AIs can train themselves and at such point we're starting to lose hold of what's going on here.

    15. DKDaniel Kokotajlo30:06

        I mean, the AIs are already training themselves, to be clear. It's more like closing the entire research loop, right? So-

    16. SBSteven Bartlett30:11

        Okay. Doing everything.

    17. DKDaniel Kokotajlo30:13

        Yeah. Like right, right now a lot of the training data is generated by AIs. A lot of the reinforcement, like the grading That happens doling out of positive and negative reinforcement is

13. 30:23 – 36:14

    Why AI Works More Like the Human Brain Than You Think

    1. DKDaniel Kokotajlo30:23

       itself done by AIs.

    2. SBSteven Bartlett30:24

       Can you explain that in layman's terms for-

    3. DKDaniel Kokotajlo30:25

       Yeah. So an important thing for everybody to understand is that modern AI systems are not software in the normal sense. I mean, they are technically software, but they're not lines of code. You know? It's not like some engineers at Anthropic went and wrote lines of code that basically says, like, you know, "When the user asks for this type of thing, then go do this type of thing for this many steps," or whatever. There's nothing like that. Instead, it's a neural net, you know?

    4. SBSteven Bartlett30:52

       What's that?

    5. DKDaniel Kokotajlo30:53

       Well, think about how the brain is a bunch of neurons connected to each other-

    6. SBSteven Bartlett30:57

       Yeah

    7. DKDaniel Kokotajlo30:57

       ... that are firing, um, signals back and forth. The brain learns over time. The types of patterns of firing that caused success, that caused a dopamine rush-

    8. SBSteven Bartlett31:08

       Mm-hmm

    9. DKDaniel Kokotajlo31:08

       ... or various other types of feedback, get reinforced and fire more often. And the types of patterns that caused failure, like touching a hot stove, get anti-reinforced to get, you know, um, destroyed [chuckles] so that they fire less often. And as a result of all of that, you, over the course of years, learn to act in the world, and you learn all sorts of skills, and you learn world models. You learn, like, beliefs about the world, and you can sort of, like, mentally simulate how it's going and stuff like that. So artificial neural nets are like that, except artificial. So it's, it starts off as a giant tangled spaghetti mess of randomly generated, uh, artificial connections called parameters. These days, there might be something like 10 trillion parameters, uh, at, in the biggest AIs. So it starts off randomly generated, so it's, of course, completely useless. Like, if you give it some input, it'll just produce gibberish as an output. But then they train it, and they start with pre-training, which is where you give it a bunch of internet text, and you show it the first piece of text, and you put that in as the input. And then it gives a gibberish output, and then you positively or negatively reinforce it based on how accurate that output was at predicting the next piece of text. Um, so it's basically playing this game of, like, predict the next word.

    10. SBSteven Bartlett32:29

        Isn't that how it happens with babies? I had a, I think I had a neuroscientist-

    11. DKDaniel Kokotajlo32:32

        Mm-hmm

    12. SBSteven Bartlett32:32

        ... tell me that babies have more neural connections, um, than adults, and yeah, it says-

    13. DKDaniel Kokotajlo32:37

        Yeah.

    14. SBSteven Bartlett32:37

        Yep, toddlers have twice as many neural connections as adults, and they, I guess they whittle down through reinforcement.

    15. DKDaniel Kokotajlo32:44

        Yep.

    16. SBSteven Bartlett32:44

        We have more pathways when we're younger, and just like the process of training an AI, we're trained down to, like, remove the ones that aren't useful and build up on the ones that are.

    17. DKDaniel Kokotajlo32:53

        Yeah, it's both pruning and strengthening.

    18. SBSteven Bartlett32:55

        Okay.

    19. DKDaniel Kokotajlo32:55

        And it seems like in humans it's actually more pruning than strengthening, but it's both. Uh, and in AI, it's the same thing. It's both. So the first portion of training is where they train the AI to predict text, which is kind of like training it to read. Um, and it, a, a similar thing does happen in humans. So basically, the, the random tangle gradually takes shape and gradually sort of coalesces into more useful circuitry that has stored lots of facts about the world and has stored lots of skills for how to, you know, process information and transform it and then produce predictions. That's just the first step. After they do the pre-training, then they try to teach it more useful skills besides just predicting text. And so, you know, by the end of the process, they've thrown lots of coding problems at it, and they've said like, "Here's a coding problem. Go." [chuckles] "Here's a coding problem. Here's an environment. You have access to this virtual computer. Here's, like, the code base you're working with. You can write code. You can edit the code. You can run the code. You can read it. You can use the internet. Go, go, go." And it does that for a while, and then based on how successful it is, reinforcement happens, and they have thousands, maybe millions of examples of coding problems like that, that they train it on. And that's why they're so good at coding now.

    20. SBSteven Bartlett34:09

        So what does superintelligence look like in this regard? Is it just more of these connections? And how would they get more connections? Can you explain that to me like I'm 12?

    21. DKDaniel Kokotajlo34:17

        So there's different AI models, right? So there's like, you know, ChatGPT-3 and ChatGPT-4, and GPT-4.5, and GPT-5, and GPT-5.5, and 5.6, right? Sometimes they're just the same previous model but with extra training.

    22. SBSteven Bartlett34:30

        Mm-hmm.

    23. DKDaniel Kokotajlo34:30

        Sometimes they're a whole new model that's been trained from scratch, including starting the whole pre-training process again. Over the last couple years, they've done several new rounds of starting over from scratch, and typically, when they start over from scratch, they make the whole thing bigger, the, the artificial brain much bigger.

    24. SBSteven Bartlett34:46

        Okay.

    25. DKDaniel Kokotajlo34:46

        Right now, they're at something like 10 trillion parameters. Back in 2020, um, it was more like 175 billion, so we've grown like two orders of magnitude, uh, in six years.

    26. SBSteven Bartlett34:58

        Two orders of magnitude?

    27. DKDaniel Kokotajlo34:59

        Yeah, like two 10Xs, so 100X.

    28. SBSteven Bartlett35:02

        Right.

    29. DKDaniel Kokotajlo35:03

        Right?

    30. SBSteven Bartlett35:03

        Right.
14. 36:14 – 41:15

    Can AI Ever Be Truly Creative?

    1. DKDaniel Kokotajlo36:14

       a bird.

    2. SBSteven Bartlett36:14

       Mm-hmm. Yeah, that's a really good analogy.

    3. DKDaniel Kokotajlo36:16

       Yeah.

    4. SBSteven Bartlett36:17

       That, that analogy helped me think through a bunch of questions people often ask about AI when they said, "Can it be creative?" But actually that analogy kind of helps me understand that actually that maybe that's not the question, it's can it produce something that you would consider to be creative?

    5. DKDaniel Kokotajlo36:30

       Yeah.

    6. SBSteven Bartlett36:30

       Because creativity is-- people think of it as like a process.

    7. DKDaniel Kokotajlo36:33

       Yeah.

    8. SBSteven Bartlett36:33

       But actually it's, it's judged based on the output, isn't it?

    9. DKDaniel Kokotajlo36:36

       I mean, you, you can get philosophical about like, is it truly creativity that they have? But you can also be like, well, I mean, just look at all the stuff they're accomplishing [chuckles] .

    10. SBSteven Bartlett36:44

        Yeah.

    11. DKDaniel Kokotajlo36:44

        You know? And, uh, it seems like they're gonna be accomplishing a lot more in the near future.

    12. SBSteven Bartlett36:48

        Yeah, I do-- I, I asked the question about how this weighs on you personally, because I can, I can sense that you're actually personally bothered.

    13. DKDaniel Kokotajlo36:55

        I mean, the-- I think the situation is crazy. Like, first of all, it's very exciting. Like, AI is really fascinating and interesting stuff. I've been following the field for more than a decade now. Um, I've been part of it, uh, for some years and, um, it's really cool, really interesting, and it's really fun to think about what's going on inside these artificial brains and why they are the way that they are, and it's really cool to see all the applications of this technology out in the world. But it really seems like we're on a pretty scary path, and the more you think about it, the more worried you get. And, you know, in stories it always ends well, but this is real life and I, I think we have to sort of stare reality in the face and tell it, and realize that, like, it might not actually end well, you know?

    14. SBSteven Bartlett37:41

        Were there any recent, dare I say, I was gonna say eureka moments, but paradigm-shifting moments where even your own sort of mental model of what's going on here and how this is gonna look were changed for better or for worse?

    15. DKDaniel Kokotajlo37:53

        For better or for worse, and probably for worse, things are kind of on track for AI 2027. There are a few things that have been diff- not exactly like paradigm shift differences, but, like, there have been some differences from what we expected at the time we wrote this. So the government has actually got involved faster than we expected and has been more aggressive than we expected, so the export controls on Mythos being, uh, the biggest example, and also threatening Anthropic with, uh, being destroyed by the Defense Product- Production Act. Um, another thing that's been surprising to us is that Anthropic in particular has gone from second place to first place in the sort of, in the race, basically.

    16. SBSteven Bartlett38:29

        Why do you think that happened? Because it seemed like ChatGPT were out front and clear as it relates, relates to-- OpenAI were out front and clear, but suddenly Anthropic have, uh, lapped them.

    17. DKDaniel Kokotajlo38:40

        Yeah, I mean, I guess they have, um, probably higher talent density, um, and better strategy, but not by a lot, but enough to make the difference.

    18. SBSteven Bartlett38:49

        Why do you think they have more talent?

    19. DKDaniel Kokotajlo38:52

        Well, they don't have more compute. Like the, what are the inputs, right? Like, they're in the lead now. They used to be behind. What are the possible explanations for this? Well, it could have been that they had more resources, like more compute, more money, but that's not true. They have less resources and less money, right? So then I guess talent is [chuckles] what is, is the next best al-alternative.

    20. SBSteven Bartlett39:11

        It's strategy.

    21. DKDaniel Kokotajlo39:11

        You could maybe say strategy.

    22. SBSteven Bartlett39:13

        Focus.

    23. DKDaniel Kokotajlo39:13

        Some, some combination of those things, yeah.

    24. SBSteven Bartlett39:15

        Yeah.

    25. DKDaniel Kokotajlo39:15

        Something that wasn't just, like, the amount of resources they had.

    26. SBSteven Bartlett39:17

        [page flips] Just like John Jones, where marginal improvements in your cognitive performance can have a massive impact, sometimes I podcast for ten hours a day. Over the last couple of weeks, I've been in filming for a TV show, and then I have like one or two days off to get all of my work done, which means there's lots of cognitive load. And so I turn to ketones because I find myself more articulate, able to think more clearly, able to work out better when I'm fueled by ketones. And so the reason I became a co-owner of this company and the reason why they now are a sponsor of this podcast is because I remember one of my team members called Christiana, she tried it once and came up to my desk and she goes, "This is the best product ever made." And I think in part that's because she really cares about those cognitive benefits, as I do, as John Jones does, and as I think most of my listeners probably will. So if you haven't tried these yet, all you have to do is go to ketone.com/stephen, and you'll also get thirty percent off your first subscription order. You'll get exclusive Ketone IQ merch and of course, cognitive benefits that might just change your life. [page flips] Much of the reason most people haven't posted content or built their personal brand is because it's hard and it's time-consuming, and we're all very, very busy, and if you've never posted something before, there's so many factors in your psychology that stop you wanting to post. What people will think of you. Am I doing this right? Is the thing I'm saying absolutely stupid? All of these result in paralysis, which means you don't post and your feed goes bare. I'm an investor in a company called Stan Store, which you've probably heard me talk about, and what they've been building is this new tool called Stanley that uses AI, looks at your feed, looks at your tone of voice, looks at your history, looks at your best performing posts, and tells you what you should post, makes those posts for you. You can also just use it for inspiration, and sometimes what we need when we're thinking about doing a post for our social media channels is inspiration. Building an audience has fundamentally changed my life, and I think it could change yours too. So I'm inviting you to give this new tool a shot and let me know what you think. All you have to do is search coach.stan.store now to get started. [page flips]

15. 41:15 – 47:58

    What Are the Real Odds of Human Extinction From AI?

    1. SBSteven Bartlett41:16

       Uh, a friend of mine who knows some of these people sat me down once upon a time in London. He's actually said this a few times to me, but I remember one particular conversation where he says that some of these AI CEOs predict the probability of extinction at being, I think he said seven percent. I don't know why I have that number in my head, but I remember it being less than ten percent. And the point he was making to me was that even if it was one percent, like if there was a hundred buttons on this table now-

    2. DKDaniel Kokotajlo41:41

       Yeah

    3. SBSteven Bartlett41:42

       ... and one of them would end the world, would I dare press any of them?

    4. DKDaniel Kokotajlo41:46

       Would you press one of the buttons? Yeah, no. [chuckles]

    5. SBSteven Bartlett41:47

       You know? Um-

    6. DKDaniel Kokotajlo41:48

       No

    7. SBSteven Bartlett41:49

       ... I wouldn't press any of them. But he made the case to me that these AI CEOs are very smart, and they understand superintelligence, and that they think actually if there was a hundred buttons on this table right now, maybe ten of them could end the world. I've heard you say, I think it was on the, The, The Daily Show, the interview you did, you said that you think there's a seventy percent chance of human extinction due to AI.

    8. DKDaniel Kokotajlo42:06

       I wouldn't say human extinction exactly. I would say something like seventy percent chance that this goes horribly wrong, like human extinction, but that's just one of several possibilities. But yeah, basically. Like for example, possibly the AIs take over and then don't actually kill everyone. You know? Maybe they do something else. [chuckles] Like just, just 'cause they've taken over doesn't mean they're just- just- definitely gonna kill us, right? They might, but they could do something else. So that's what-- That's, that's why I don't usually say, like- 70% chance of, like, actual human extinction, but 70% chance of, like, something like AI is taking over, some, some sort of very big catastrophe like that that could lead to human extinction.

    9. SBSteven Bartlett42:39

       I guess I got two points there, which is you've been around these CEOs. I mean, you've worked for Sam Altman at OpenAI before you quit. Do you think that they think there's a chance of human extinction?

    10. DKDaniel Kokotajlo42:49

        Yes. But I think that an important thing to understand is that, like, people sort of believe what they need to believe in order to think that they're good people and that they need to keep doing what they're doing. This is what rationalization is. And so I think that the tech CEOs have, like, genuinely convinced themselves that, like, probably things are gonna be fine, and that the way to make things fine is for them to keep doing what they're doing. And, like, they need to, like, make sure that, like, you know, Sam needs to make-- Sam's probably thinking like, "Can't let Dario or Elon [chuckles] get there first," you know. I know Dario's thinking, "Sam can't get there first." Elon's thinking that. Like, you know, they, they, they've all probably convinced themselves that, like, "Oh yeah, like, maybe it'll go horribly wrong, but, like, probably it's gonna be fine, and probably, you know, I should be the one in charge."

    11. SBSteven Bartlett43:35

        It appears to me that Anthropic are the only ones that are talking about the potential chance of extinction or a catastrophic event or, um, the down- the real downsides still. They seem to be the only ones that are still publishing on it, and now they're actually becoming the enemy in many respects of the-

    12. DKDaniel Kokotajlo43:50

        Yeah

    13. SBSteven Bartlett43:50

        ... the tech industry in San Francisco. I'm watching a lot of interviews, and it's, everyone's attacking Dario because he's saying, "Listen, things could go bad." They're calling him a doomer-

    14. DKDaniel Kokotajlo43:57

        Yeah

    15. SBSteven Bartlett43:57

        ... uh, and questioning his incentives. Even with Mythos, which is a, an, a Claude model that they started to warn the world about, again, he is attacked immediately for saying that.

    16. DKDaniel Kokotajlo44:06

        Yeah.

    17. SBSteven Bartlett44:07

        My question is, do you see him as being slightly different from Sam in this regard?

    18. DKDaniel Kokotajlo44:12

        Yeah. I mean, it seems like Anthropic and Dario have been more willing to say and do things that are costly to their bottom line, uh, at least in the last year or so. That's an example of it. Um, like, I don't think that really wins them favors in the administration or among their investors to say that type of thing. And, you know, m- a better example is just the whole fight between the Department of War and Anthropic was an example of them doing something that, like, cost them a lot of money, and even more importantly, cost them a lot of power for something that, like, like, they could have just signed the contract, you know. That said, I really don't want to be in a situation where we're like, "Which CEO is the least bad CEO? Let's support that one." You know? Like, none of these people should be trusted, uh, with that much power, basically.

    19. SBSteven Bartlett44:57

        Nobody should.

    20. DKDaniel Kokotajlo44:58

        Nobody should.

    21. SBSteven Bartlett44:59

        Regardless.

    22. DKDaniel Kokotajlo44:59

        Regardless, yeah.

    23. SBSteven Bartlett45:00

        Mm-hmm. So, uh, on this point of the buttons, you, you, you do believe that they think there's a credible chance of extinction?

    24. DKDaniel Kokotajlo45:06

        Yeah, but they've convinced themselves that, like, "It's probably fine, and also it'll be even worse if I'm not doing it," you know? Like, that, that's what they'll say inside the companies too. Like, the, people will be like, "Okay, well, if we stop, what about the other guys? Like, they're not gonna stop," you know?

    25. SBSteven Bartlett45:21

        Yeah, this is, this has always been why I've had this outstanding question, which is how does this not go bad when human incentives seem to rule the day when you look at history, and all of the human incentives are saying, "Well, if you, you're damned if you do," i.e., you're damned if you carry on developing these bigger and bigger and bigger AI brains, but you're also then damned if you don't from an, a geographical perspective because the United States will lose to that country, or this company will lose to that company. So when you just look at human incentives and goes how does, how does-- if just purely incentives and disincentives, how does this end? Well, it carries on going.

    26. DKDaniel Kokotajlo45:50

        Seems like it. I mean, there, there is a caveat to that, which is a hopeful caveat, which is that, first of all, if the world wakes up to all of this, then there can be a more serious conversation about regulation and international treaties and things like that, and that can change the incentives, right? So the government could come in and say, like, "Actually, here are some rules that you all have to follow," and because they're rules that you all have to follow, then you're not incentivized to, like, break them anymore because you get punished if you break them, and everyone else is also following them too, and so, you know, it's fine. So, so there is that sort of, like, ray of hope that, like, we can change the incentives if the government, and especially the US government, but then later other countries, act to, to change the incentives. But that's not going to happen until people sort of wake up to all of this. The second thing is that even individually, at some point, you know, Dario or Sam or Elon might realize that, like, actually, it's, like, not even in their own interest to, to keep racing unilaterally. And it, it-- well, and the problem with that is it's only if it gets extremely obvious and extremely dire. So, like, in, in AI 2027, in that scenario, there's this choice point that I mentioned, and in one case, the AIs are misaligned, in the other case, the AIs are aligned. At that choice point, we have, like, one branch that depicts the, the misalignment ending and one branch that depicts, like, they, they slow down a bit and solve the alignment issues.

    27. SBSteven Bartlett47:14

        Mm-hmm.

    28. DKDaniel Kokotajlo47:14

        The instigator for that choice point is they see some evidence that their AI might be misaligned and plotting against them, right? So if you actually see that evidence, then it's like, "Oh gosh, uh, maybe we shouldn't put it in charge of everything and let it rip," you know? Because that evidence is staring us right in the face that it's, it's untrustworthy, you know? But if they don't see that sort of very clear evidence, then I think they're gonna convince themselves that they need to keep going, you know. But maybe they will see very clear evidence like that, in which case, even if we don't have regulation, they might just sort of voluntarily stop. Um, so that's the second ray of hope. Like, overall, I don't think that we're, like, definitely doomed, you know. Like I said, 70%, but, like, I could see it working out pretty well

16. 47:58 – 54:45

    What AI Will Do to Jobs

    1. DKDaniel Kokotajlo47:58

       as well.

    2. SBSteven Bartlett47:59

       Hmm. What about, uh, jobs?

    3. DKDaniel Kokotajlo48:02

       Yeah. [chuckles] So I think, I think-- I'm excited to at some point get into the new thing, which is the more optimistic-

    4. SBSteven Bartlett48:09

       Mm-hmm

    5. DKDaniel Kokotajlo48:09

       ... positive vision. Uh, and that will have a lot to say about this because in the, in the, in the prediction, you know, in AI 2027, by the time everyone loses their jobs, there are worse things happening. Or, like, it's, it's kind of like too late by that point. Um, but yes, like once-- if, I mean, just, just think about it. If the companies do manage to build superintelligence, then by definition- They're going to be able to take almost all the jobs or all the jobs, right? 'Cause it's better, faster, cheaper than the best humans at everything.

    6. SBSteven Bartlett48:38

       And that, again, the timeline is by the end of sort of, sort of 2030, you reckon, you think superintelligence might arrive. I'm trying to think about when we could start to see job displacement in the economy.

    7. DKDaniel Kokotajlo48:47

       We're already starting to see a little bit of it now, but not very much.

    8. SBSteven Bartlett48:49

       Why?

    9. DKDaniel Kokotajlo48:50

       Um, 'cause AIs aren't good enough yet. Like they, they're, they're, they're impressive, but they're not like, they're not just a, a drop-in replacement for a human worker in almost any field.

    10. SBSteven Bartlett49:00

        And do you think that will be sudden?

    11. DKDaniel Kokotajlo49:02

        I think it'll be sudden because of the intelligence explosion dynamics or recursive self-improvement dynamics. So you could imagine a different world where it's gradual.

    12. SBSteven Bartlett49:13

        Mm-hmm.

    13. DKDaniel Kokotajlo49:13

        And, and this is, this is maybe how it is in a lot of science fiction is, you know, the AIs gradually get better at a bunch of things and, you know, they gradually automate like this one industry like pharma, and then they automate like, you know, steering drones, then they automate like driving cars or something like that. Um, but what's different about the real world is that the companies have converged on this strategy of automating themselves first, you know, automating the AI research process. And so if they're allowed to continue with this strategy, we're not going to see like, you know, the robo-taxis and like the plumber robots and, you know, the lawyer AIs. We're not gonna see that sort of like broad diffusion of AI into the economy happening first because that's not what they're focusing on first. They're focusing on automating themselves, automating their own research so that they can do everything that they're doing faster, and they want that to sort of get going and get to, you know, very high levels of intelligence, very high levels of general intelligence, um, and then deploy more out into the economy, right? So by the time it's actually coming for like all these different jobs, they will have had fully autonomous AI research happening for months, maybe years, you know? And that means that like the AIs will be vastly superhuman at AI research and probably also vastly superhuman at lots of other things just as a side effect, you know. If you're wondering what this looks like, well, we wrote about what it looks like. It's sort of like this, this wave smashing through the economy after they do the intelligence explosion internally.

    14. SBSteven Bartlett50:52

        What I'm hearing there is that because the AI will be able to improve itself and train itself, it'll be getting better at everything at once, and then it will be released at kind of once. Uh, uh, is that accurate or-

    15. DKDaniel Kokotajlo51:03

        Yes, but it's, it's not, it's not even exactly that because even if it's mostly just getting better at the things that it's doing, like re- research, that'll have some spillover effects to other skills as well. And then when it turns to focusing on the, those, those other skills, it'll be able to do them very fast.

    16. SBSteven Bartlett51:17

        What jobs remain in such a scenario, do you think?

    17. DKDaniel Kokotajlo51:21

        I think that's actually a political question, not a technical question.

    18. SBSteven Bartlett51:23

        Because?

    19. DKDaniel Kokotajlo51:24

        Because on a technical level, all the jobs can be done by the AIs if they've reached that level, and so it's a question of what jobs are allowed for them to do.

    20. SBSteven Bartlett51:35

        And what kind of jobs wouldn't be allowed, do you think?

    21. DKDaniel Kokotajlo51:38

        That depends on who's in charge. So there'd be some sort of political conversation about like what we're gonna allow and disallow.

    22. SBSteven Bartlett51:43

        I mean, in this scenario, the humans are still controlling them, the AIs.

    23. DKDaniel Kokotajlo51:46

        Depends on what you mean by control, right? So there's like, there's, do the AIs actually have the goals and values that you want them to have, and are they going to robustly do that and behave as intended into the future? And then there's like, are they obeying your orders for now?

    24. SBSteven Bartlett51:59

        Are they obeying the orders is really what I'm saying.

    25. DKDaniel Kokotajlo52:01

        Yeah. So like even in AI 2027, in the scenario where the AIs take over and kill everyone, there's a period of like several years where they're still obeying orders and they're, you know, taking some jobs but not other jobs, and they're helping to make better weapons that the US government can use to like do its arms race with China and so forth. And that's why they're able to get so much power so quickly, is because the governments and the corporations and so forth trust them and is deliberately deploying them into all of these positions because it thinks that things are fine. But because these things are neural nets, you can't just like look inside and see what it's really thinking. You can't really tell if it's actually-

    26. SBSteven Bartlett52:42

        I think this is a really important point.

    27. DKDaniel Kokotajlo52:43

        Yeah.

    28. SBSteven Bartlett52:43

        Because unlike software where we can look at the code and see what's going on theoretically, with AI, you're saying that we don't know what, why it's making the decisions that it's making-

    29. DKDaniel Kokotajlo52:52

        That's right

    30. SBSteven Bartlett52:52

        ... 'cause we can't get inside.
17. 54:45 – 1:00:21

    The Skills That Will Still Matter in an AI World

    1. SBSteven Bartlett54:45

       in part, but which jobs are genuinely likely to survive AI, and what skills should people/students focus on over the next 10 years?

    2. DKDaniel Kokotajlo54:53

       That's kind of like, like imagine if you were someone living in Mexico in, like, 1500, and then you hear that, like, the conquistadors are coming. You could be asking yourself like, "Okay, well, what sort of job should I be switching to, to, like, survive this transition?" But, like, you have a lot more to worry about besides that. But yes, I think I would say that, like, if we manage to avoid the loss of control problem, and we end up with humans still in charge of the AIs, and humans can, like, say what the AI's goals and values are supposed to be, even as they become much smarter than humans and even as they run the whole economy, then probably there will be regulation that protects some areas, and you can try to guess at what those areas might be. Maybe stuff that's more like, like, like judges potentially.

    3. SBSteven Bartlett55:41

       What about podcasters? Be honest.

    4. DKDaniel Kokotajlo55:44

       Probably not podcasters, I think. Um, stuff like, you know, being a nanny maybe, right? Like, I think that even if there's a robot nanny that's, like, really, really good, I think a bunch of people might prefer to have an actual human because they might be creeped out by the idea of a really good robot nanny. So you can sort of, you can sort of reason like that. There's also, like, stuff that might be legally protected, like maybe judges, for example, like, are gonna be legally required to be humans and not robots.

    5. SBSteven Bartlett56:10

       [chuckles] Some people say, though, there's gonna be so many jobs created that we can't foresee right now, like there was in the Industrial Revolution or the internet boom or whatever.

    6. DKDaniel Kokotajlo56:18

       The problem with that is that, um, past technological advancements have been more narrow. They've, like, automated some things, but not everything. But we are talking about a hypothetical future situation in which everything gets automated, so there isn't any new job that you could do that the AI couldn't also do. Except if it's, like, protected by regulation or something. That's, that's, that's also a thing. But so, like, for example, right now there's this sort of like cycle where, you know, the AIs learn to do a certain thing, like write copy or, like, draft code or, like, debug something, and then humans who used to do that thing switch to managing AIs or switch to doing the other stuff that the AIs can't do. And that's why there's been this dynamic historically of, you know, new jobs opening up and people flooding to them. But if it gets to the point where the AIs can do everything that humans can do and better and faster and cheaper, then whatever that new job is that you might have switched to, like, the AIs can switch to that too, and they'll already be, be better at it than you.

    7. SBSteven Bartlett57:21

       Because we haven't seen widespread unemployment yet in the economy, do you think people are getting a little bit complacent? Because what I'm seeing on my timeline is a lot of people saying, "I told you so. I told you everything would be fine." And when you look at the, the US unemployment rate, currently the f- it's flat to slightly down. If you look at the UK, it is up. The trend is up compared to last year. We're at about 5% unemployment. The US is at 4.2% unemployment.

    8. DKDaniel Kokotajlo57:44

       Yeah. Basically, nobody has said that there would be mass unemployment by now, or at least we didn't say that, you know. And we were historically one of the more bullish people on AI progress. In AI 2027, because of the dynamics that we just described, the mass unemployment doesn't happen until 2028 or 2029 after they already have superintelligence. Because, again, the companies aren't trying to cause mass unemployment as step one. That's like step three after, [chuckles] you know, it's like step one, automate themselves. Step two, have this recursive self-improvement to get to superintelligence. Step three, expand out into the economy and automate everything. And so this is really unfortunate from humanity's perspective, because one might have hoped that if there was this broad wave of automation going through the economy, people would sit up and pay attention and think about where all this is headed and demand good regulations from the government. But that's not actually what the strategy of the companies are taking. You know, they're going to be getting the superintelligence first and then doing the broad wave of automation, which means that by the time they're actually doing all of that, uh, well, it's already gonna be moving very fast, and the AIs will already be very powerful.

    9. SBSteven Bartlett58:50

       In your 2027 report, so you wrote that in 2025, but it, it's called AI 2027. You said that in mid-2025, we'd have the autonomous employee, which is sort of like-

    10. DKDaniel Kokotajlo59:00

        Mm-hmm

    11. SBSteven Bartlett59:00

        ... AI agents taking instructions over Slack or Teams. That happened. I've actually got an AI agent in my WhatsApp-

    12. DKDaniel Kokotajlo59:07

        Oh, yeah?

    13. SBSteven Bartlett59:07

        ... which I talk to. Of course, yeah, I've got Cl-

    14. DKDaniel Kokotajlo59:08

        Yeah

    15. SBSteven Bartlett59:08

        ... Claude bot exploded obviously around the world and, and now, um, you know, Claude have talked about, uh, their new Slack integration. But lots of people are using agents now.

    16. DKDaniel Kokotajlo59:16

        Yeah.

    17. SBSteven Bartlett59:16

        And that happened, I'd say for us at the t- uh, we really sort of caught onto it the, the start of 2026. You also said that by, by 2026, companies begin replacing entire corporate departments with AI agent subscriptions. 2027, the final job, AI automates the job of the human AI researchers themselves and begins the machine learning research to upgrade and build the next generation of AIs.

    18. DKDaniel Kokotajlo59:36

        Yeah, yeah. So again, timelines. We are uncertain about how long it will take to achieve these milestones. In this scenario, they happen at those times, but by the time we had actually published this scenario, our timelines had shifted back a little bit, specifically mine had. So, like, my 50% mark was 2028-

    19. SBSteven Bartlett59:55

        Mm-hmm

    20. DKDaniel Kokotajlo59:55

        ... for the, for the full automation of AI research milestone, not 2027. Uh, and then other people on my team had more like 2030, 2031, things like that. So I, I, I kinda wanna, like, maybe try to illustrate this with a, you know, remember we have, like, this probability distribution. It's like a smeared out probability mass and, like, the 50% mark is this particular year, but there's, like, a lot of possibility that it happens-

    21. SBSteven Bartlett1:00:18

        Later

    22. DKDaniel Kokotajlo1:00:19

        ... years earlier-

    23. SBSteven Bartlett1:00:19

        Or before

    24. DKDaniel Kokotajlo1:00:20

        ... or years later,

18. 1:00:21 – 1:04:36

    AI 2040: What the Future Could Look Like

    1. DKDaniel Kokotajlo1:00:21

       right?

    2. SBSteven Bartlett1:00:21

       Got you. What is this AI 2040?

    3. DKDaniel Kokotajlo1:00:24

       So AI 2027 was our best guess prediction as to how things would actually go.

    4. SBSteven Bartlett1:00:28

       Yeah.

    5. DKDaniel Kokotajlo1:00:29

       AI 2040 Plan A is our recommendation for how things should go. So we called it AI 2040 because in this scenario, uh, they build superintelligence in 2040. Instead of much sooner because they delay things

    6. SBSteven Bartlett1:00:42

       Why do they delay things?

    7. DKDaniel Kokotajlo1:00:44

       To manage the risks and make sure that power is distributed equitably. They basically, like, regulate AI development so that it still continues, but at a slower, more reasonable pace, uh, in a more transparent and safe way, and spread out over more countries and companies. And as a result, they get to superintelligence in 2040 instead of in, say, 2030. And then we call it Plan A because, well, it's our recommendation. Like, we've, we've come up with a plan for what government should do, and, uh, the scenario is an illustration of what it might look like to implement that plan. In a similar way to how AI 2027 is kind of an, an illustration of what it w- might look like to do what the companies are currently planning to do.

    8. SBSteven Bartlett1:01:26

       Mm-hmm.

    9. DKDaniel Kokotajlo1:01:26

       If that makes sense.

    10. SBSteven Bartlett1:01:27

        And is this wishful thinking, or is this what you think is gonna happen?

    11. DKDaniel Kokotajlo1:01:31

        [chuckles] No, it's definitely not what we think is going to happen.

    12. SBSteven Bartlett1:01:33

        It's not what you think is gonna happen?

    13. DKDaniel Kokotajlo1:01:34

        No, no. What we think is gonna happen is still something more like this, right? We, we don't expect the world to listen to us, right? This is our recommendation, but we, we, we hope that, that people do something like this, and we think it's possible. But it's not our, like, prediction for what's going to happen by default, you know?

    14. SBSteven Bartlett1:01:51

        So I do wanna run through the plans, the potential plans, and also Plan A. But, um, just to close off on how things might look after the... 'cause I think I, I wanted to touch on robotics, too, and I've got this graph here which talks about share of labor output-

    15. DKDaniel Kokotajlo1:02:03

        Yeah, yeah

    16. SBSteven Bartlett1:02:04

        ... um, which I found to be quite striking. I, I've been sat here wondering, as an employer-

    17. DKDaniel Kokotajlo1:02:08

        Mm-hmm

    18. SBSteven Bartlett1:02:08

        ... who employs hundreds and hundreds of people, when, when all this stuff is gonna happen? And, you know, we're still hiring more people as things stand. There are some roles where our consideration is changing, shifting considerably.

    19. DKDaniel Kokotajlo1:02:21

        Mm-hmm.

    20. SBSteven Bartlett1:02:21

        And I'd have to say that, you know, we're probably in the phase where our teams are AI-powered, and they're using agents to do some-

    21. DKDaniel Kokotajlo1:02:26

        Mm-hmm

    22. SBSteven Bartlett1:02:26

        ... of their work now. But I'm wondering, as an employer, like, when is it... whe- when does this happen?

    23. DKDaniel Kokotajlo1:02:32

        Yeah, great question. So if, if we could maybe zoom in on this a little bit.

    24. SBSteven Bartlett1:02:34

        Yeah, we'll put it on the screen.

    25. DKDaniel Kokotajlo1:02:35

        Um, so this is in the AI 2040 Plan A scenario.

    26. SBSteven Bartlett1:02:39

        Mm-hmm.

    27. DKDaniel Kokotajlo1:02:39

        And notably, in that scenario, there's significant regulation introduced in 2029 that slows down the pace of AI development. In the scenario, they do that sort of at the last moment. So in the scenario, if they hadn't done that, then it was about to take off similar to how it does in AI 2027. Um, but as you can see, like, in the scenario, there's still a bunch of jobs at the point that they implement it. And this gets back to what I was saying earlier, is that if you wait until most people have lost their jobs to regulate the AI companies, that's already too late because they will probably already have superintelligent AI by then, because their strategy is to first get superintelligent AI, then do all that stuff.

    28. SBSteven Bartlett1:03:19

        And I think you say that it would collapse the economy. It'd cause even more harm to suddenly regulate something that all of us and all of our lives were then at that point relying on.

    29. DKDaniel Kokotajlo1:03:26

        Oh, but it's a risk well worth taking. I mean, we-- It's true that right now a lot of people use AI for a lot of things, but, like, if we could somehow slow or halt AI development now to set up a better way to do it, that would be well worth it, um, even though there would be significant costs.

    30. SBSteven Bartlett1:03:41

        But you can't over here, right, can you, at this point, where AI and robotics are doing most of the labor output?
19. 1:04:36 – 1:09:33

    The Different Futures AI Could Create

    1. DKDaniel Kokotajlo1:04:36

       it safe.

    2. SBSteven Bartlett1:04:36

       But it's, uh... I, I guess I noticed here that in both your scenarios, eventually AI and robotics do pretty much all the jobs.

    3. DKDaniel Kokotajlo1:04:43

       Yes.

    4. SBSteven Bartlett1:04:44

       So you kind of side there with Elon when Elon says that working will be a choice?

    5. DKDaniel Kokotajlo1:04:50

       Uh...

    6. SBSteven Bartlett1:04:53

       Because, I mean, wouldn't it have to-

    7. DKDaniel Kokotajlo1:04:53

       Yes. I mean, if it-- by definition, if it can do all the things, then it can do all the things. I think that there's a question of, like, should we allow there to be AIs that can do all the things, right? Some people think that the answer is no, and we should just shut it all down and prevent these types of AIs from being created in the first place. And we're actually kind of sympathetic to that. We, we have our... Should we bring up the plans diagram?

    8. SBSteven Bartlett1:05:17

       Yeah.

    9. DKDaniel Kokotajlo1:05:18

       Thanks. Yeah. So our scenario is called AI 2040 Plan A. It's a scenario in which they slow down AI development to make superintelligence happen in 2040 instead of earlier, and Plan A is our recommendation, so this is sort of illustrating our recommendation. But for comparison, we made, like, mini scenarios illustrating different alternative plans, which we call Plan S, Plan B, Plan C, and Plan D. Plan D is basically the same thing that happens in AI 2027. Like, the race continues. There's very little regulation. Um, you can read about that in AI 2027. Plan C, also very similar to what happens in the slowdown ending of AI 2027, where they solve the alignment problems. So in that ending, they, like, slow down a little bit, pivot more resources to AI alignment and AI safety research, get lucky and succeed, and now they have aligned AIs, and then they speed up again and take all the jobs and beat China and all those things. Plan B is... it's kind of like Plan C in that... Well, basically, in Plan B, you're, uh, being more aggressive towards China, and you're, like, taking actions to sabotage or cyberattack them to, like, keep them behind so that you have more breathing room to, to solve the alignment problems yourself. Plan A is our recommendation. It's, uh, domestic regulation and then an international deal to continue building AI, but in a much better way. Plan S is shut it all down. If you want to have a future where- There aren't AIs running around that can do everything better and faster than humans. You kind of want something like plan S.

    10. SBSteven Bartlett1:06:55

        What, what do you want?

    11. DKDaniel Kokotajlo1:06:57

        Plan A is our recommendation. I think that I'm sympathetic to plan S, but for reasons we explain, we recommend plan A instead.

    12. SBSteven Bartlett1:07:05

        And what do you think is most probable, if you're being honest?

    13. DKDaniel Kokotajlo1:07:07

        Plan D.

    14. SBSteven Bartlett1:07:08

        Which is that they just carry on.

    15. DKDaniel Kokotajlo1:07:09

        Which is the AI 2027 type of thing, where they keep racing. They don't really slow down significantly, um, and, uh, things happen extremely fast. The diagram sort of explains, like, roughly the reasoning behind this too. So, like, there's this high level thing of, like, do you want to keep racing as fast as possible to make the AIs smarter and smarter, to put them in charge of more things so that we can beat China? You know? If you're happy with that, then you can get down into this variation of options here. If you are worried about that, well, you get to something like this. There's more different options besides these, but this is kind of, like, the ones that we could compress onto a screen.

    16. SBSteven Bartlett1:07:50

        Do you have children?

    17. DKDaniel Kokotajlo1:07:51

        Yeah, I have two children. It's kind of sad. Like, I think that one way or another, this will probably all be over by the time they're old enough to join the workforce. So I don't think they'll ever join the workforce.

    18. SBSteven Bartlett1:08:07

        When you say this will be all over by the time they join the work- what do you mean by this will be all over?

    19. DKDaniel Kokotajlo1:08:13

        So these milestones that I described, like AIs automating the AI research, AIs getting super intelligent, um, AIs then exploding out into the economy, taking the jobs, building robot factories to build more robots to build more factories, et cetera, GDP starting to go vertical, that sort of thing is what I mean. Like, all of those events transpiring. Maybe there's, like, you know, 10, 20% chance or something that hits a wall and, and none of this comes to pass, even if we don't do anything.

    20. SBSteven Bartlett1:08:43

        How old's your oldest?

    21. DKDaniel Kokotajlo1:08:45

        Six.

    22. SBSteven Bartlett1:08:45

        Six. Boy, girl?

    23. DKDaniel Kokotajlo1:08:48

        Girl.

    24. SBSteven Bartlett1:08:48

        Girl. So your daughter comes to you and says, "Dad, what shall I, um, what shall I study in school?"

    25. DKDaniel Kokotajlo1:08:53

        I mean, again, like, if these radical transformations happen, then the world will just look completely different, and what sort of jobs you set yourself up for basically won't matter that much, probably. I would say, um, that the thing to do is, well, A, try to make it actually go well. Like, if you can exert any influence at all on history and how this all develops, you should be trying very hard to steer the future in better directions. And then separately from that, on a personal level, you should focus on, well, being a good person and doing things that are sort of good in their, for their own sake rather than good because they'll set you up for later employment, because that later employment is gonna be very uncertain, um,

20. 1:09:33 – 1:13:18

    Will AI Become Earth's Apex Species?

    1. DKDaniel Kokotajlo1:09:33

       basically.

    2. SBSteven Bartlett1:09:34

       Elon talks about this age of abundance we're heading towards. Uh, age of abundance sounds-

    3. DKDaniel Kokotajlo1:09:38

       There'll definitely be abundance. The question is, who controls the abundance, and what do they do with it, right? Are the AIs controlled by anyone, [laughs] or are they doing their own thing? And then if they are controlled by people, who controls them, and what do they do? And what's the sort of, like, political structure governing how they make those decisions?

    4. SBSteven Bartlett1:09:58

       I think it was Geoffrey Hinton that said to me, he said there's no example in nature where a more intelligent species is, has less control than a less intelligent species, thus saying that we're quite arrogant to think that in a world where there's this artificial brain that's a gazillion times the size of mine, that I'm gonna give it orders.

    5. DKDaniel Kokotajlo1:10:20

       Yeah. I mean, that, that's the thing is I, I think it's like, that should be our default assumption is that like, well, there's these brains. We can't see exactly what they're thinking. We're gonna make them smarter than us and put them in charge of everything. [laughs]

    6. SBSteven Bartlett1:10:33

       And then we're gonna give them bodies.

    7. DKDaniel Kokotajlo1:10:34

       Yeah, and then they're gonna be autonomously building new factories and so forth. And like, how is this supposed to end well again? Like, isn't this just exactly like us picking a new species that's then going to outcompete us when it doesn't need us anymore? Like, I think that is just the default trajectory. Now, there's a whole argument we can get into about, like, ways that we could get off of that default trajectory. So for example, there's research into interpretability that I described previously, and if that research bears fruit, then you will be able to actually see what they're thinking, and then that would be an excellent tool for shaping them and controlling them and making sure that they do what we want, right? There's other sorts of, um, AI alignment research agendas that are making progress, and if enough of those agendas succeed sufficiently, we can avoid this problem. Of course, also there's the regulatory side too, where, like, part of what makes this difficult is that we're building these AIs in race conditions, you know? Like, the, the companies are secretive about their recipes for making these AIs because it's secrets that they want to protect so that other people can't copy them. And so a lot of it is happening, you know, behind closed doors. Only a few people can really see the recipes f- that they're using to train these AIs and, and so forth, and then oftentimes when the AIs behave in unexpected ways or even just, like, blatantly misaligned ways, sometimes that information doesn't really flow out to the public because the companies are not really incentivized to tell everyone about how they messed up and how their AI is evil. It's just not very conducive to scientific progress on these issues. If the regulatory system was different, then perhaps we could be in a better situation, make faster progress. Also, of course, we wouldn't be planning to put these AIs in charge of everything as fast as possible, [laughs] and we wouldn't be planning to, like, l- let them self-improve, you know? Like, the- these are choices that we could not make, you know?

    8. SBSteven Bartlett1:12:14

       [paper rustles] Xin chào mọi người. Chào mừng các bạn. I don't speak Vietnamese, but this show can because of AI video technology from our sponsor, HeyGen. I get messages every single week from those of you listening to The Diary Of A CEO all around the world, and you express how much impact it's had on you and your life. And if that's true, then those conversations shouldn't only reach people in English. HeyGen can take one recording of me and deliver it in any language while keeping my voice, timing, and expressions intact. But you don't need a studio like this to make it work for you. Record 15 seconds of yourself and get an AI avatar that delivers studio quality video in over 175 languages. We're up to 20 languages now, and we're not the only ones using it. HeyGen is already used by 30 million people, including 85% of the Fortune 100. Whether you're building an audience on social media, launching an online course, or rolling out training across your team, check out HeyGen now. Your first three videos are totally free at heygen.com/doac. That's H-E-Y-G-E-N.com/doac. See you there.

21. 1:13:18 – 1:15:37

    How AI CEOs Really Make Decisions

    1. SBSteven Bartlett1:13:19

       Ilya was, as you said, he was one of the leaders at OpenAI, and he left, and he started his own company, now Safe Superintelligence. Very curious name of a company, Safe Superintelligence, after leaving OpenAI.

    2. DKDaniel Kokotajlo1:13:29

       Yeah.

    3. SBSteven Bartlett1:13:29

       Did you ever get to work with him?

    4. DKDaniel Kokotajlo1:13:30

       Uh, I wasn't directly working with him. I had a couple chats with him.

    5. SBSteven Bartlett1:13:33

       W- w- do you think he's, he's genuinely concerned as well?

    6. DKDaniel Kokotajlo1:13:36

       I think he is, but I think he's, I think he's similar to these other CEOs where-- I mean, just think about the sort of incentives that they're under, right? Like, they can sort of see the problem, and then they can be like, "Okay, but, like, if I don't-- if I stop, if I quit my job and/or do something else, that's not gonna solve the problem 'cause the other CEOs are gonna keep going. And even if all of us didn't go, then maybe China would keep going. So, like, man, it seems like this is just gonna happen one way or another, whether I do anything about it or not. I guess I should be involved, [chuckles] you know, and, like, maybe I can make it go well, and at any rate, like, I don't wanna be out in the cold while these other people I don't trust are in charge of everything." So they all sort of, like, reason through all of this and then convince themselves that, like, the thing to do is for them-

    7. SBSteven Bartlett1:14:21

       To build an AI company

    8. DKDaniel Kokotajlo1:14:21

       ...and to do it better. And I think Ilya is just the latest example of this. Elon's another example. Dario's another example. You know, arguably OpenAI at the beginning, Sam was an example, although, like, Elon and Dario were at OpenAI [chuckles] early on, so.

    9. SBSteven Bartlett1:14:35

       What do you think they should all do then?

    10. DKDaniel Kokotajlo1:14:37

        So I think what should happen is some sort of international regulation or at least domestic regulation similar to what we described in Plan A.

    11. SBSteven Bartlett1:14:43

        Okay. So t- walk me through Plan A.

    12. DKDaniel Kokotajlo1:14:45

        Yeah. So in this scenario, AI takes longer to the, to get to recursive self-improvement and full automation of AI research than it does in AI 2027. We figured that we should try to illustrate, like, a range of different possibilities because we do have those sort of uncertainty intervals. So we chose 2030 as the moment when full automation would finally be achieved and things would really kick off. And then working backwards from that, when's the last moment you could really have good regulation? 2029. So in this scenario, AI progress slows down a little bit naturally, and the AI companies keep, keep racing, but they don't quite succeed in automating, uh, themselves in 2027 or in 2028 or in 2029, but they're getting really close, and they're gonna do it in 2030. And then in 2029, the government steps in and regulates them. What regulations do they do? Well, they basically just shut it down temporarily.

22. 1:15:37 – 1:18:58

    Will AI Decide the 2028 Election?

    1. SBSteven Bartlett1:15:37

       Can I ask, um, how does the elections overlay with your time frames here? Because there's gonna be a big election, isn't there, in 2028?

    2. DKDaniel Kokotajlo1:15:45

       Mm-hmm.

    3. SBSteven Bartlett1:15:46

       And it seems now that sentiment has really, really turned against AI in, in sort of in the general public, and that it will be one of the big-ticket items on the, on the ballot.

    4. DKDaniel Kokotajlo1:15:54

       We think that it'll be maybe the most important issue in the presidential election in 2028. Um, I think a lot of people, m- most people will be quite concerned about where things are headed. And that's part of why we, we chose to depict things the way that we're doing in this scenario, because that helps explain why they might do this sort of regulation in 2029, is that the voters have been demanding it and the presidential candidates have been promising it.

    5. SBSteven Bartlett1:16:15

       And in this scenario and in 2027, would the general public have felt the consequences of AI much more severely than they have now by then?

    6. DKDaniel Kokotajlo1:16:23

       Yes. Although still, even in 2029 in this scenario, they still mostly have the jobs as, as depicted here, right? So in, in 2029 in this scenario, lots of jobs now involve managing AI agents. You, you mentioned you have an AI agent, right?

    7. SBSteven Bartlett1:16:36

       Mm-hmm.

    8. DKDaniel Kokotajlo1:16:36

       Well, in 2029, in this scenario, the AI agents will be much better. Still, though, not enough to just completely do everything. You know, that was the sort of thing that would come in 2030 in this, in this timeline. Again, we're uncertain about timelines. Things could go faster than depicted in this scenario, and in fact, I think things probably will go a bit faster than depicted in this scenario, but we're uncertain. We already did the very fast timeline scenario, so now we're doing a slower timeline scenario. But maybe we should talk about the high-level goals. So they want to have AI continue, but in a slower pace-

    9. SBSteven Bartlett1:17:06

       Who's they?

    10. DKDaniel Kokotajlo1:17:06

        ...so they can make it safe. The politicians, you know, the president and the people who voted for the president and, you know, the heads of other governments and so forth. So goal one, slow things down. Um, goal two, make it more transparent so that the scientific community can catch up to this stuff and make more progress, and also so that we don't have to take the company's word for it when they say that their systems are safe and when they say that they haven't, you know, put in any biases into their systems, for example. That's a concentration of power issue. We also want to avoid a situation where there's an intense concentration of power. So in addition to these, the transparency and the slowdown, we actually think it's actively good for there to be multiple AI companies across multiple different countries that have similar levels of very advanced AI capability and for there to be, like, broad diffusion of AI into society rather than, you know, a single mega project that has all the best AIs, for example.

    11. SBSteven Bartlett1:18:00

        Mm-hmm.

    12. DKDaniel Kokotajlo1:18:00

        And the nice thing about that is you kind of get that by default if you do the first two things. If you slow it down and if you make it more transparent, then that means there's breathing room for other projects to sort of catch up, right? And the transparency just, like, literally helps them catch up because then they can, like, copy, copy some of the ideas. And then I think the fourth thing would be reversibility. So in what follows in this scenario, we are going to be building up a lot of data centers and a lot of robots. We're gonna be transforming the world at a, at a sort of, like, slower pace. Though still a very fast pace, but slower. And if things go wrong and the deal breaks down and everyone starts racing each other again to get to superintelligence as fast as possible, that would be very scary. And so the fourth principle is basically build the new data centers in such a way that if everything breaks down and everyone starts racing again, the newly built data centers get destroyed so that we're sort of back to square one again instead of in an even worse race where there's even more AIs and robots and compute everywhere.

23. 1:18:58 – 1:22:11

    Is There a Safe Path to Accelerating AI?

    1. DKDaniel Kokotajlo1:18:58

       Um, so I can sort of walk you through the timeline if you're interested.

    2. SBSteven Bartlett1:19:01

       Sure.

    3. DKDaniel Kokotajlo1:19:02

       Or the president talks to China, talks to the leaders of a bunch of other countries and says, "We're going to basically halt AI development until we can figure out a s- a plan for how to do it in w- in ways that achieve these goals." So they basically send inspectors to each other's data centers, like Chinese inspectors come to US data centers, US inspectors go to Chinese data centers, and verify that they are doing inference and not training. Developing new AIs, that's That involves training them. But just taking existing AIs and using them to serve customers, that's called inference. And so the sort of like solution they come up with here in this scenario is, we'll allow them to keep doing inference, but not training for now until we can get the new training data set, center set up. So they retrofit the existing data centers to serve inference. People can still keep talking to their AI agents, but they're gonna stop getting better and better for like six months to a year while they build the new data centers that are going to be the transparent data centers, and that's where the training's going to happen. Once they get those new data centers set up in 2030, then AI research continues. This is a bit spicy. We advocate for total research transparency, which means that on the training data centers that are training the new models, they basically have to publish everything. Which means you get to see all the details of the recipes for training these models. You get to see the architectures, et cetera. We think that sort of open science is really important for solving the alignment problem fast enough, because you don't wanna have these sort of biased companies making the decisions about whether the AIs are safe. Um, and we also think it's important for just good regulations more generally, because right now, most of the expertise in the world on AI is sort of concentrated in Silicon Valley, and the, the governments in particular kind of are-- don't really understand AI that well.

    4. SBSteven Bartlett1:20:47

       Mm-hmm.

    5. DKDaniel Kokotajlo1:20:47

       And imagine an alternative. Instead of total research transparency, you had like an auditor system where the government says, "Here are some rules for how to make the AI safe," and then we're gonna have like an agency that like goes into the companies and asks them questions and tries to make sure that they're following the rules. That creates this sort of adversarial dynamic where the company is incentivized to, like, fool the re- [chuckles] the regulator.

    6. SBSteven Bartlett1:21:09

       Yeah. Yeah.

    7. DKDaniel Kokotajlo1:21:10

       You know? And, and also if they, if they discover some new problem that's not even on the government's radar, they might be incentivized to, like, not tell the government about it, right?

    8. SBSteven Bartlett1:21:18

       Yeah.

    9. DKDaniel Kokotajlo1:21:18

       So if you have the total transparency, it helps the government make better decisions faster.

    10. SBSteven Bartlett1:21:22

        But it kills their competitive advantage.

    11. DKDaniel Kokotajlo1:21:24

        Yes. Anthropic's not gonna like this. [chuckles] You know, OpenAI's not gonna like this. This would be, uh, probably bad for the valuations. I don't think it would kill them completely, but it means that it would commoditize more, right? So it means that there'd be like a bunch of AI companies that would catch up to the frontier. They would train AIs that are, like, roughly similar, roughly equivalent. They could still make money by doing that and then selling their AIs, but they wouldn't have a monopoly. They wouldn't have anything close to a monopoly, which I think is good for humanity, although it's bad for the bottom line of those particular companies. Notably, it's good for the bottom line of lots of other companies. Like, if you're a company that's behind and you don't, you're not Anthropic or you're not OpenAI, then you would love this because this helps you catch up. You know, or this, this helps you to, like, um, capture more of the value from the chips you're selling, for example, or from the, like, downstream product

24. 1:22:11 – 1:25:15

    By 2031, AI Could Do 20% of Cognitive Work

    1. DKDaniel Kokotajlo1:22:11

       that you're making.

    2. SBSteven Bartlett1:22:11

       And by 2031, then you have one-fifth of all cognitive labor done by AI.

    3. DKDaniel Kokotajlo1:22:17

       Yeah. So what's happening here is that we're imagining that the government of the United States and the government of these other countries that are involved in this agreement that are sort of implementing similar regulations, um, they don't have to be exactly the same. Uh, but that's another thing that's nice about the transparency, is that if you have this sort of transparency, then if two governments are implementing different regulations, like if one of them is, like, telling their companies to go slower or, like, banning more stuff than the other one is, they can both see.

    4. SBSteven Bartlett1:22:45

       Yeah.

    5. DKDaniel Kokotajlo1:22:45

       Like, "Oh, you're letting them do that sort of thing and you're not? Like, maybe we should let them do this too," [chuckles] you know? So it, it helps to sort of naturally equalize the regulations to some extent without having there to be a central power that just gets to make regulations for everybody.

    6. SBSteven Bartlett1:22:59

       Mm-hmm.

    7. DKDaniel Kokotajlo1:23:00

       So anyhow, we're imagining that when they, when they get this transparency set up, they basically agree to ban the dangerous stuff to allow the not so dangerous stuff, and there's a constant ongoing conversation about, like: Well, what's dangerous and what's not? What should we ban? What should we allow? What about this country? What about that country? That conversation evolves over time, but the gist of it is, a- at least if they do it the way that we recommend it, is that they don't do an intelligence explosion. They don't let the AIs, you know, autonomously self-improve. Instead, they slowly and carefully scale up the AIs that they currently have and invest lots into finding ways to make them more interpretable, uh, to make them more easy to control, to understand better how they work and so forth. The result is that AI progress continues, but it's not quite as fast, and it's much, much, much safer and more transparent.

    8. SBSteven Bartlett1:23:45

       But still through these, y- you know, we see job disruption.

    9. DKDaniel Kokotajlo1:23:48

       It is continuing 'cause they are building more data centers, right? Like, this whole time, they're building more and more data centers, more and more chips, and they're continuing to, like, make there be a, a larger and larger population of AIs, so to speak, and that causes this huge transformation over the course of the 2030s. Sort of a big thing that we sort of want people to take away is that even if you heavily restrict AI progress, you still get this sort of crazy transformation. You know, in this scenario, they basically allow progress to continue, but at a slower, more safe pace here in 2030, and then as a result, it takes until 2035 to get to top expert-level AI. So remember, they were on track to do that in 2030, but then sort of at the last moment they stopped. But because it was sort of so close to the last moment, that means that, like, they can sort of get there pretty soon if they want to, and it's just a matter of, like, how long they, they allow it to go, right? So they sort of, they sort of slow it down, spread it out, leisurely arrive at this level after five years. By this point, they've built up massive amounts of data centers everywhere. So it's not just that the AIs are smarter and able to do all the things that humans can do, but also there's a lot more of them, and there's a lot of robots and so forth. So by this, by this point, you kind of have the economy that a lot of people would have imagined with AGI, where there's AIs, there's lots of them, they're able to do all sorts of jobs. There's robots, there's lots of them, they're able to do all sorts of physical work, and basically the economy is being run by these machines.

25. 1:25:15 – 1:32:41

    Should Everyone Receive AI Dividends?

    1. SBSteven Bartlett1:25:15

       So in 2031, you, you have the one-fifth of all cognitive labor done by AI. In 2023, you have 60 million AIs running at 100X speed. In 2033, there's cash dividends to all Americans.

    2. DKDaniel Kokotajlo1:25:29

       Mm-hmm.

    3. SBSteven Bartlett1:25:30

       Um, I've got a- Explain, explain this to me.

    4. DKDaniel Kokotajlo1:25:35

       Yeah. So if the AIs are gonna be taking people's jobs-

    5. SBSteven Bartlett1:25:38

       Mm.

    6. DKDaniel Kokotajlo1:25:38

       ... then it's very important that people not starve to death and still have money. And if companies are going to be using AIs and robots to take all these jobs, then that means that there needs to be some sort of taxation scheme or something to, like, make sure that people still have a, a slice of that pie.

    7. SBSteven Bartlett1:25:55

       Mm-hmm.

    8. DKDaniel Kokotajlo1:25:55

       The pie is gonna grow huge, but you still need to actually give people a slice of the pie. And our proposal for how to do that, we call it the citizens' dividend. Basically, people have shares in a agency that sells permits to the robot companies and to the compute companies, and makes profit from selling those permits, and then those are-- people have shares in that entity. It starts off small. It starts off something like $25,000 per person, uh, and then by the end, it's something like ten million dollars per citizen.

    9. SBSteven Bartlett1:26:26

       Per, per person?

    10. DKDaniel Kokotajlo1:26:27

        Per person, per year.

    11. SBSteven Bartlett1:26:29

        Factoring in inflation? Like-

    12. DKDaniel Kokotajlo1:26:30

        Yes

    13. SBSteven Bartlett1:26:30

        ... what do you mean?

    14. DKDaniel Kokotajlo1:26:31

        Factoring in inflation.

    15. SBSteven Bartlett1:26:31

        So we're all gonna be m- multimillionaires?

    16. DKDaniel Kokotajlo1:26:33

        Yes. If this happens, which it probably won't. But if it happens, this is where it would go. And again, this is the thing I wanna emphasize, is that if you get to the point where your AIs are close to being able to do all the research, and then you sort of pause and slow down, that means that, like, you still have a lot of transformation ahead of you. Because if you allow those AIs to, like, still proceed slowly and, like, start to automate various jobs and so forth, after some years, they will in fact have done that. And they will have, you know, built huge amounts of new data centers, huge amounts of new chip fabs, huge amounts of new robots, robot factories, et cetera. You know, we're not sure obviously how fast this will go exactly, but we've thought about it a lot, and we have our, our guesses, and this is sort of like our median guess.

    17. SBSteven Bartlett1:27:13

        What does this mean, 2037: The Apocalyptic Arrival of Truth on Earth?

    18. DKDaniel Kokotajlo1:27:18

        Yeah. So, like, this is the point where we say they get to top expert level AI. So it's not super intelligence in the sense that it's not, like, vastly smarter than humans at things, because they deliberately pause it at the level of top experts. So, so here they're going slow. Here they've just actually stopped, but they've stopped at a point where the AIs are just actually really good at everything. So kind of... They've definitely got AGI, maybe they got, like, weak superintelligence. Because they have so many of these AIs, and because they think faster than humans, you know, they just run much faster, that's going to transform society dramatically. So we talk about some of the ways in which it transforms society. Like, this is sort of life after work. We talk about what it would be like to be living on your citizens- citizens' dividend and not have a job anymore in this sort of world. Um, here we talk about all the scientific changes and all the social changes that would come from all of the intellectual progress and activity that would be generated by all of these AIs. So for example, here is things like cancer cures and, like, you know, people living in apartments that were built by robots two years ago.

    19. SBSteven Bartlett1:28:21

        That's tw- 2036.

    20. DKDaniel Kokotajlo1:28:22

        Mm-hmm.

    21. SBSteven Bartlett1:28:23

        Providing, again, we stop in 2029.

    22. DKDaniel Kokotajlo1:28:25

        Yeah.

    23. SBSteven Bartlett1:28:26

        And providing, I mean, a conservative, this is a conservative timeframe.

    24. DKDaniel Kokotajlo1:28:30

        Yeah. Like, unfortunately, I actually think that things will happen faster than this by default, and that if we don't slow down, things will happen much faster than this. Once you get to the point where you've got, you know, a billion AIs running day and night, and they're each better than the best humans at everything, and so they're doing a lot of science, they're doing a lot of talking to each other, they're doing a lot of thinking, everyone's constantly talking to their AI assistants and so forth, there's gonna be a lot of scientific progress. There's gonna be a lot of changes to politics, to ideologies. It's gonna be very disruptive and crazy, and we get into some of the ways in which it is, uh, later, basically.

    25. SBSteven Bartlett1:29:05

        I- I'm still, still not super clear on what this means, the apocalyptic arrival of truth on Earth. It's just, it's just because there's so many AIs, AIs that are so smart that they're uncovering, making new discoveries in sciences.

    26. DKDaniel Kokotajlo1:29:15

        Let me give you an example. Lie detectors.

    27. SBSteven Bartlett1:29:17

        Yeah.

    28. DKDaniel Kokotajlo1:29:17

        So that's an example of a, a technology that might be invented.

    29. SBSteven Bartlett1:29:21

        Yeah.

    30. DKDaniel Kokotajlo1:29:21

        You know? Right now, we don't have good lie detectors. We have very bad lie detectors that, like, sort of work, but don't, don't fully work. But once you've had these top expert level AIs-
26. 1:32:41 – 1:34:11

    What It Will Feel Like to Live Through the AI Revolution

    1. DKDaniel Kokotajlo1:32:41

       happen. So our main scenario is mostly talking about the policy choices made and the broad scale effects on society. We figured it would also be nice to accompany this with a little mini scenario that describes what it would actually feel like to live through this from an ordinary person's perspective.

    2. SBSteven Bartlett1:32:58

       Okay.

    3. DKDaniel Kokotajlo1:32:58

       Um, 2029, everyone's yelling at each other. The presidents are negotiating something, and they've paused AI, but you still have access to the existing AIs, so it doesn't really feel that different, although it definitely is, like, something exciting happening. 2031, they've started progress again. The AIs are really smart. More people have lost their jobs. It's, like, really starting to actually affect things, but I think still most people have their jobs, but their jobs are sort of transformed. So, like, by 2031, it's like most white-collar jobs involve working with AIs to a large extent or managing teams of AIs or collaborating with them somehow.

    4. SBSteven Bartlett1:33:28

       And what will people-

    5. DKDaniel Kokotajlo1:33:28

       Also, there are some things like robo-taxis that are basically just working. Citizens dividend, you know. Ideally, this would happen sooner. Like, i-i- in our scenario, they kind of do things at the last minute [chuckles] . You know? So, like, a lot of these policy things are, like, happening kind of, like, just in time. Obviously, we would recommend that you do them sooner and, and do a better job of them too, but... So 2033, you start getting your, your checks from your dividend.

    6. SBSteven Bartlett1:33:49

       So, so you're forecasting that there will be a citizens check that your model says it could be around 25,000 at the start per person.

    7. DKDaniel Kokotajlo1:33:56

       And then it would grow as the economy grows.

    8. SBSteven Bartlett1:33:58

       But also as, I guess, job displacement takes hold. They're gonna need to, to grow that-

    9. DKDaniel Kokotajlo1:34:02

       Yep

    10. SBSteven Bartlett1:34:02

        ... check to make sure you can grow.

    11. DKDaniel Kokotajlo1:34:02

        And that, and that's why it's kind of last possible moment, because if you waited to implement this until, like, 2037, then, like, everyone would have already lost their jobs by the time

27. 1:34:11 – 1:46:12

    How People Find Purpose After AI Replaces Jobs

    1. DKDaniel Kokotajlo1:34:11

       that happens, right?

    2. SBSteven Bartlett1:34:12

       People losing their jobs, uh, especially if it happens, uh, quickly like, like we see on this sort of graph here, is gonna cause lots of problems in terms of civil unrest, social unrest, purpose, mental health, these kinds of things, theoretically.

    3. DKDaniel Kokotajlo1:34:25

       Yes.

    4. SBSteven Bartlett1:34:26

       How do you think about that?

    5. DKDaniel Kokotajlo1:34:28

       Uh, it's, it's gonna be rough, and hopefully, we can navigate that well. We think that at a high level, people need to have money, and also people need to have power, and I think these are, like, somewhat different things. It's like why are jobs important? Well, there's a lot of reasons why jobs are important, but I think the main ones are, um, well, it's how people get money so [chuckles] so they can survive and get the things that they want by buying the things that they want. So if people are gonna be losing their jobs, you need some other way of people getting money. And then there's also the power thing, which is that right now people have political power in part due to their economic power. People can threaten to go on strike, for example, or, you know, countries that are ruled by dictators can't just completely, you know, genocide an entire subpopulation. Or they can, but, like, it's costly for them to do so because then they'll have less money because that subpopulation is contributing to their economy and contributing tax revenue and so forth. But if you end up in a world where actually nobody's contributing tax menue- revenue except for the AI companies and the robot companies, then you're, you, the government, are less incentivized to care about what, you know, the common people think. So, so when people lose their jobs, they're not just threatened with lack of, loss of income, they're also threatened with loss of political power, and so we think that it's important to, like, do things to push against that.

    6. SBSteven Bartlett1:35:43

       But what does that look like? How do you, how do people have power in such a world?

    7. DKDaniel Kokotajlo1:35:47

       Well, in democracies at least, they still have votes.

    8. SBSteven Bartlett1:35:49

       Okay.

    9. DKDaniel Kokotajlo1:35:50

       So I think that it's very important for there to be, uh, regulations on the use of AI that help make the public discourse more sane and more, um, actually giving the people what is in their interest and what they want and avoiding a sort of, um, opposite outcome where, you know, the m- masses are easily manipulated by AI-powered media, for example. Or where everyone's talking all day to their AI advisors, and the AI advisors are, like, subtly steering them away from voting for the candidate that would not be what the AI companies want because [chuckles] the AI companies have this other candidate that they like better, and they're, like, secretly biasing their AIs to, like, steer people towards voting for that candidate, right? So, so we wanna be in a situation where, um, people have AIs that are actually trustworthy and that are truth-seeking AIs, honest AIs, and that don't have any sort of, like, political agendas put into them by the AI companies or by the government. You know, you wanna avoid a situation where the AI company-- where, where the government has issued some sort of secret order that, like, the AIs have to be such and such a way. You know, the Department of War dispute versus Anthropic is, like, a, an interesting sort of foreshadowing of this, right? Where, um, Anthropic was giving their AIs to the Department of War. The Department of War wanted to use them for certain things and was upset that Anthropic's AIs were, like, not supposed to be used for those things. Uh, the things in particular were domestic surveillance and, uh, autonomous robots.

    10. SBSteven Bartlett1:37:24

        Mm-hmm.

    11. DKDaniel Kokotajlo1:37:25

        There's gonna be a lot more issues like that coming up, and you want it to be the case that, like, people know what they're getting, and that if people are, like, spending hours a day talking to their chatbot, that chatbot doesn't have political biases put into it or a secret agenda or things like that, and instead has been trained to, like, give honest, true answers to things. And I think if you can do that, it can improve the discourse and help people to use their votes to put even better regulations and even better p- politicians in place and so forth. And you can sort of essentially bootstrap this to having something where People's power is even more secure than it is today

    12. SBSteven Bartlett1:37:55

        A lot of the stuff we've, we've covered in part, so, you know, the wars and drones and missiles, we're already seeing this around the world at the moment, which is really, really interesting. Um, and w- we've talked about robots outnumbering humans as well, which is part of this prediction. Some of the ones down here I found to be really curious, which is people will be protected by AIs wherever they go.

    13. DKDaniel Kokotajlo1:38:15

        Mm, yeah. In this scenario, they delay the creation of superintelligence until 2040, and in fact, they pause from 2035, but then they let it go after that, and then they let the AIs become vastly superintelligent. And we think that once the AIs are vastly superintelligent, the world will transform even more radically than what happens in the 2030s in this scenario. So in the 2030s in this scenario, it's more like human level. You know, the AIs are not-- They're, they're doing the same sorts of things that human experts would have done. They're just doing it a bit better, a bit faster, and a lot cheaper, and there's a lot more of them. And the robots are still, you know, doing the same sorts of things that human workers would have done. There's just more of them and they're cheaper. And because of exponential growth, uh, you start with a world that looks not that different from today in 2029, and then by 2039, you end in a world that's radically transformed, where everyone's living in these, like, fancy new apartments that were built by robots two years ago. There's, like, giant special economic zones that are full of robots and solar panels and factories producing more robots and solar panels and factories and so forth. Most of the economy is AIs and robots, and people don't have jobs anymore. That sort of transformation is what you get if you pause at human level. But if you go beyond the superintelligence, there's a whole nother transformation coming that's gonna look more like magic. Think about how the technology of today would look like magic to someone from 500 years ago.

    14. SBSteven Bartlett1:39:37

        Mm-hmm.

    15. DKDaniel Kokotajlo1:39:38

        You know? And that's without even, like, a qualitative improvement in intelligence, right? Like, the humans of today aren't, like, qualitatively smarter than the humans from 500 years ago. It's just that we've had more time to do research-

    16. SBSteven Bartlett1:39:48

        Mm-hmm

    17. DKDaniel Kokotajlo1:39:49

        ... and we have more, like, money and resources to build, you know, prototypes and experiments and run experiments and so forth. But if you had a point where there were billions and billions of AIs that were not only faster than humans, but, like, qualitatively way, way, way better at everything, and in particular at doing scientific research, we should expect that some of the things that they develop will seem like magic to us.

    18. SBSteven Bartlett1:40:11

        Mm-hmm.

    19. DKDaniel Kokotajlo1:40:11

        And we'll just completely, like, we did not think that was even possible, you know? People don't wanna die. People don't wanna be hit by cars. People don't wanna be, like, attacked by a random mass murderer.

    20. SBSteven Bartlett1:40:21

        Cancer's gone.

    21. DKDaniel Kokotajlo1:40:22

        I mean, not just cancer. Like, you know, all, all, a lot of the stuff that happens in science fiction will probably have happened by then. So things like people scanning their brains and uploading into, into computers, right? Or self-replicating robots in the asteroid belt, uh, creating more and more satellites to, uh, produce more and more power to produce more and more self-replicating robots and so forth.

    22. SBSteven Bartlett1:40:43

        Most people still live on Earth, but the trend is to move to space.

    23. DKDaniel Kokotajlo1:40:46

        That's right. Yeah. So, like, if, if you end up in the situation where the entire human economy is just, like, a tiny drop in the bucket that is the entire economy, and it's just, like, this huge amounts of robots and AIs that are moving incredibly quickly, then what you want is Earth to be mostly left as something like a preserve. You know? I think a lot of people are worried about the environment being destroyed, which it totally would be if it wasn't protected. And, uh, you know, there's a lot of people who sort of like their lives as it is and don't want to be uploaded or live in some crazy new future thing. And it seems to us like the reasonable solution to these issues is, uh, create new living spaces off the planet with some of that vast economic wealth and activity that's happening for the people who want that sort of thing. And then that way the Earth can be preserved.

    24. SBSteven Bartlett1:41:41

        Data cen- picture here of data centers in the ocean. Uh-

    25. DKDaniel Kokotajlo1:41:44

        Yeah.

    26. SBSteven Bartlett1:41:45

        I mean, there's three images there of different environments where humans might live.

    27. DKDaniel Kokotajlo1:41:50

        Again, like, our proposal was you preserve, like, 99% of the Earth, uh, mostly as is, as historic or environmental, for its historic or environmental reasons. But then, like, some parts of it you designate as special economic zones where the robots can go crazy and dig giant pit mines and produce factories and so forth. Um, we were thinking it would be good to build the data centers on the ocean instead of, um, on land for a variety of reasons, although later space would be better, and we could see that being reasonable as well.

    28. SBSteven Bartlett1:42:21

        What about immortality in a world of AI? Um, 20-- uh, well, 3045, you say you've lived a dozen lifetimes and are immortal, passing from life to life as if by reincarnation. I mean, there's a lot of billionaires at the moment that are focused on longevity. I mean, Bryan Johnson's said he's got this central rule, which is do not die right now-

    29. DKDaniel Kokotajlo1:42:42

        Yeah

    30. SBSteven Bartlett1:42:42

        ... because we're in the age of AI, and it's conceivable that with superintelligence we'll be able to choose when we die.
28. 1:46:12 – 1:49:56

    Would He Shut Down AI Forever If He Could?

    1. SBSteven Bartlett1:46:12

       You, um, you've spent, what, must be almost coming up to 15 years thinking about this stuff. Um, if this here was a button, and if you press that button, your Plan S would occur, and it would shut down every data center that is currently training a frontier AI model, uh, for good. There would never be any other-

    2. DKDaniel Kokotajlo1:46:35

       Mm

    3. SBSteven Bartlett1:46:35

       ... AI labs, um, working on these problems. Would you press that button?

    4. DKDaniel Kokotajlo1:46:40

       I was, I was about to slam it until you said for good. [chuckles]

    5. SBSteven Bartlett1:46:43

       Oh, okay.

    6. DKDaniel Kokotajlo1:46:44

       Like, I think, I think if it was a sort of temporary shutdown, I would totally slam that button because we are not ready to do this, you know? Like, [chuckles] civilization is not ready to have these companies automate themselves and then get smarter and smarter and then have the superintelligent... Like, no. There's a bunch of reasons why that's really, uh, dangerous. But I would be at least hesitant to press this button if it permanently foreclosed the possibility of ever doing it again for sure.

    7. SBSteven Bartlett1:47:10

       But, but if you think that Plan D is probable, which is this race we're on to superintelligence-

    8. DKDaniel Kokotajlo1:47:15

       If I had a choice between D and S, I think that I would press it.

    9. SBSteven Bartlett1:47:18

       Well, it, it comes down to what you think, right? 'Cause if you think that's, that is what's gonna happen, Plan D, and the only alternative-

    10. DKDaniel Kokotajlo1:47:26

        I didn't say this is what's going to happen.

    11. SBSteven Bartlett1:47:28

        Probabilistically.

    12. DKDaniel Kokotajlo1:47:28

        I said... Yeah, yeah.

    13. SBSteven Bartlett1:47:29

        Yeah.

    14. DKDaniel Kokotajlo1:47:29

        Like, like I'd be like-

    15. SBSteven Bartlett1:47:29

        Yeah

    16. DKDaniel Kokotajlo1:47:29

        ... this is the most likely, maybe this is the second most likely, maybe this is the third most likely. They are all possible.

    17. SBSteven Bartlett1:47:35

        So with your current perspective on whatever one you think is gonna happen, would you press the button? I'm giving you a, an S, a definite S-

    18. DKDaniel Kokotajlo1:47:40

        Yeah, yeah

    19. SBSteven Bartlett1:47:41

        ... or whatever you think is gonna happen.

    20. DKDaniel Kokotajlo1:47:42

        Oof. That's tough. [sighs] What is the scope of the shutdown? So is it-

    21. SBSteven Bartlett1:47:51

        It's no one can train an AI model again, ever again.

    22. DKDaniel Kokotajlo1:47:55

        Ugh. That's real rough 'cause like I said, there's loads of benefits that we could get from AI if we do it right. Um-

    23. SBSteven Bartlett1:48:02

        I think I, I al- I've almost put you in the position of Sam Altman [chuckles] -

    24. DKDaniel Kokotajlo1:48:04

        Yeah

    25. SBSteven Bartlett1:48:05

        ... to some degree.

    26. DKDaniel Kokotajlo1:48:06

        Yeah. Um, let me th- Do you mind if I just take a moment-

    27. SBSteven Bartlett1:48:10

        Yeah

    28. DKDaniel Kokotajlo1:48:10

        ... to think about this?

    29. SBSteven Bartlett1:48:10

        You can think. I prefer you to think.

    30. DKDaniel Kokotajlo1:48:12

        Yeah. I think I would not press the button, but I'm, I feel very torn about it. Um, the reason why I think I would not press the button is that I still have substantial hope that we can get something much better than this, something more like this. And I think that... Basically, I think that if we don't build powerful AI systems eventually, then we're probably going to die as a civilization eventually, you know, like 100 years from now, 200 years from now, something like that, like nuclear war, pandemic, something, you know. I, I don't think human civilization right now is, like, super, super stable. Um, and so I think that basically what I was about to say was the possible benefits for posterity and for all the billions and billions of people who could live in the future outweigh the, like, the current level of risk. But actually-
29. 1:49:56 – 1:57:25

    What Can We Actually Do About AI?

    1. SBSteven Bartlett1:49:57

       So what I, I always think about the personas of, like, the audience that are watching, and these are, you know, they're, they're very curious people, especially on the subject of AI, as we've seen. But they, they wanna know, like, what it means for them. I think a lot of them also wanna know what they can do.

    2. DKDaniel Kokotajlo1:50:09

       Ah, yes. Yeah. What can people do? Well- I think that if you either have talent or passion, you can get directly involved. There's lots of organizations that are worried about these things and that are trying to do something about it, like political advocacy or technical research, or like building useful tools that will hopefully help people be better and stuff. But if you don't wanna, like, make any major career changes or, or things like that, then I would say just pay more attention to these issues and talk about it more with people. Do stuff like, you know, emailing your congressman or whatever. It, it doesn't change things that much, but it does help. I think that especially for this particular issue, the core problem is that people aren't taking it seriously yet. Like, if the sorts of things that I was just saying to you for the last hour or two were just, like, top of everybody's mind, we wouldn't even be here. Like, there, there would already be much more significant regulation [chuckles] in place, you know? And not only would there be more s- heavy regulation in place, but there would've been better regulation in place that's less, you know, less like a cudgel and more like a scalpel, and that's, like, more sensitive to what's y- actually bad and what's not so bad and so forth, and there'd be more expert people in the government and advising the government and so forth. So just in general, like, the more people wake up to these concerns and to these projections, uh, I think the more likely it is that we can do good stuff before it's too late.

    3. SBSteven Bartlett1:51:36

       What about how they should vote at the polls? We've got an election coming up in the United States in a couple of years' time, but there's elections happening all over the world all the time.

    4. DKDaniel Kokotajlo1:51:44

       You should ask the candidates what they think about all this AI stuff. You should try to get them to, like, have opinions [chuckles] and then you should vote for the candidates whose opinions are better on this topic. This is the most important thing happening, uh, in our lifetimes, probably in all of history, in fact, and it's very important that it go well, and so it's what all the, all the leaders of all the countries should be thinking about and making plans for.

    5. SBSteven Bartlett1:52:05

       Isn't it such a weird thing to be alive at this moment in time? Like, I was thinking about all the times that I could have been born, and I guess my ancestors probably thought the same, but I was thinking as you were speaking, I was like, I think it's when you referred to it as, like, the final show.

    6. DKDaniel Kokotajlo1:52:16

       Yeah.

    7. SBSteven Bartlett1:52:17

       What was the t- phraseology you used?

    8. DKDaniel Kokotajlo1:52:18

       I said the, the clim- it was the run-up to the climax or something.

    9. SBSteven Bartlett1:52:21

       Yeah. I mean, w-what a, what a crazy thing to be born in the run-up to the climax where everything you're describing here is within my lifetime conceivably, hopefully.

    10. DKDaniel Kokotajlo1:52:30

        Yeah.

    11. SBSteven Bartlett1:52:31

        Um, or maybe not hopefully. What a crazy time to be alive.

    12. DKDaniel Kokotajlo1:52:35

        Certainly.

    13. SBSteven Bartlett1:52:36

        I noticed that when I m- asked you if you had kids, your demeanor changed quite considerably.

    14. DKDaniel Kokotajlo1:52:40

        It was, yeah.

    15. SBSteven Bartlett1:52:42

        It's like you dropped into a different state. Obviously, that's been central to the r- rumination that you've been experiencing.

    16. DKDaniel Kokotajlo1:52:50

        Well, it is a sad topic, right? Like, when, when I had kids, like, the reason to have kids is in large part about the future, you know? Like, it's not just, like, a cuddly thing to have with you in the moment. It's 'cause you have all these hopes and dreams about how they'll grow up and how they'll go do their own thing and be their own person and stuff. And because of what's happening with AI, I think a lot of those dreams are in jeopardy.

    17. SBSteven Bartlett1:53:12

        Presumably, you still would have had kids.

    18. DKDaniel Kokotajlo1:53:14

        I've actually flip-flopped on this-

    19. SBSteven Bartlett1:53:15

        Really?

    20. DKDaniel Kokotajlo1:53:15

        ... occasionally. Yeah. Ba-basically, the top line answer is I'm not sure. The, my first child was had, we, we had her when we were, um, in twenty nine-- She was born in twenty nineteen.

    21. SBSteven Bartlett1:53:25

        Yeah.

    22. DKDaniel Kokotajlo1:53:26

        So this is before my timeline shortened a lot. So at this, at this point, I was interested in AI. I was tracking the field. I was making forecasts, but I didn't, like, actually expect it to happen soon, you know?

    23. SBSteven Bartlett1:53:35

        Mm-hmm.

    24. DKDaniel Kokotajlo1:53:36

        And then this caused, like... When I, when I did start thinking like, "Oh my gosh, it's gonna be happening, like, real soon, um, like, by twenty thirty," you know, um, that caused some reconsidering. And so I basically told my wife, like, "Let's not have any more kids. It's too uncertain," you know? But that turned out to be really hard because, especially for my wife, like, we already had one kid and, like, no siblings. Um, so eventually, I sort of gave in and was like, "Okay. Well, you know what? We already have one. It's gonna be all right. Like, maybe, maybe the future will be good, and even if it's not, like, well, we're all in the same boat together."

    25. SBSteven Bartlett1:54:15

        It's quite chilling what you're saying. It's chilling because you know more than me, and if you're at home saying to your wife, "Listen, maybe we should pause on having more children and building a family because of what's going on with AI."

    26. DKDaniel Kokotajlo1:54:26

        To be clear, is it... Yes. I mean, yes. It's very concerning. [chuckles] I am, I am chilled. Uh, this is bad. This is what I've been saying. I hope things go well. I think things might go well. Um, I think that there's a lot we can do to, like, steer things in a better direction.

    27. SBSteven Bartlett1:54:39

        I mean, one of those things as well, I have to say, is just speaking about it. It's, I think, a lot of the progress we've seen with governments waking up and, you know, we've seen certain things with people booing certain people at certain events-

    28. DKDaniel Kokotajlo1:54:51

        Yeah

    29. SBSteven Bartlett1:54:52

        ... um, is, is a downstream from people like yourself actually coming on shows like this and all the other podcasts and-

    30. DKDaniel Kokotajlo1:54:58

        Yeah
30. 1:57:25 – 2:00:48

    Is It Already Too Late to Change Course?

    1. SBSteven Bartlett1:57:26

       Do you think it's too late?

    2. DKDaniel Kokotajlo1:57:27

       No. I don't think it's too late. If I thought it was too late, I wouldn't be here.

    3. SBSteven Bartlett1:57:31

       Hmm. Where would you be?

    4. DKDaniel Kokotajlo1:57:33

       With my family.

    5. SBSteven Bartlett1:57:36

       What's your closing message to the general public, if you had to have a closing statement to them?

    6. DKDaniel Kokotajlo1:57:42

       Maybe I would say that, like, you're gonna hear a lot of things, and you already have been hearing a lot of things about AI, and it's going to sound like science fiction. But sometimes things which sound like science fiction happen in reality. And in fact, many times [chuckles] historically, things w- used to be science fiction have then become reality. And people need to stop thinking about what does or doesn't sound like science fiction and just start thinking about, like, the trends and, you know, the actual trends that this technology is on, and reading and forecasting how it's going to go, and then taking seriously the possibility that it could go something like this, and then thinking about what should be done about that.

    7. SBSteven Bartlett1:58:21

       And where would you direct them to get more information?

    8. DKDaniel Kokotajlo1:58:24

       Y- you can go to ai2027.com to read our previous scenario. You can go to ai2040.com, Plan A, to read our new proposal for what needs to be done. Um, these things are not just a sci-fi story. They also have lots of, like, explainers and links to other things. And so they're kind of, like, a nice jumping-off point to, to learn about all of this stuff.

    9. SBSteven Bartlett1:58:46

       Mm.

    10. DKDaniel Kokotajlo1:58:47

        Um, if you want, I could, um, after this is over, like, give a reading list of, like, other papers and articles and-

    11. SBSteven Bartlett1:58:55

        Please do

    12. DKDaniel Kokotajlo1:58:55

        ... you know, blogs to follow and so forth.

    13. SBSteven Bartlett1:58:57

        And I'll link them all below in the comments section. So if you're listening now, go ahead and take a look at the comments sec- the, um, description of this episode, and you'll see a bunch of links, which is Daniel's recommendations of what you should read. You know, I think it's, it's just a really, really great moment in time to get educated on this stuff. Um, humans have a, an inclination because of cognitive dissonance where we feel uncomfortable about something to bury our heads in the sand and avoid it.

    14. DKDaniel Kokotajlo1:59:20

        Yeah.

    15. SBSteven Bartlett1:59:20

        But actually, I think this is one such time to do the very opposite for many reasons.

    16. DKDaniel Kokotajlo1:59:24

        Yeah.

    17. SBSteven Bartlett1:59:24

        To, to inform yourself so you know what actions to take, but also because AI, uh, you know, u- unavoidably is gonna be a huge part of all of our lives and careers.

    18. DKDaniel Kokotajlo1:59:32

        Yeah. Yeah. Thank you. And that, that's a good way to, to say it. It's gonna matter a lot. It's gonna, it's gonna be everywhere soon, and, um, we need to do something about it before it's too late.

    19. SBSteven Bartlett1:59:43

        And what about AI Future Project?

    20. DKDaniel Kokotajlo1:59:44

        That's our organization. We spent a year writing AI 2027 after I left OpenAI, and then we spent another year writing AI 2040 Plan A.

    21. SBSteven Bartlett1:59:52

        Daniel, thank you.

    22. DKDaniel Kokotajlo1:59:53

        Thank you.

    23. SBSteven Bartlett1:59:53

        Thank you for all the work that you do. I can see how much you care about this stuff, and it's your care. It's funny, care itself makes others feel care. And, um, seeing how personal this is for you and seeing how much you've dedicated your life to this, but also hearing that you-

    24. DKDaniel Kokotajlo2:00:06

        Yeah

    25. SBSteven Bartlett2:00:06

        ... you basically walked away from $2 million to be able to speak to the public about this information is incredibly admirable. And, uh, I, I think voices like yours are more important now than they've ever been on this subject, so please do keep fighting the fight that you're fighting. And that's one of information, it is of honesty, and it is, uh, of saying what, what is often the quiet part out loud.

    26. DKDaniel Kokotajlo2:00:25

        Thank you.

    27. SBSteven Bartlett2:00:25

        And doing really, really smart research. I'll link everything we've t- discussed today below, um, and I hope we can chat again sometime soon.

    28. DKDaniel Kokotajlo2:00:31

        Thank you.

    29. SBSteven Bartlett2:00:32

        YouTube have this new crazy algorithm where they know exactly what video you would like to watch next based on AI and all of your viewing behavior, and the algorithm says that this video is the perfect video for you. It's different for everybody looking right now. Check this video out. I bet you, you might love it

Episode duration: 2:00:49