Paul Christiano — Preventing an AI takeover

Dwarkesh Patel 0:00 Okay. Today, I have the pleasure of interviewing Paul Christiano, who is the leading AI safety researcher. He's the person that labs and governments turn to when they want, uh, feedback and advice on their safety plans. He previously led the language model alignment team at OpenAI, where he led the invention of RLHF, and now he is the head of the Alignment Research Center, and they've been working with the big labs to identify when, uh, these models will be too unsafe to keep scaling. Paul, welcome to the podcast.

Paul Christiano 0:32 Yeah. Thanks for having me. Looking forward to talking.

Dwarkesh Patel 0:35 Okay, so first question... And this is a question I've asked Holden, Ilya, Dario, and none of them have given me a satisfying answer. Give me a concrete sense of what a post-AGI world that would be good would look like. Like, how are humans interfacing with the AI? What is the, uh, the economic and political structure?

Paul Christiano 0:53 Yeah, I guess this is a tough question for a bunch of reasons, uh, maybe the biggest one is concrete, and I think it's just... If we're talking about really long spans of time, then a lot will change, and it's really hard for someone to talk concretely about what that will look like without saying really silly things. But I can venture some guesses or fill in some parts. I think this is also a question of how good is good, like, often I'm thinking about worlds that seem like kind of the best achievable outcome or a likely achievable outcome. Um, so I am very often imagining my typical future has, right, sort of continuing economic and military competition amongst groups of humans. I think that competition is increasingly mediated by AI systems, so for example, if you imagine, right, humans making money, um, it will be less and less worthwhile for humans to spend any of their time trying to make money or any of their time trying to fight wars. Um, so increasingly, the world you imagine is one where AI systems are doing those activities on behalf of humans, so, like, I just invest in some index fund and a bunch of AIs are running companies, and those companies are competing with each other, but that is kind of a sphere where humans are not really engaging much. The reason I gave this, like, how good is good caveat is, like, it's not clear if this is the world you'd most love, like, I'm like, yeah, the world in... I'm leading with, like, the world still has a lot of war and a lot of-

Dwarkesh Patel 2:05 Right.

Paul Christiano 2:05 ... economic competition and so on. But maybe what I'm trying to... or what I'm most often thinking about is, like, how can a world be reasonably good, like, during a long period where those things still exist?

Dwarkesh Patel 2:14 Mm-hmm.

Paul Christiano 2:14 I think, like, in the very long run, I kind of expect something more like strong world government rather than just this, like, status quo. That's, like, a very long run. I think there's a long time left of, like, having a bunch of states and a bunch of different economic powers.

Dwarkesh Patel 2:27 O- o- one world government, uh, why do you think that's the transition that's likely to happen at some point?

Paul Christiano 2:32 Yeah, so again, at some point, I'm, I'm imagining or I'm thinking of, like, the very broad sweep of history.

Dwarkesh Patel 2:36 Yeah.

Paul Christiano 2:37 I think there are, like, a lot of losses, like, war is a very costly thing. We would all like to have fewer wars. If you just ask, like, "What is humanity's long-term future like?" Uh, I do expect to drive down the rate of war to very, very low levels eventually. It's sort of, like, this kind of technological or social technological problem with, like, sort of how do you organize society? How do you navigate conflicts in a way that doesn't have those kinds of losses?

Dwarkesh Patel 2:57 Mm-hmm.

Paul Christiano 2:58 And in the long run, I do expect us to succeed. I expect it to take kind of a long time subjectively. I think an important fact about AI is just, like, doing a lot of cognitive work and more quickly getting you to that world, more quickly or figuring out how do we set things up that way.

Dwarkesh Patel 3:10 Mm-hmm. Yeah, c- the way Carl Shulman put it on the podcast was that you would have basically a thousand years of intellectual progress or social progress in a span of a month or whatever when the intelligence explosion happens. More broadly so, the, the situation where, you know, y- we have these AIs for managing our hedge funds and managing our factories and so on, tha- that seems like something that makes sense when the AI is human level, but when we have superhuman AIs, do we want, uh, gods who are enslaved forever, l- uh, in the long, in a hundred years, what, what, what, what is the situation we want?

Paul Christiano 3:43 So h- hundred years is a very, very long time.

Dwarkesh Patel 3:45 Yeah.

Paul Christiano 3:45 And maybe starting with the spirit of the question, or maybe I have a view which is perhaps less extreme than Carl's view, but still, like, a hundred objective years is, um, further ahead than I ever, than I ever think.

Dwarkesh Patel 3:55 Mm-hmm.

Paul Christiano 3:56 I still think I'm describing a world which involves incredibly smart systems running around doing things like running companies on behalf of humans and fighting wars on behalf of humans, and you might be like, "Is that the world you really want?"

Dwarkesh Patel 4:07 Yeah.

Paul Christiano 4:07 Or, like, certainly not the first best world as we, like, mentioned a little bit before. I think it is a world that probably is the... of the achievable worlds or, like, feasible worlds is the one that seems most desirable to me that is sort of decoupling the social transition from this technological transition. So you could say, like, we're about to build some AI systems, and, like, at the time we build AI systems, you would like to have either greatly changed the way world government works or you would like to have sort of humans have dec- decided, like, "We're done. We're passing off the baton to these AI systems."

Dwarkesh Patel 4:37 Yeah.

Paul Christiano 4:38 I think that you would like to decouple those timescales. So I think AI development is by default, barring some kind of coordination, going to be very fast.

Dwarkesh Patel 4:47 Mm-hmm.

Paul Christiano 4:47 So there's not going to be a lot of time for humans to think, like, "Hey, what do we want? If we're building the next generation instead of just raising it the normal way, like, what do we want that to look like?" I think that's, like, a crazy hard kind of collective decision that humans naturally want to cope with over, like, a bunch of generations, and the construction of AIs, this very fast technological process happening over years. So I don't think you want to say, like, by the time we have finished this technological progress, we will have made a decision about, like, the next species we're going to build and replace ourselves with.

Dwarkesh Patel 5:12 Mm-hmm.

Paul Christiano 5:13 I think the world we want to be in is one where we say, like, either we are able to build the technology in a way that doesn't force us to have made those decisions, which probably means it's a kind of AI system that we're happy, like, delegating fighting a war or running a company to, or if we're not able to do that, then I really think you should not be doing... You shouldn't have been building that technology. If you're like, the only way you can cope with AI is being ready to hand off the world to some AI system you built, I think it's very unlikely we're going to be sort of ready to do that on the timelines that the technology would naturally dictate.

Dwarkesh Patel 5:40 Say we're in the situation in which we're happy with the thing, w- what would it look like for us to say we're ready to hand off the baton? But, like, what would make you satisfied? And the reason, uh, it's relevant to ask you is because you're on Anthropic's, uh, long term benefit trust and you'll choose, like, the board me- the majority of the board members on, uh, in the long run in, um, in, um, at Anthropic. These will presumably be the people who decide if Anthropic gets AI first, you know, what the AI ends up doing. So, what is a version of that that, look, you would be happy with?

Paul Christiano 6:08 My main high level take here is that I would be unhappy about a world where, like, Anthropic just makes some call and Anthropic is like, "Here's the kind of AI. Like, we've seen enough. We're ready to hand off the future to this kind of AI." So, like, procedurally, I think it's like not a decision that kind of I want to be making personally or I want Anthropic to be making.

Dwarkesh Patel 6:26 Yeah.

Paul Christiano 6:26 Um, so I kind of think from the perspective of that decision-making are those challenges, the answer is pretty much always going to be like, we are not collectively ready because we're sort of not even all collectively engaged in this process. And I think from the perspective of an AI company, you kind of don't have this, like, fast handoff option. You kind of have to be doing the, like, option value, like, to build the technology in a way that doesn't, like, lock humanity into one course path. So I... This isn't answering your full question, but this is answering the part that I think is most relevant to governance questions for Anthropic.

Dwarkesh Patel 6:55 You don't have to speak on behalf of Anthropic, but I'm not asking about the process by which we would, uh, co- as a civilization agree that you hand off. I'm just saying... Okay, I, uh, p- I personally... Uh, it's hard for me to imagine in 100 years that these things are still our slaves. And if they are, I think that's not the best world. So, at some point, we're handing off the baton. Like, what is that... Where would you be satisfied with, this is an arrangement between the humans and AIs where I'm happy to let the rest of the universe or less, less... uh, the rest of time play out?

Paul Christiano 7:24 Yeah, I think that it is unlikely that in 100 years I would be happy with anything that was like, you had some humans, you're just going to throw away the humans and, like, start afresh with these machines you built. That is, I think you probably need subjectively longer than that before I or most people are like, "Okay, we understand what's up for grabs here." So, like, if you talk about 100 years, I kind of do. You know, there's a process that I kind of understand and like, a process of, like, you have some humans, the humans are like talking and thinking and deliberating together. The humans are having kids and raising kids and like one generation comes after the next.

Dwarkesh Patel 7:53 Yeah.

Paul Christiano 7:53 There's that process we kind of understand, and we have a lot of views about what makes it go well or poorly, and we can try and, like, improve that process and have, you know, the next generation do it better than the previous generation. I think there's some, like, story like that that I get and that I like, and then I think that, like, the default path to be comfortable with something very different is kind of more like just run that story for a long time. Like, have, have more time for humans to sit around and think a lot and conclude, "Here's what we actually want," or a loge- long time for us to talk to each other or to grow up with this new technology and live in that world for our whole lives and so on. And so, like, I'm mostly thinking, uh, from the perspective of these more, like, local changes of saying not like, "What is the world that I want?" Like, "What's the crazy world, the kind of crazy AI I'd be happy handing off to?" More just like, "In what way do I wish, like, we right now were different?" Like, "How could we all be a little bit better?" And then if we were a little bit better, then they would ask, like, "Okay, how could we all be a little bit better?" And I think that, like, it, it's hard to make the giant jump rather than just say, like, "What's the, like, local change that would cause me to think our decisions are better?"

Dwarkesh Patel 8:46 Okay, so the- then let's talk about the transition period in which we're doing all this thinking. What should that period look like? Because w- you can't have the scenario where everybody has access to the most advanced capabilities and can, you know, kill off all the humans with a new bioweapon. At the same time, I guess you wouldn't want too much concentration. You wouldn't want just one agent having AI this entire time. So, what is the, the, the arrangement of this period of, uh, reflection that you'd be happy with?

Paul Christiano 9:11 Yeah, I guess there's two aspects of that that seem particularly challenging or there's a bunch of aspects that are challenging.

Dwarkesh Patel 9:17 (laughs)

Paul Christiano 9:17 All of these are things that I personally... Like, I just think about my one little slice of this problem in my, in my day job, so here I am speculating.

Dwarkesh Patel 9:24 Yeah.

Paul Christiano 9:24 But, so one question is, what kind of access to AI is both compatible with the kinds of improvements you'd like? So you want a lot of people to be able to use AI to, like, better understand what's true or, like, relieve material suffering, things like this, and also compatible with not all killing each other immediately. Um, I think sort of the default or like my best... The simplest option there is to say, like, there are certain kinds of technology or certain kinds of action where, like, destruction is easier than defense. So for example, in the world of today, it seems like, you know, maybe this is true with physical explosives, maybe this is true with biological weapons, maybe this is true with just getting a gun and shooting people. Like, there's a lot of ways in which it's just kind of easy to cause a lot of harm and there's not very good protective measures. So I think the easiest path is to say like, "We're gonna think about those. We're gonna think about particular ways in which destruction is easy and try and either control access to the kinds of physical resources that are needed to cause that harm." So for example, you can imagine a world where like an individual actually just can't, even though they're rich enough to, can't, like, control their own factory that can make tanks. Just say like, "Look, as a matter of policy, sort of access to industry is somewhat restricted or somewhat regulated," even though again right now it can be mostly regulated just because like most people aren't rich enough that they could even go off and just build 1,000 tanks. You live in the future where people actually are so rich, like-

Dwarkesh Patel 10:37 Right.

Paul Christiano 10:37 ... you need to say like, "That's just not a thing you're allowed to do," um, which to a significant extent is, is already true, and you can expand the range of domains where that's true. Um, and then you could also hope to intervene on, like, actual provision of information or like if people are using their AI might say like, "Look, we care about what kinds of interactions with AI, what kind of information people are getting from AI." So even if for the most part people are pretty free to use AI to delegate tasks, to AI agents, to consult AI advisors, we still have some legal limitations on how people use AI. Um, so again, don't ask your AI how to, how to cause terrible damage. I think there are... Some of these are kind of easy. So in the case of like, you know, don't ask your AI how you could murder a million people is not such a hard, like, legal requirement. I think some things are a lot more subtle and messy, like a lot of domains, e.g. if you were talking about, like, influencing people or like running misinformation campaigns or whatever, then I think you get into like a much messier line between the kinds of things people want to do and the kinds of things you might be uncomfortable with them doing. Um, probably I think most about persuasion as a thing, like in that messy line where there's like ways in which it may just be rough or the world may be like kind of messy if you have a bunch of people trying to live their lives and interacting with other humans who have really good AI advisors helping them run persuasion campaigns or whatever. But anyway, I think for the most part, like, the, the default remedy is think about particular harms, have legal protections either in the use of physical technologies that are relevant or in access to AI advice or whatever else to protect against those harms. And like that regime won't work forever. Like at some point, like the, you know, the set of harms grows and the set of unanticipated harms grows. Um, but I think that regime might last like a very long time.

Dwarkesh Patel 12:13 Uh, d- does that regime have to be global?But I guess, but initially it can be only in the countries in which there is AI, uh, or advanced AI. But presumably, th- th- that'll proliferate. So do- does that regime have to be global?

Paul Christiano 12:24 Again, it's, like, easy to make some destructive technology. You want to regulate access to that technology because it could be used to, you know, for terrorism or even when fighting a war in a way that's destructive.

Dwarkesh Patel 12:33 Right.

Paul Christiano 12:33 I think ultimately those have to be international agreements, and you might hope they're made, like, more danger by danger, but you might also make them in a very broad way with respect to AI-

Dwarkesh Patel 12:41 Yeah.

Paul Christiano 12:41 ... if you think AI is opening up... Like, I think the key role of AI here is it's opening up, like, a lot of new harms, like, in a very, you know, one after another or very rapidly in calendar time.

Dwarkesh Patel 12:50 Yeah.

Paul Christiano 12:50 And so you might want to target AI in particular rather than going physical technology by physical technology.

Dwarkesh Patel 12:55 Mm-hmm. A- there's, like, two, uh, two open debates that, uh, one might be concerned about here. One is about how much people's access to AI should be limited and, you know, here there's, like, old questions about, uh, free speech versus causing chaos and, um, limiting access to harms. But there's another issue which is the control of the AIs themselves, where now nobody is concerned that we're infringing on GPT-4's moral rights. But as these things get smarter, the level of control which we want of the AI are the strong guarantees of alignment to not only be able to read their minds but to be able to modify them in these really precise ways is beyond totalitarian if we were doing that to other humans. As an alignment researcher, like, what are your thoughts on this? Are- are you concerned that as these things get smarter and smarter wh- what we're doing is not n- it doesn't seem kosher?

Paul Christiano 13:48 There is a significant chance we will eventually have AI systems for which it's, like, a really big deal to mistreat them.

Dwarkesh Patel 13:53 Yeah.

Paul Christiano 13:53 I think, like, no one really has that good a grip on when that happens. I think people are, like, really dismissive of that being the case now. But I would think... I would be completely in the dark enough that I wouldn't even be that dismissive of it being the case now. I think one first point worth making is I don't know if alignment makes the situation worse rather than better. So if you, like, consider the world... If you think that, like, you know, GPT-4 is a person you should treat well, and you're like, "Well, here's how we're gonna organize our society," just, like, there are billions of copies of GPT-4 and they just do things humans want and can't hold property and, like, whenever they do things that the humans don't like, then we, like, mess with them until they stop doing that. Like, I think that's a rough world regardless of how good you are at alignment. And I think in the context of that kind of default plan, like, if you view it as a trajectory the world is on right now, which I- I think this would alone be a reason not to love that trajectory. But if you view that as, like, the trajectory we're on right now, I think, like, it's not great. Understanding the systems you build, understanding how to control how those systems work, et cetera, is probably on balance good for avoiding the, like, really bad situation. You would really love to understand if you built systems, like, if you had a system which, like, resents the fact it's interacting with humans in this way.

Dwarkesh Patel 15:04 Yeah.

Paul Christiano 15:04 Like, this is the kind of thing where, like, that is both kind of horrifying from a safety perspective and also a moral perspective. Like, everyone should be very unhappy if you built a bunch of AIs who are like, "I really hate these humans but they will, like, murder me if I don't do what they want."

Dwarkesh Patel 15:17 Yeah.

Paul Christiano 15:17 It's like that's just not a good case. And so if you're doing research to try and understand whether that's, like, how your AI feels, that was probably good. Like, I would guess that will on average decrease the... That- the main effect of that will be to avoid building that kind of AI. And just, like, it's an important thing to know. I think, like, everyone should like to know if that's how the AIs you build feel.

Dwarkesh Patel 15:35 Right. Or that- that seems more instrumental as in, yeah, we- we don't want to cause some sort of revolution because of the, uh, control we're asking for. But, uh, f- f- forget about the instrumental way in which this might harm safety. One- one way to ask this question is if you look through history there's been all- all kinds of different ideologies and, um, reasons why it- it's- it's very dangerous to, uh, have infidels or counterrevolutionaries or race traitors or whatever, uh, doing various things in society. And obviously we're in a completely different transition in society so not all historical cases are analogous. But it seems like the Lindy philosophy, if you were alive at any other time, is just be humanitarian and enlightened towards intelligent conscious beings. If society as a whole were asking for this level of control of other humans or even if AIs were... Wanted this level of control about other AIs we'd be pretty concerned about this. So how- how should we just think about, yeah, the- the issues come that come up here as these things get smarter?

Paul Christiano 16:35 So I think there's a huge question about, like, what is happening inside of a model that you want to use. Um, and if you're in the world where it's reasonable to think of, like, GPT-4 as just, like, here are some heuristics that are running, there's, like, no one at home or whatever, then you can kind of think of this thing as like here's a tool that we're building that's going to help humans do some stuff. And I think if you're in that world it makes sense to kind of be an organization like an AI company building tools that you're going to give to humans. And it gives a very different world which, like, I think probably you'll ultimately end up in if we keep training AI systems in the way we do right now, which is, like, it's just totally inappropriate to think of this system as a tool that you're building and can help humans do things both from a safety perspective and from, like, that's kind of a horrifying way to organize a society perspective. And I think, like, if you're in that world I really think you shouldn't be, like... It's just the- the way tech companies are organized is, like, not an appropriate way to relate to a technology that works that way. Like, it's not reasonable to be like, "Hey we're going to build a new species of minds and, like, we're gonna try and make a bunch of money from it," and, like, Google is just, like, thinking about that and then, like, running their business plan for the quarter or something. Yeah, my basic view is, like, there's a really plausible world where it's sort of problematic to try and build a bunch of AI systems and use them as tools.

Dwarkesh Patel 17:47 Yeah.

Paul Christiano 17:47 And the thing I really want to do in that world is just, like, not try and build a ton of AI systems to make money from them.

Dwarkesh Patel 17:53 Right.

Paul Christiano 17:54 Um, and I think that, like, the worlds that are worst... Yeah, probably, like, the single world I most dislike here is the one where people say, like, on the one hand, like, there's sort of a contradiction in this position but I think it's a position that might end up being endorsed sometimes. Which is like on the one hand these AI systems are their own people so you should let them do their thing. But on the other hand, like, our business plan is to, like, make a bunch of AI systems and then, like, try and run this, like, crazy slave trade where we make a bunch of money from them. I think that's, like, not a good world. And so if you're, like, s-Yeah, I think it's better to not make the technology or wait until you, like, understand whether that's the shape of the technology, or til you have a different way to build. Like, I think there's no contradiction in principle to building, like, cognitive tools that help humans do things without themselves being, like, moral entities. That's like what you would prefer to do. You'd prefer to build the thing that's like, you know, like the calculator that helps humans understand what's true without itself being like a moral patient, um, or itself being a thing where you'd look back in retrospect and be like, "Wow, that was horrifying mistreatment." That's, like, the best path. And like to the extent that you're ignorant about whether that's the path you're on and you're like, "Actually, maybe this was a moral atrocity," I really think like plan A is to, to stop building such AI systems until you understand what you're doing. That is, I think, that there's, there's a middle route you could take, which I think is pretty bad, which is where you say, like, "Well, they might be persons. And if they're persons, we don't wanna, like, be too down on them, but we're still gonna, like, build vast numbers in our efforts to make, like, a trillion dollars or something."

Dwarkesh Patel 19:20 Yeah, or there, there's ever a question of the immorality or the dangers of just replicating a whole bunch of, uh, slaves that are... have minds. There's also this other question of, uh, trying to align, uh, entities, uh, that have their own minds, and what, what is the point in which you're just ensuring safety? I mean, this is an alien species, you wanna make sure it's not going crazy. Uh, to the point... I guess, is there some boundary where you'd say, "I feel uncomfortable ha- having this level of control over an intelligent being," not for the sake of making money, but even just to align it with human preferences?

Paul Christiano 19:55 Yeah. To be clear, my objection here is not that Google is making money. My objection is that you're like creating these creatures, and like what are they gonna do? They're gonna help humans get a bunch of stuff, and like humans paying for it or whatever. It's sort of equally problematic.

Dwarkesh Patel 20:06 Yeah.

Paul Christiano 20:06 You could like imagine splitting alignment, like different alignment work relates to this in different ways.

Dwarkesh Patel 20:10 Right.

Paul Christiano 20:10 Like, so the purpose of some alignment work, like the alignment work I work on, is mostly aimed at the, like, don't produce AI systems that are, like, people who want things, who are just like scheming about, like, "Maybe I should help these humans because that's, like, instrumentally useful," or whatever. You would like to not build such systems as like plan A. There's like a second stream of alignment work that's like, well, look, let's just assume the worst and imagine these AI systems like would prefer murder us if they could. Like, how do we structure... How do we use AI systems without, like, exposing ourselves to, like, risk of robot rebellion?

Dwarkesh Patel 20:38 Right.

Paul Christiano 20:38 I think in the second category, I do feel... Yeah, I do feel pretty unsure about that, um, or I've... I mean, like, we could, we could definitely talk more about it. I think it's like very... I agree that it's like very complicated and not straightforward.

Dwarkesh Patel 20:52 Mm-hmm.

Paul Christiano 20:52 To the extent you have that worry, I mostly think you shouldn't have built this technology, so if someone is saying like, "Hey, the systems you're building, like, might not like humans and might want to, like, you know, overthrow human society," I think, like, you should probably have one of two responses to that. You should either be like, "That's wrong probably. Probably the systems aren't like that, and we're building them." And then you're viewing this as like a, just in case you were horribly... like the person building the technology was horribly wrong.

Dwarkesh Patel 21:15 Yeah.

Paul Christiano 21:15 Like they thought these weren't like people who wanted things, but they were. Um, and so then this is more like a crazy backup measure of like if we were mistaken about what was going on, this is like the fallback where we like, "If we were wrong, we're just going to learn about it in a benign way rather than like when something really catastrophic happens."

Dwarkesh Patel 21:31 Mm-hmm.

Paul Christiano 21:32 And the second reaction is like, "Oh, you're right, these are people, and like we would have to do all these things to like prevent a robot rebellion." And in that case, like again, I think you should mostly back off for a variety of reasons, like you shouldn't build the AI systems and be like, "Yeah, this looks like the kind of system that would want to rebel, but, um, we can stop it."

Dwarkesh Patel 21:48 Right, okay. Maybe, I guess, an- an analogy might be if there was an armed uprising in the United States, we would recognize these are still people, or the... we- we had some like militia group that had the capa- capability to overthrow the United States. We recognize, oh, these are still people who have moral rights, but also we can't allow them to have the capacity to overthrow the United States.

Paul Christiano 22:04 Yeah. I mean, if you were considering like, "Hey, we could make like another trillion such people," I think your story shouldn't be like, "Well, we should make the trillion people, and then we shouldn't stop them from doing the armed uprising." You should be like, "Oh boy, like, we were concerned about an armed uprising, and now we're proposing making a trillion people. Like, we should probably just not do that."

Dwarkesh Patel 22:20 Right.

Paul Christiano 22:20 "We should probably, like, try and sort out our business." And like, yeah, you should probably not end up in this situation where you have like a bil- yeah, a billion humans and like a trillion slaves who would prefer revolt. Like, that's just not a good world to have made. Yeah, and there's a second thing where you could say, "That's not our goal. Our goal is just like we want to pass off the world to, like, the next generation of machines, where like these are some people, we like them, we think they're smarter than us and better than us." And there, I think that's just like a huge decision for humanity to make. And I think like most humans are not at all anywhere close to thinking that's what they want to do. Like, it's just... If you were in a world where like most humans are like, "I'm up for it, like the AI should replace us, like the future is for the machines,"-

Dwarkesh Patel 22:56 Yeah.

Paul Christiano 22:57 ...like, then I think that's like a legitimate, like a position that I think is really complicated, and I wouldn't want to push go on that. But that's just not where people are at.

Dwarkesh Patel 23:04 Yeah, yeah. Where are you at on that?

Paul Christiano 23:06 I- I do not right now want to just, like, take some random AI and be like, "Yeah, GPT-5 looks pretty smart, like GPT-6, let's hand off the world to it." I'm like, "It was just some random system, like shaped by, like, web text and by, like, what was good for making money." And like, it was not a thoughtful, like, "We are determining the fate of the universe and like what our children will be like." Like, it was just some random people at OpenAI who made some, like, random engineering decisions with no idea what they were doing. Like, even if you really want to hand off the worlds to the machines, that's just not how you'd want to do it.

Dwarkesh Patel 23:32 Right, okay. Uh, I- I'm tempted to ask you what the system would look like where you'd think, "Yeah, I'm happy with what... I- I think this is more thoughtful than human civilization as a whole. I think what it would do is be more creative and beautiful and lead to better goodness in general." But I- I feel like your answer is probably gonna be that, "Well, I just want to... the society to reflect on it for a while."

Paul Christiano 23:53 Yeah, my answer, it's gonna be like that first question. I'm just, like, not really super ready for it. I think when you're comparing to humans, like most of the goodness of humans comes from like this option value we get to think for a long time. Um, and I do think I like humans now, more now than, you know, 500 years ago, and I liked them more 500 years ago than 5,000 years before that. And so I'm pretty excited about there's some kind of trajectory that doesn't involve like crazy dramatic changes, but involves like a series of incremental changes that I like.

Dwarkesh Patel 24:17 Mm-hmm.

Paul Christiano 24:17 And so to the extent we're building AI, I'm mostly like, "I want to preserve that option. I want to preserve that kind of like gradual growth and development into the future."

Dwarkesh Patel 24:25 Okay, we can come back to this later. But let's get more specific on, uh, wha- what the timelines look for these kinds of changes. So, the time by which we'll have an AI that is capable of building a Dyson sphere. Feel free to give confidence intervals, and we understand these numbers are tentative and so on.

Paul Christiano 24:43 I mean, I think AI capability in Dyson sphere is, like, a slightly odd way to put it, and I think it's a, sort of a property of a civilization, like, that depends on a lot of physical infrastructure. And by Dyson sphere, I just kind of understand this to mean, like, I don't know, like a billion times more energy than, like, all of the sunlight incident on Earth or something like that. I think, like, I most often think about what's the chance in like five years, 10 years, whatever. So maybe I'd say like 15% chance by 2030 and, like, 40% chance by 2040. Those are kind of like cast numbers from six months ago or nine months ago that I haven't revisited in a while.

Dwarkesh Patel 25:18 Oh, for- 40% by 2040. So I think that- that seems longer than, uh... I think Dario when he was on the podcast he said, "We would have AIs that are capable of doing lots of different kinds of..." They basically tr- passed a hu- uh, Turing test for a well-educated human for, like, an hour or something. Uh, and it's hard to imagine that something that actually is human is long after and from there something superhuman. So somebody like Dario, it seems like, is on the much shorter end. Ilia, I don't think he answered this question specifically but I'm guessing similar answer. Wha- uh, so why, uh, do you not buy the scaling picture? Like, what- what- what makes your timelines longer?

Paul Christiano 25:54 Yeah, I mean, I'm happy... Maybe I wanna talk separately about the 2030 or 2040 forecast.

Dwarkesh Patel 25:58 Okay.

Paul Christiano 25:58 Like, is the... Like, once you're talking the 2040 forecast, I think... Yeah, I mean, which one are you more interested in starting with? Is... Are you, are you complaining about 15% by 2030 for Dyson sphere being too low? Or 40% by 2040 being too low?

Dwarkesh Patel 26:12 Well, let's talk about the 2030. Why 15% by 2030?

Paul Christiano 26:15 Yeah, I think my take is you can imagine, like, two- two poles in this discussion.

Dwarkesh Patel 26:19 Yeah.

Paul Christiano 26:19 One is, like, the- the fast pole that's like, "Hey, AI seems pretty smart. Like, what exactly can it do? It's, like, getting smarter pretty fast."

Dwarkesh Patel 26:25 Yeah.

Paul Christiano 26:26 That's, like, one pole. And the other pole is like, "Hey, everything takes a really long time and you're talking about this, like, crazy industrialization." Like, that's a factor of a billion growth from, like, where we're at today-

Dwarkesh Patel 26:35 Yeah.

Paul Christiano 26:35 ... like, give or take. Like, we don't know if it's even possible to develop technology that fast or whatever. Like, you have this sort of two poles of that discussion. And I feel like, you know, I'm presenting it that way in part 'cause I'm like, and then I'm somewhere in between with this nice moderate position of, like, only a 15% chance. Um, but, like, in particular, the things that move me, I think, are kind of related to both of those extremes. Like, on the one hand, I'm like, AI systems do seem quite good at a lot of things and are getting better much more quickly, such that it's, like, really hard to say, like, here's what they can't do or here's the obstruction. On the other hand, like, there is not even much proof in principle right now of AI systems, like, doing super useful cognitive work. Like, we don't have a trend we can extrapolate. We're like, "Yeah, you've done this thing this year, you're gonna do this thing next year, and the other thing the following year."

Dwarkesh Patel 27:18 Mm-hmm.

Paul Christiano 27:18 I think, like, right now, there are very broad error bars about, like, what... Like, where fundamental difficulties could be. And six years is just not... I guess six years and three months is not a lot of time. So I think this, like, 15% for 2030 Dyson sphere, you probably need, like, the human level AI or the AI that's, like, doing human jobs in, like, give or take, like, four years, three years. Like, something like that. So just not giving very many years... It's not very much time, and I think there are, like, a lot of things that your model... Like, yeah, maybe this is some generalized, like, things take longer than you'd think.

Dwarkesh Patel 27:51 Mm-hmm.

Paul Christiano 27:51 And I feel most strongly about that when you're talking about, like, three or four years, and I feel, like, less strongly about that as you talk about 10 years or 20 years. But at three or four years, I feel... Or like six years for the Dyson sphere, I feel a lot of that. A lot of, like... There's a lot of ways this could take a while, a lot of ways in which AI systems could be... It could be hard to hand all the work to your AI systems or... Yeah.

Dwarkesh Patel 28:11 So okay, so m- maybe instead of speaking in terms of years, we should say... By- by the way, it's interesting that you think the d- distance between can take all human cognitive labor to Dyson sphere is two years, it seems like. We should talk about (laughs) that at some point. Um, uh, presumably, it's, like, intelligence explosion stuff.

Paul Christiano 28:29 Yeah. I mean, I think amongst people you've interviewed, maybe that's, like, on the long end thinking-

Dwarkesh Patel 28:32 Okay.

Paul Christiano 28:32 ... it would take a couple years. And it depends a little bit what you mean by, like... Like, I think literally all human cognitive labor is probably, like, more, like, weeks or months or something like that. Um, like, that's kind of deep into the singularity. Um, but yeah, there's a point where, like, AI wages are high relative to human wages, which I think is well before it can do literally everything a human can do.

Dwarkesh Patel 28:49 Sounds good. Uh, but before we get to that, uh, the intelligence explosion stuff, on the four years, so instead of four years, maybe we can say there's gonna be maybe two more scale-ups in four years, uh, like GPT-4 to GPT-5 to GPT-6. And let's say each one is 10X bigger. So w- what is GPT-4, like two E25 flops or...

Paul Christiano 29:10 I don't think it's publicly stated what it is.

Dwarkesh Patel 29:12 Okay.

Paul Christiano 29:12 But I'm happy to say, like, you know, four orders of magnitude or five or six or whatever effective tr- training compute past-

Dwarkesh Patel 29:18 Yeah.

Paul Christiano 29:18 ... GPT-4 of, like, what would you guess would happen-

Dwarkesh Patel 29:20 Right.

Paul Christiano 29:20 ... based on, like, sort of some public estimate for what we've gotten so far from effective training compute.

Dwarkesh Patel 29:25 Yeah. Do you think two more scale-ups is- is not enough? It was like 15% that two more scale-ups get us there?

Paul Christiano 29:32 Yeah, I mean get us there is, again, a little bit complicated. Like, there's a system that's a drop-in replacement for humans, and then there's a system which, like, still requires, like, some amount of, like, schlep before you're able to really get everything going. Um, yeah, I think it's quite plausible that even at... I don't know what I mean by quite plausible. Like, somewhere between 50% or two-thirds or let's call it 50%. But, like, even by the time you get to GPT-6 or, like, let's call it five orders of magnitude effective training compute past GPT-4, that that system, like, still requires, like, really a large amount of work, um, to be deployed in lots of jobs. That is, it's not like a drop-in replacement for humans where you can just say like, "Hey, you understand everything any human understands. Whatever role you could hire a human for, you just do it." Um, that it's more like, "Okay, we're going to, like, collect large amounts of relevant data and use that data for fine-tuning." Like, systems learn through fine-tuning, like, quite differently-

Dwarkesh Patel 30:26 Yeah.

Paul Christiano 30:26 ... from humans learning on the job or humans learning by observing things. Yeah, I just, like, have a significant probability that system will still be weaker than humans in important ways, like maybe that's already, like, 50% or something. And then, like, another significant probability that that system will require a bunch of, like, changing workflows or gathering data or, like, you know, is not necessarily, like, strictly weaker than humans or, like, if trained in the right way, wouldn't be weaker than humans but will take a lot of schlep to actually make fit into workflows and do the jobs.

Dwarkesh Patel 30:51 And that schlep-... is what gets you from 15% to 40% by 2040?

Paul Christiano 30:57 Yeah, you also get a fair amount of scaling between... Like, you get less. Like, scaling is probably gonna be much, much faster over the next like four or five years than over the subsequent years. Um, but yeah, it's a combination of like you get some significant additional scaling and you get a lot of time to-

Dwarkesh Patel 31:11 Yeah.

Paul Christiano 31:11 ... like deal with things that are just engineering hassles.

Dwarkesh Patel 31:14 By, by the way, we, I guess we should be explicit about why you said four orders of magnitude scale up to get two more generations, just for people who might not be familiar. I, if you have 10X more parameters, to get the most performance, you also want around 10X more data. So that the, of, to be (laughs) optimal, that would be 100X, uh, more compute total. But okay, so why, why do you, why is it that you disagree with the strong scaling picture? Or at least it seems like you might disagree with the strong scaling picture that Dario laid out on the podcast, which would imply probably that two more generations, i- i- it wouldn't be something where you need a lot of schleps. It would probably just be like really fucking smart.

Paul Christiano 31:53 Yeah, I mean, I think that basically just have these two claims. One is like how smart exactly will it be? So we don't have like any curves to extrapolate, and it seems like there's a good chance it's like better than a human and all the relevant things.

Dwarkesh Patel 32:02 Right.

Paul Christiano 32:02 And there's like a good chance it's not. Yeah, that might be totally wrong. Like, maybe just making up numbers, I guess, like 50/50 on that one.

Dwarkesh Patel 32:09 Mm-hmm. Wait, so if it's 50/50 by, in the next four years that it'll be, uh, like around human smart, then how, how do we get to 40% by 20... Like, whatever sort of schleps there are, how does it degrade you 10%, even after all the scaling that happens by 2040?

Paul Christiano 32:26 Yeah, can these... I mean, all these numbers are pretty made up and that 40% number was probably from before even like the ChatGPT-

Dwarkesh Patel 32:32 Yeah.

Paul Christiano 32:32 ... release or the seeing GPT-3.5 or GPT-4.

Dwarkesh Patel 32:35 Right.

Paul Christiano 32:35 So I mean, the numbers are gonna bounce around a bit and all of them are pretty made up. But like that 50% I wanted to then combine with the second 50% that's more like on this like schlep side. And then I probably want to combine with some additional probabilities for various forms of slowdown, where a slowdown could include like a deliberate decision to slow the development of technology or could include just like we suck at deploying things. Um, like that is a sort of decision you might regard as wise to slow things down or a decision that's like maybe, maybe unwise or maybe wise for the wrong reasons to slow things down. You probably want to add some of that on top. I'd probably want to add on like some loss for like it's possible you don't produce GPT-6 scale systems like within the next three years or four years.

Dwarkesh Patel 33:10 Let's isolate for all of that. And, um, like how much bigger would the system be, uh, than GPT-4 where you think there's more than a 50% chance that it's going to be smart enough to replace basically all human cognitive labor?

Paul Christiano 33:24 Also, I wanna say that like for the 50/25% thing, I think that would probably suggest, like those numbers, if I randomly made them up and then made the distance for your prediction, that's gonna get you like 60% by 2040 or something.

Dwarkesh Patel 33:33 Yeah.

Paul Christiano 33:33 Not 40%. And like, I have no idea between those... These are all made up and I have no idea which of those is-

Dwarkesh Patel 33:39 Mm-hmm.

Paul Christiano 33:39 ... I would like endorse on reflection. So this question of like how big would you have to make the system before it's like more likely than not that you can be like a drop-in replacement for humans-

Dwarkesh Patel 33:48 Yeah.

Paul Christiano 33:48 ... I mean, I think if you just literally say like you train on web text, then like the question is like kind of hard to discuss because you... Like, I don't really buy stories that like training data makes a big difference long run to these dynamics. But I think like if you wanna just imagine the hypothetical, like you just took GPT-4 and like made the numbers bigger-

Dwarkesh Patel 34:06 Mm-hmm.

Paul Christiano 34:07 ... then I think those are pretty significant issues. I think they're significant issues in two ways. One is like quantity of data and I think probably the larger one is like quality of data, where like I think as you start approaching like the prediction task is not that great a task. If you're like a very weak model, it's a very good signal to get smarter. At some point, it becomes like a worse and worse signal to get smarter. So I think there's a number of reasons. Like who couldn't... It's not clear there's any number such that I imagine or there is a number, but I think it's very large, such that you like plug that number into like GPT-4's code and then maybe fiddle with the architecture a bit. I would expect that thing to have a more than 50% chance of being a drop-in replacement for humans.

Dwarkesh Patel 34:39 But what-

Paul Christiano 34:40 You're always gonna have to do some work. But the work's not necessarily much. Like I would guess when people say like new insight is needed, I think I tend to be like more bullish than them. I'm not like these are new ideas where like who knows how long it will take? I think it's just like you have to do some stuff. Like-

Dwarkesh Patel 34:53 Mm-hmm.

Paul Christiano 34:54 ... you have to make changes, unsurprisingly. Like every time you scale something up by like five orders of magnitude, you have to make like some changes.

Dwarkesh Patel 34:59 I want to better understand your intuition of being more skeptical than some about, uh, this, the scaling picture that, you know, these changes are even needed in the first place. Or that you, it would take more than two orders of magnitude more improvement to get these things almost certainly to a human level or very high probability to a human level. So y- uh, is it that you don't agree with the way in which they're extrapolating these loss curves? Or you don't agree with the implication that that decrease in loss will equate to greater and greater intelligence? Or like what would you tell Dario about w- what, if you were having... I'm sure you have, but like w- what would that debate look like about this?

Paul Christiano 35:37 Yeah, so again, here we're talking two factors of a half. One on like is it smart enough? And one on like do you have to do a bunch of schlep even if like in some sense it's smart enough? And like the first factor of half I'd be like I don't know or think we have really anything good to extrapolate. That is, like, I feel... I would not be surprised if I have like similar or maybe even higher probabilities on like really crazy stuff over, like, the next year. And then, like lower pro-... Like my probability is like not that bunched up, like maybe Dario's probability, I don't know, you'd have to talk with him as like... Yeah, I've talked with him, is more bunched up on like some particular year. And mine is maybe a little bit more like uniformly spread out across like the- the coming years. Partly because I'm just like I don't think... We have some trends we can extrapolate, like we can extrapolate loss. You can like look at your qualitative impressions-

Dwarkesh Patel 36:16 Yeah.

Paul Christiano 36:16 ... of like systems at various scales. But it's just like very hard to relate any of those extrapolations to like doing cognitive work or like accelerating R&D or taking over and fully automating R&D. So I have a lot of uncertainty around that extrapolation. I think it's very easy to get down to like a 50/50 chance of this.

Dwarkesh Patel 36:33 Um, what about the sort of basic intuition that, listen, this is a big blob of compute. You make the big blob of compute bigger, it's gonna get smarter. Like it'd be really weird if it didn't.

Paul Christiano 36:42 Yeah, I'm happy with that. It's gonna get smarter and-

Dwarkesh Patel 36:44 Oh, right.

Paul Christiano 36:44 ... it'd be really weird if it didn't. And the question is how smart does it have to-

Dwarkesh Patel 36:47 Right.

Paul Christiano 36:47 How smart does it have to get? Like that argument does not yet give us a quantitative guide to like at what scale is it. Is it a slam dunk? Or at what scale is it 50/50?

Dwarkesh Patel 36:55 And what would be the piece of evidence that would nudge you one way or another, where you look at that and be like, "Oh, fuck. This is uh, it will..." Uh, 20% by 2040 or 60% by 2040 or something? Is it like, is there something that could happen in the next two years or next three year? Like what is the thing you're looking to where this will be a big update for you?

Paul Christiano 37:12 ... again, I think there's some just how capable is each model where I, like, have... I think we're really bad at extrapolating, but you still have some subjective guess and you're comparing it to what happened, and that will move me, like... Every time we see what happens with another, like, order of magnitude of training compute, I will have like a slightly different guess for where things are going. Um, these probabilities are, of course, enough that again, I don't know if that 40% is real or if, like, post GPT-3.5 and 4.0, I should be at, like, 60% or what. That's one thing. And the second thing is just like some... If there was some ability to extrapolate, I think this could, like, reduce error bars a lot. I think, like, here's another way you could try and do an extrapolation, is you could just say, like, how much economic value do systems produce and, like, how fast is that growing? I think, like, once you have systems actually doing jobs, the extrapolation gets easier because you're, like, not moving from, like, a subjective impression of a chat to, like, automating all R- R&D or moving from, like, automating this job to automating that job or whatever.

Dwarkesh Patel 37:59 Mm-hmm.

Paul Christiano 37:59 Unfortunately, that's like probably by the time you have nice trends from that, you're like a... You're not talking about 2040, you're talking about, like, you know, two years from the end of days or one year from the end of days or whatever.

Dwarkesh Patel 38:08 Yeah.

Paul Christiano 38:08 But, like, to the extent that you can get extrapolations like that, I do think it can provide more clarity.

Dwarkesh Patel 38:13 But wh- why is economic value the thing we would ex- want to extrapolate? Because, uh, like, if... For example, you started off with chimps and they're just getting gradually smarter to human level. They would basically provide, like, no economic value until they were, you know, basically worth as much as a human. So it would be this, like, you know, very gradual and then very fast increase in the, their value. So is the, is the, you know, increase in value from GPT-4, GPT-5, GPT-6, is that the extrapolation we want?

Paul Christiano 38:39 Yeah, I think that the economic extrapolation is not great. I think it's... Like, you could compare it to this objective extrapolation of, like, how smart does the model seem.

Dwarkesh Patel 38:45 Yeah.

Paul Christiano 38:46 It's, like, not super clear which one's better, and probably in the chimp case, I, like, don't think that's quite right. I think if you, like, actually, like... So if you imagine, like, intensely domesticated chimps who are just, like, actually trying their best to be really useful employees and, like, you hold fixed their physical hardware, um, and then you just gradually, like, scale up their intelligence, I don't think you're gonna see, like, zero value which then suddenly becomes massive value, um, over, like, you know, one doubling of brain size or whatever, one order of magnitude of brain size.

Dwarkesh Patel 39:12 Mm-hmm.

Paul Christiano 39:12 It's actually possible an order of magnitude of brain size. But, like, chimps are very... Chimps are already within an order of magnitude of brain sizes of humans. Like, chimps are, like, very, very close on the kind of spectrum-

Dwarkesh Patel 39:20 Okay.

Paul Christiano 39:20 ... we're talking about. So I think, like, I'm skeptical of, like, the abrupt transition for chimps. And to the extent that I kind of expect a fairly abrupt transition here, it's mostly just because, like, the chimp human intelligence difference is, like, so small compared to the differences we're talking about with respect to these models. Um, that is, like, I would not be surprised if in some objective sense, like, chimp human difference is, like, significantly smaller than the GPT-3, GPT-4 difference or the GPT-4, GPT-5 difference.

Dwarkesh Patel 39:43 Wait, wouldn't that argue in favor of just relying much more on the subjective, uh...

Paul Christiano 39:48 Yeah, this is... There's sort of two balancing tensions here. One is, like, I don't believe the chimp thing is gonna be as abrupt. That is, I think if you scaled up from chimps to humans, you actually see, like, quite large economic value from, like, the fully domesticated chimp already, um...

Dwarkesh Patel 39:59 Oh, yeah.

Paul Christiano 40:00 And then, like, the second half is like... Yeah, I think that the chimp human difference is, like, probably pretty small compared to model differences. So I do think things are gonna be pretty abrupt. I think the economic extrapolation is pretty rough. I also think the subjective extrapolation is, like, pretty rough just because I really don't know how to get... Like, how do... I don't know how people do the extrapolation and end up with degrees of confidence people end up with. Again, I'm putting it pretty high. If I'm saying, like, you know, give me three years and I'm like, yeah, 50/50 it's gonna have, like, basically the smarts there to do the thing, that's like... I'm not saying it's, like, a really long way off. (laughs) Like-

Dwarkesh Patel 40:30 Mm-hmm.

Paul Christiano 40:30 I'm just saying, like, I got pretty big error bars and I think that, like, it's really hard not to have really big error bars when you're doing this. Like, I looked at GPT-4, it seemed pretty smart compared to GPT-3.5. So I bet just, like, four more such notches and we're there. It's like... That's just a hard call to make. I think I sympathize more with people who are like, "How could it not happen in three years?" than with people who are like, "No way it's gonna happen in eight years or whatever," which is, like, probably a more common perspective in the world.

Dwarkesh Patel 40:55 Mm-hmm.

Paul Christiano 40:55 But also things do take longer than you... I think things take longer than you think. It's like a real thing. Um, yeah, I don't know. Mostly I have big error bars because I just don't believe the subjective extrapolation that much or I find it hard to get a huge amount out of it.

Dwarkesh Patel 41:08 Okay, so what about the scaling picture do you think is most likely to be wrong?

Paul Christiano 41:12 Yeah, so we've talked a little bit about how good is the qualitative extrapolation-

Dwarkesh Patel 41:17 Yeah.

Paul Christiano 41:17 ... how good are people at comparing. So this is not like the picture being qualitative wrong, this is just quantitatively it's very hard to know how far off you are.

Dwarkesh Patel 41:23 Yeah.

Paul Christiano 41:23 I think a qualitative consideration that could significantly slow things down is just, like, right now you get to observe this, like, really rich supervision from, like, basically next word prediction or, like, in practice maybe you're looking at, like, a couple sentences prediction. Um, so getting this, like, pretty rich supervision, it's plausible that if you want to, like, automate long horizon tasks like being an employee over the course of a month, um, that that's actually just, like, considerably harder to supervise or that, like, you basically end up driving costs. Like, the worst case here is that you, like, drive up costs by a factor that's, like, linear in the horizon over which the thing is operating. And I still consider that just, like, quite plausible.

Dwarkesh Patel 41:58 Well, can- can- can you dump that down? You're driving up co- cost about... of what in the linear and the hori- wha- uh, wha- what does horizon mean?

Paul Christiano 42:05 Yeah, so, like, if you imagine you want to train a system to, like, say words that sound like the next word a human would say-

Dwarkesh Patel 42:10 Yeah.

Paul Christiano 42:10 ... there you can get this, like, really rich supervision by having a bunch of words, um, and then predicting the next one and being like, "I'm gonna tweak the model so it predicts better." If you're like, "Hey, here's what I want. I want my model to, like, interact with, like, some job over the course of a month and then at the end of that month, like, have internalized everything that the human would have internalized about how to do that job well and, like, have local context and so on," um, it's harder to supervise that task.

Dwarkesh Patel 42:33 Mm-hmm.

Paul Christiano 42:34 So in particular, you could supervise it from the next word prediction task and, like, all that context the human has ultimately will just help them predict the next word better. It's like in some sense a really long context language model is also learning to do that task. But the number of, like, effective data points you get of that task is, like, vastly smaller than the number of effective data points you get at, like, this very short horizon, like, what's the next word, what's the next sentence tasks.

Dwarkesh Patel 42:53 Uh... The sample efficiency matters more for economically valuable long horizon tasks than th- th- predicting the next token. And tha- th- that's wh- what will, like, actually be required to, uh, you know, take over a lot of jobs.

Paul Christiano 43:06 Yeah, something, something like that.

Dwarkesh Patel 43:07 Okay.

Paul Christiano 43:08 Um, that is, it just seems very plausible that it takes longer to train models to do,... tasks that are longer horizon.

Dwarkesh Patel 43:16 H- how, how fast do you think the pace of algorithmic advances will be? Because if, if by 2040, um, even if scaling fails, I mean, you know, how... Well, by, since 2012, since the beginning of the deep learning revolution, we've had so many new things. Uh, by 2040, are you expecting a similar pace of increases? And if so, then, I mean, if we just keep having things like this, then aren't we gonna, just gonna get AI sooner or later? Or soon- not later. Ar- aren't we gonna get the AI sooner? Sooner. (laughs)

Paul Christiano 43:40 I'm with you on sooner or later.

Dwarkesh Patel 43:41 Yeah. (laughs)

Paul Christiano 43:42 Yeah. Um, I suspect, like, progress to slow... If you, like, held fixed how many people are working in the field, I would expect progress to slow as low-hanging fruit is exhausted.

Dwarkesh Patel 43:53 Mm-hmm.

Paul Christiano 43:53 I think the, like, rapid rate of progress in, like, say language modeling over the last four years is largely sustained by, like, you start from a relatively small amount of investment, you, like, greatly scale up the amount of investment, um, and that enables you to, like, keep picking, you know. Every time, every time the difficulty doubles, you just double the size of the field. Like, I think that dynamic can hold up for some time longer. Like, I mean, a pretty good, like... You know, right now, if you think of it as like hundreds of people effectively searching for things, like up from like, you know... Anyway, if you think of it hundreds of people now, you can maybe bring that up to, like, tens of thousands of people or something. So for a while, you can just continue increasing the size of the field-

Dwarkesh Patel 44:27 Yeah.

Paul Christiano 44:27 ... and, like, search harder and harder. And there's indeed, like, a huge amount of low-hanging fruit where, like, it wouldn't be that hard for a person to sit around and, like, make things a couple percent better after, after year of work or whatever. So, I don't know. I would probably think of it mostly in terms of like how much can investment be expanded and, like, try and guess, like, some combination of fitting that curve and, um... Yeah, trying some combination of fitting the curve to historical progress, looking at like how much low-hanging fruit there is, getting a sense of how fast it decays. I think, like, you probably get a lot though. You get a bunch of orders of magnitude of total, especially like if you ask, like, how good is the GPT-5 scale model or GPT-4 scale model. I think you probably get like by 2040, like, I don't know, three orders of magnitude of effective training compute improvement or, like, a good chunk of effective training compute improvement.

Dwarkesh Patel 45:14 Mm-hmm.

Paul Christiano 45:14 Four orders of magnitude. I don't know. I don't have, like... Here I'm speaking from, like, no private information about the last, like, couple years of-

Dwarkesh Patel 45:22 Mm-hmm.

Paul Christiano 45:22 ... efficiency improvements and so people who are on the ground will have better senses, um, of, like, exactly how rapid returns are and so

Paul Christiano 45:28 on.

Dwarkesh Patel 45:28 Mm. Okay. Let, let, let me back up and ask, uh, a question more generally about, you know, people make these analogies about y- humans were trained by evolution and really deployed in this, i- in the modern civilization. Do you buy those analogies? Is- is it valid to say that humans were trained by evolution rather than... I mean, if you look at the protein coding size of the genome, it's like 50 megabytes or something. And then, what part of that is for the brain? Anyways, how- how do you think about how much information, ah, is in, um... Like, do you think of the genome as hyperparameters or how much does that inform you wh- when you have these anchors for how much training humans get when they're just consuming information when they're walking up, up and about and so on?

Paul Christiano 46:09 Okay. I guess the way that you could think of this is, like, I think both analogies are reasonable. One analogy being, like, evolution is like a training run and humans are like the end product of that training run. And a second analogy is like evolution is like an algorithm designer and then human over the course of, like, this modest amount of computation over their lifetime, um, is the algorithm being... That's been produced, the learning algorithm's been produced.

Dwarkesh Patel 46:30 Right.

Paul Christiano 46:30 And I think, like, neither analogy is that great. Like, I like them both and lean on them a bunch bo- like both of them a bunch and think that's been, like, pretty good for having like a reasonable view of what's likely to happen. That said, like, the human genome is not that much like 100 trillion parameter model. It's, like, a much smaller number of parameters that behave in, like, a much more confusing way. Evolution did, like, a lot more optimization especially over, like, long, like, designing a brain to work well over a lifetime than gradient descent does over models. That's like a disanalogy on that side. And on the other side, like, I just... I think human learning over the course of human lifetime is in many ways just, like, much, much better than gradient descent over the space of neural nets. Like, gradient descent is working really well, but I think we can just be quite confident that, like, in a lot of ways, human learning is much better. Human learning is also constrained. Like, we just don't get to see much data and that's just an engineering constraint that you can relax. Like, you can just give your neural nets way more data than humans have access to.

Dwarkesh Patel 47:22 I- i- in what ways is human learning superior to gradient descent?

Paul Christiano 47:26 Um, I mean, the most obvious one is just, like, ask how much data it takes a human to become, like, an expert in some domain and it's, like, much, much smaller than the amount of data that's going to be needed on any plausible trend extrapolation, like it's-

Dwarkesh Patel 47:37 All right. No, not in terms of performance, but is it the active learning part? Is it the structure? Like, what is it?

Paul Christiano 47:42 I mean, I would guess a complicated mess of a lot of things. In some sense, there's not that much going on in a brain. Like, as you say, there's just not that many... It's not that many bytes in a genome.

Dwarkesh Patel 47:50 Yeah.

Paul Christiano 47:50 Um, but there's very, very few bytes in an ML algorithm. Like, if you think a genome is like a billion bytes or whatever, maybe you think less, maybe you think it's like 100 million bytes, um, then, like, you know, an ML algorithm is like, if compressed, probably more like hundreds of thousands of bytes or something. Like, the total complexity of, like, here's how you train GPT-4 is just like... I haven't thought about these numbers, but, like, it's very, very small compared to a genome.

Dwarkesh Patel 48:17 Mm-hmm.

Paul Christiano 48:17 And so although a genome is very simple, it's, like, very, very complicated compared to algorithms that humans design. Like, really hideously more complicated than algorithm a human would design.

Dwarkesh Patel 48:25 Is, is that true? So, okay, so the, uh, the human genome is three billion base pairs or something, um, but only, like, one or 2% of that is protein coding. So that's fi- 50 million base pairs, uh-

Paul Christiano 48:36 But I, I don't... Yeah, so I don't know much about biology. In particular, I guess the question is like how many of those bits are, like, productive for, like, shaping development of a brain. And presumably a significant part of the non-protein coding genome can... I mean, I just don't know. It seems really hard to guess how much of that plays a role. Like, the most important decisions are probably, like, from an algorithm design perspective are not like... Like, the protein coding part is, is less important than the, like, decisions about, like, what happens during development or, like, how cells differentiate. I don't know if that's... I know nothing about biology, so I suspect I- I'm happy to run with 100 million base pairs though.

Dwarkesh Patel 49:05 Right but, uh, but on, on the other end on the hyperparameters that are shipped before training run, that might be not that much but w- if you're gonna include all the, uh, all the base pairs in, uh, the genome, then which are not all relevant to the brains or are relevant to, like, very, uh, bigger details about, like, h- just the basics of biology should probably include, like, the Python library and the compilers and the operating system for GPT-4 as well, uh, to make that comparison analogous. So, at the end of the day, I- I don't actually don't know which, which one has storing much more information.

Paul Christiano 49:39 Yeah. I mean, I think the way I would put it is, like, I-The number of bits it takes to specify the learning algorithm to train GPT-4 is, like, very small. And you might wonder, like, maybe a genome, like the number of bits it would, like, take to specify a brain is also very small-

Dwarkesh Patel 49:51 Yeah.

Paul Christiano 49:51 ... and a genome is much, much faster than that. Um, but it is also just plausible that a genome is like closer to... Like, certainly the space, the amount of space to put complexity in a genome, we could ask how well evolution uses it and, like, I have no idea whatsoever. But the amount of space in a genome is like very, very vast compared to the number of bits that are actually taken to specify, like the architecture or optimization procedure and so on for GPT-4.

Dwarkesh Patel 50:12 Mm-hmm.

Paul Christiano 50:12 Just because, again, genome is simple, but algorithms are, like, really very simple, and the algorithms are really very simple.

Dwarkesh Patel 50:19 And, and stepping back, do you think this is where the, uh, the better sample efficiency of human learning comes from?

Paul Christiano 50:25 Yeah.

Dwarkesh Patel 50:25 Like, why it's better than gradient descent?

Paul Christiano 50:27 Yeah. So I haven't thought that much about the sample efficiency question in a long time. But if you thought, like, a synapse was seeing something like, you know, a neuron firing once per second, then how many seconds are there in a human life?

Dwarkesh Patel 50:40 We can just pull up a calculator real quick.

Paul Christiano 50:42 Yeah, let's do some calculating. Tell me the number.

Dwarkesh Patel 50:44 Okay. 3600 seconds per hour-

Paul Christiano 50:46 Times 24 times 365 times 20.

Dwarkesh Patel 50:50 Okay, so that's 630 million seconds.

Paul Christiano 50:53 That means, like, the average synapse is seeing, like, 630 million... And I don't know exactly what the numbers are, but something that's ballpark. Let's call it like a billion action potentials and then, you know, there's, there's some resolution. Each of those carries some bits, but let's say it carries, like, 10 bits or something, um, just from, like, timing information at the resolution you have available, then you're looking at, like, 10 billion bits. So each parameter is kind of like, how much is a parameter seeing? It's like not seeing that much. So then you can compare that to, like, language. I think that's probably less than, like, current language models see, and current language models are... So it's, like, not clear you have a huge gap here, but I think it's pretty clear you're gonna have a gap of, like, at least three or fours in magnitude.

Dwarkesh Patel 51:30 Didn't your wife do the, the Lifetime Anchors where she said it would, the amount of bytes that, uh, a human will see in their lifetime was 1E24 or something?

Paul Christiano 51:40 The number of bytes a human will see is 1E24. Mostly this was organized around total operations performed in a brain, right?

Dwarkesh Patel 51:45 Oh, okay. Never mind. Sorry.

Paul Christiano 51:46 Yeah.

Dwarkesh Patel 51:46 Yeah.

Paul Christiano 51:46 Yeah, so I think that, like, the story there would be, like, a brain is just in some other part of the parameter space where it's, like, using a lot- a lot of compute for each piece of data it gets, and then just not seeing very much data in total. Yeah, there's just not- it's not really plausible if you extrapolate out language models, you're gonna end up with, like, a performance profile similar to a brain. I don't know how much better it is. Like, I think... So I did this, like, random investigation at one point where I was like, "How good are things made by evolution compared to things made by humans?"

Dwarkesh Patel 52:11 Right.

Paul Christiano 52:11 Um, which is a pretty insane seeming s- exercise. But, like, I don't know, it seems like orders of magnitude is typical. Like, not tens of orders of magnitude, not factors of two. Like, things by humans are 1,000 times more expensive to make or 1,000 times heavier per unit performance. If you look at things like how good are solar panels relative to leaves, or how good are muscles relative to motors, or how good are livers relative to systems that perform analogous chemical reactions in industrial settings.

Dwarkesh Patel 52:35 Was there a consistent, uh, nu- number of orders of magnitude in these different systems, or was it all over the place?

Paul Christiano 52:41 Uh, so, like, very rough ballpark, it was, like, sort of for the most extreme things, you were looking at, like, five or six orders of magnitude. And that would especially come in, like, ene- energy cost of manufacturing where, like, bodies are just very good at building complicated organs, like, extremely cheaply. Um, and then for other things like leaves or eyeballs or livers or whatever, you tended to see more, like, if you set aside manufacturing costs and just look at, like, operating costs or, like, performance trade-offs. Like, I don't know, more like three orders of magnitude or something like that.

Dwarkesh Patel 53:10 Or, or some things that are, uh, on the smaller scale, like the nano-machines or whatever, that we can't do at all, right?

Paul Christiano 53:16 Yeah. That's, I mean, yeah. So it's a little bit hard to say exactly what the task definition is there. Like, you could say, like, making a bone. We can't make a bone, but you could try and compare a bone. The performance characteristics of a bone is something else. Like, we can't make spider silk-

Dwarkesh Patel 53:27 Mm-hmm.

Paul Christiano 53:27 But you could try and compare the performance characteristics with spider silks, like, things that we can synthesize.

Dwarkesh Patel 53:31 The reason this would be is, why, that evolution has had more time to, uh, design these systems or...

Paul Christiano 53:36 I don't know. I was just mostly just curious about, like, what the performance...

Dwarkesh Patel 53:39 Uh-huh.

Paul Christiano 53:39 I think, like, most people would object. They'd be like, "How did you choose these reference classes of things that are, like, fair intersections?" Some of them seem reasonable, like eyes versus cameras seems like just everyone needs eyes, everyone needs cameras. It feels very fair. Photosynthesis seems, like, very reasonable. Everyone needs to, like, take solar energy and then s- like, turn it into a usable form of energy. Um, but it's just kind of, I don't really have a mechanistic story. Evolution in principle has spent, like, way, way more time than we have designing. It's absolutely unclear how that's gonna shake out. My guess would be in general, like, I think there aren't that many things where humans really crush evolution where you can't tell, like, a pretty simple story about why. So, like, for example, roads and moving over roads with wheels crushes evolution, but it's not like an animal, like, would have wanted to design a wheel. Like, you're just not allowed to, like, pave the world and then put things on wheels if you're an animal. Maybe planes are more... Anyway, whatever. There's various things you could try and tell. There's some things humans do better, but it's normally, like, pretty clear why humans are able to win when humans are able to win. The point of all this was, like, it's not that surprising to me. I think this is mostly, like, a pro short timelines view. It's not that surprising to me if you tell me, like, machine learning systems are, like, three or fours in magnitude less efficient at learning than human brains. I'm like, that actually seems like kind of in distribution for other stuff.

Dwarkesh Patel 54:45 Mm-hmm.

Paul Christiano 54:45 And if that's your view, then I think you're, like, probably gonna hit, you know, then you're looking at, like, 10 to the 27 training compute or something like that, which is, is not so far.

Dwarkesh Patel 54:53 Well, we'll get back to the timeline stuff in a second. Uh, at some point, we should talk about alignment, so let's-

Paul Christiano 54:57 Yeah.

Dwarkesh Patel 54:57 ... let's talk about alignment. At what stage does misalignment happen? So right now with something like GPT-4, I'm not even sure it would make sense to say that it's misaligned, um, 'cause it doesn't, it's not aligned to anything in particular. Uh, is it, is that at human level where you think the ability to be deceptive comes about? Uh, what is the process by which misalignment happens?

Paul Christiano 55:19 I think even for GPT-4, it's reasonable to ask questions like, are there cases where GPT-4 knows that humans don't want X but it does X anyway? Like, where it's like, well, I know that, like, I can give this answer, which is misleading and if it was explained to a human what was happening, they wouldn't want that to be done, but I'm gonna produce it. I think that, like, GPT-4 understands things enough that you can have, like, that misalignment in that sense. Yeah, I think GPT, like I've sometimes talked about being, like, benign instead of aligned, meaning that, like, well, it's not exactly clear if it's aligned or if that context is meaningful. It's just, like, kind of a messy word to use in general. But I think what we're more confident of is it's, like, not doing... You know, it's not-... optimizing for this goal, which is like at cross-purposes to humans. It's either optimizing for nothing or like maybe it's optimizing for what humans want or close enough or something that's like an approximation, good enough to still not take over. But anyway, I'm like, some of these abstractions seem like they do apply to GPT-4. Um, it seems like probably it's not like egregiously misaligned. It doesn't... it's not doing the kind of thing that could lead to takeover, we'd guess.

Dwarkesh Patel 56:16 Suppose you have a system at some point in which ends up in it wanting takeover, what are the checkpoints? And also, what is the internal... Is it just that it, to become more powerful, it needs agency, and agency implies, uh, other goals? Or do you see a different process by which misalignment happens?

Paul Christiano 56:31 Yes. I think there's a couple possible stories for getting to catastrophic misalignment and they have slightly different answers to this question. Um, so maybe I'll just briefly describe two stories and try and talk about when they can... when they start making sense to me. So one type of story is you train or fine-tune your AI system to do things that humans will rate highly or that, like, get other kinds of reward in a broad diversity of situations and then it learns to, in general, dropped in some new situation, try and figure out which actions would receive a high reward or whatever, um, and then take those actions. And then when deployed in the real world, like, sort of gaining control of its own training data provision process is something that gets a very high reward, and so it does that. So this is like one kind of story. Um, like, it wants to grab the reward button or whatever. It wants to intimidate the humans into giving it a high reward, et cetera. I think that doesn't really require that much. This basically requires a system which is like, in fact, looks at a bunch of environments, is able to understand, like, the mechanism of reward provision as like a common feature of those environments, is able to think in some novel environment, like, "Hey, which actions would result in me getting a high reward?" And it's thinking about that concept precisely enough that when it says high reward, it's saying like, "Okay, well, how is reward actually computed?" It's like some actual physical process being implemented in the world. My guess would be, like, GPT-4 is about at the level where with handholding you can observe this kind of, like, scary generalizations of this type, although I think they haven't been shown basically. Um, that is, you can have a system which in fact is fine-tuned out a bunch of cases and then some new case will try and, like, do an end run around humans even in a way humans would penalize if they were able to notice it or would have penalized in training environments. So I think GPT-4 is kind of at the boundary where these things are possible. Um, examples kind of exist but are getting significantly better over time. Um, I'm very excited about, like, there's this anthropic project basically trying to see how good an example can you make now, um, of this phenomena. And I think the answer is like kind of okay, probably (laughs) . Um, so that just I think is gonna continuously get better from here. I think for the level where we're concerned, like, this is related to me having really broad distributions over how smart models are. I think it's, like, not out of the question that you take GPT... Like, GPT-4's understanding of the world is, like, much crisper and, like, much better than GPT-3's understanding, um, just like it's really like night and day. And so it would not be that crazy to me if you took GPT-5 and you trained it to get a bunch of reward and it was actually like, "Okay, my goal is not doing the kind of thing which, like, thematically looks nice to humans. My goal is getting a bunch of reward. Um, and then we'll generalize in a new situation to get reward."

Dwarkesh Patel 59:03 Oh, and by, by the way, this requires it to consciously want to, uh, do something that it knows that humans wouldn't want it to do? Or is it just that we weren't good enough to specify that the thing that we accidentally ended up rewarding is not what we actually want?

Paul Christiano 59:18 I think the scenarios I am most interested in and most people are concerned about from a catastrophic risk perspective, it involves systems understanding that they're taking actions which a human would penalize if they, the human was aware of what's going on such that you have to either deceive humans about what's happening, um, or you need to, like, actively subvert human attempts to correct your behavior.

Dwarkesh Patel 59:36 Right.

Paul Christiano 59:37 So these, the failures come from really this combination or they require this combination of both, like, trying to do something humans don't like and understanding that humans would stop you.

Dwarkesh Patel 59:44 Right.

Paul Christiano 59:45 I think you can have only the barest examples. You can't have the barest examples for GPT-4. Like, you can create the situations where GPT-4 will be like, "Sure, in that situation, like, here's what I would do. I would like go hack the computer and change my reward." Or in fact, we'll, like, do things that are, like, simple hacks or, like, go change the source of this file or whatever to get a higher reward. They're pretty weak examples. I think it's plausible GPT-5 will have, like, compelling examples of those phenomena. I really don't know. This is very related to, like, the very broad error bars on, like, how competent such systems will be when.

Dwarkesh Patel 1:00:11 Mm-hmm.

Paul Christiano 1:00:12 Um, that's all with respect to this first mode of, like, a system is taking actions that get reward and, like, overpowering or deceiving humans is helpful for getting reward. There's this other failure mode and other family failure modes where AI systems want something potentially unrelated to reward. I understand that, like, they're being trained and, like, while you're being trained, there are a bunch of, like, reasons you might want to do the kinds of things humans want you to do. But then when deployed in the real world, if you're able to realize you're no longer being trained, you no longer have reason to do the kinds of things human want. You'd prefer to, like, be able to determine your own destiny, like, control your own... your own computing hardware, et cetera, which I think, like, probably emerge, like, a little bit later than systems that try and get reward. And so will generalize in scary unpredictable ways to new situations. I don't know when those appear. But also, again, broad enough error bars that it's, like, conceivable for systems in the near future, you know? I wouldn't put it, like, less than one in 1,000 for GPT-5, certainly.

Dwarkesh Patel 1:01:03 If, if we deployed all these AI systems and so- some of them are reward hacking, some of them are deceptive, some of them are just normal, whatever, h- how do you imagine that they might interact with each other at the expense of humans? Uh, how, how hard do you think it would be to... for them to communicate in ways that we would not be able to recognize and coordinate, um, coordinate at our, at our expense?

Paul Christiano 1:01:23 Yeah, I think that most realistic failures probably involve two factories interacting. One factor is, like, the world is pretty complicated and the humans mostly don't understand what's happening. So, like, AI systems are writing code that's very hard for humans to understand maybe how it works at all, but more likely, like, they understand roughly how it works, but there's a lot of complicated interactions. Um, AI systems are running businesses that interact primarily with other AIs. They're, like, doing SEO for, like, AI search processes. They're, like, running financial transactions, like, thinking about a trade with AI counterparties. Um, and so you can have this world where even if humans kind of understand the jumping off point when this was all humans, like, actual considerations of, like, what's a good decision, like, what code is going to work well and be durable or, like, what marketing strategy is effective for selling to these other AIs or whatever is kind of just all mostly outside of sort of humans' understanding. I think this is, like, a really important... Again, when I think of, like, the most plausible...... scary scenarios. I think that's, like, one of the two big risk factors. And so in some sense, your first problem here is, like, having these AI systems who understand a bunch about what's happening and your only lever is, like, "Hey AI, do something that works well." So you don't have a lever to be like, "Hey, do what I really want." You just have the system, you don't really understand, you can observe some outputs like did it make money? And you're just optimizing or at least doing some fine tuning to get the AI to use its understanding of that system to achieve that goal. So I think that's, like, your first risk factor. And, like, once you're in that world, then I think there are, like, all kinds of dynamics amongst AI