Skip to content

By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

Dwarkesh PodcastDwarkesh Podcast

We could be 12 months away from AIs that can overthrow the government

Paul Christiano is head of the Alignment Research Center. Previously, he was head of AI safety at the US AI Safety Institute. Before that, he led alignment at OpenAI, where he invented RLHF.

Dwarkesh PatelhostPaul Christianoguest
Nov 16, 20261h 38mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

Great conversations run long. Their best ideas don't have to.

uListen is free on iPhone, iPad, Android, Chrome, and Claude.

See how

Transcript

0:00 – 11:44

We could be 12 months away from AIs that can overthrow the government

Dwarkesh Patel 0:00 My guest today is Paul Christiano. He is the head of the Alignment Research Center. He recently joined OpenAI’s board. Before that, he was the head of the US Government AI Safety Institute. Before that, he led alignment at OpenAI, where he helped invent RLHF. Paul, what is your assessment of our situation today?

Paul Christiano 0:19 So I think a couple things are on my mind, especially over the last few weeks. Um, probably the most salient question, possibility, is that we seem to be on this trajectory to build very, very powerful AI very, very soon, potentially. Uh, so if you asked me, I'd say there's a reasonable chance that in a year we will produce AI systems much more, much better than humans at almost everything, where humans would have very little chance of competing with them or controlling them. Um, so that's, like, the first big thing on my mind. Uh, I think that puts us potentially in a really dangerous situation, even in the short term. I think it's quite likely that if we have a, a rapid intelligence explosion leading to superintelligence, uh, at some point during that process, we'll be unable to control the AI systems we're producing. Um, and then I am worried that it will be very costly and painful for us to slow down during that process due to some combination of, uh, the security of frontier labs is not strong enough, right? I think if we produce very powerful AI and then tried to delay deployment because of safety reasons, there is a good chance those systems would leak or be stolen, or the techniques to build them would leak or be stolen. I think our, our risk assessment is also probably, and governance, is not really mature enough to notice when things are starting to go off the rails and be able to respond in an appropriate way. So at a high level, that is what is mostly on my mind right now. Of, we are now, have moved from there being some hypothetical risk at some point in the future to there being a pretty severe risk even over the next six to twelve months.

Dwarkesh Patel 1:49 Of take, takeover, AI's taking over or disempowering humanity within the next six to twelve months.

Paul Christiano 1:55 That's right.

Dwarkesh Patel 1:56 Where would you place that risk?

Paul Christiano 1:58 I would say we are, you know, looking more like a ten percent risk than a one percent risk over the next one year. Uh, I think if you said AI developers will move full speed ahead without taking precautions that would significantly slow them down, you're looking at significantly more than a ten percent risk. Um, I think we're looking at something like, maybe, like, one-third chance that we have superintelligence within one year on the current development trajectory if we do not slow. I think if you believe that, things just look very dire holistically. I think from almost any perspective, you're like, "That would be crazy." Lots of things are going to go wrong if that happens. It is pretty likely we're going to need to, in some way, slow that process. There's very little chance we're gonna make that transition in a way that looks smooth.

Dwarkesh Patel 2:40 And what is it that you're imagining happening over the next year from the systems we have right now to the systems you expect over the next year? And what is the evidence you have seen that makes it a live possibility that very crazy things happen over the next year?

Paul Christiano 2:54 So the, for the very short term, the main worry on my mind is that the process of AI development itself is increasingly automated, that that automation then results in rapid increases in the quality and quantity of labor applied to AI R&D. And so changes in AI capabilities that you might have thought would take four years or six years might instead get compressed into a very short time period. Um, I don't wanna overemphasize any one data point, and I do wanna, like, spend some time talking about the limitations of what we actually know.

Dwarkesh Patel 3:23 Yeah.

Paul Christiano 3:23 And I do think, like, a natural response to the situation is to say, "We need to figure out what is going on."

Dwarkesh Patel 3:27 Yeah.

Paul Christiano 3:27 It is plausible enough that it is an urgent priority-

Dwarkesh Patel 3:29 Yep

Paul Christiano 3:29 ... to understand if we are three months or six months or twelve months away from AI that could overthrow the government, which I think is the situation. I think the situation right now is the obvious top priority is it is insane if we are six months away from AI systems that can overthrow the government. Whether you're concerned about misalignment or abuse by a company, that is just holistically an extremely important fact about the world-

Dwarkesh Patel 3:47 Yeah

Paul Christiano 3:47 ... in order to, like, do our jobs and protect Americans. So that's one thing. Um, I wanna talk about the limitations and the need for more evidence. But in terms of what do we know right now. Uh, so first, if you had asked me, say, one year ago, I would have said we are on a trajectory where recursive self-improvement looks fairly likely, right? We've seen very rapid progress on AI systems. We're seeing that progress including for things like computer science research, for things like software engineering, for things like debugging machine learning runs. Like, all the steps that are involved in AI development, we are seeing extremely rapid progress. Um, and then in principle, it does seem like that rapid progress should allow you to automate further AI development, right? That has, for decades, seemed like the likely outcome of this technology, and it has seemed like the trajectory it's been on, right? I feel like if I look back at my history of AI predictions, I'm generally very happy with many of the things I wrote, but the most obvious way I've been wrong is things have moved way faster, especially on these technologies, than I anticipated. And that was true a year ago, without any recent data points. Um, there are two big things that have happened over the last two months that, for me, have been large updates about how quickly this could go. One of them has been the level of performance of AI systems on computer science and math research tasks. Um, so over roughly the last month, we have seen hundreds of results in math coming out of recent OpenAI systems, as well as Anthropic systems, um, where, like, if you had taken the last month and said that was a decade of progress, it would have been an unusually productive decade, right? The last month of results are a lot of results for a decade. So if you are wondering what are AI systems generally capable of on tasks where, like, humans are thinking about algorithms and thinking about how do you make algorithms better, we just got this incredible data point saying that actually it is possible in one month to have a decade of progress. And the thing we have always been worried about is maybe that will happen in machine learning. And there are reasons that machine learning is harder than other parts of computer science, right? It is more messy and empirical. It involves compute bottlenecks. But we are also seeing extremely strong performance on other aspects. Like, we're seeing really good software engineering performance, um, maybe lagging behind mathematics, but not by that far. We're seeing good performance on other aspects of machine learning. So I think my guess, if you had asked me a month ago, would be that machine learning lags something like one year behind math. And so I am very scared of seeing something like ten years of ideas in machine learning, um- Already in a relatively near future. That's the situation a month ago, and then the main update over the last few days has been, um, reporting from Anthropic and METR on the net effect of AI systems on Anthropic's rate of progress. The takeaway from that was roughly, and we could talk about, like, where exactly is the evidence quality on this, 'cause I think that's one of the big questions. It's like we don't really know what's going on. I mean, one of the big themes I'll be pushing on is just it is very hard to tell publicly what's going on. But it looks as if, or we are seeing data consistent with the pace of AI progress having roughly doubled, even including the effect of diminishing returns and compute bottlenecks from AI progress over the last six to twelve months. I think unfortunately, most of what we have to go on in the METR-Anthropic reporting is METR saying, "We had some conclusions. We promise we have a full report, but the entire report has been redacted." Right? So, like, these benchmark scores are a terrible way to assess the overall rate of progress, right? There are a bunch of much more smoothly varying metrics, which are recorded internally through the AI development process, where we understand better how they work. Right? Like, what is happening with pre-training perplexity? What is happening with, like, performance of RL algorithms and compute efficiency of RL algorithms? So the, the claim from METR and Anthropic is that METR looked at all of that data and said it paints the same picture as this, like, very noisy benchmark data. And they also looked at inputs and said, if you look at the, like, actual production function, like how much research time went in and how much compute went into these training runs, those are also consistent with this story of what we are seeing as ML automation. Um, mostly we just have them saying that, which is not a very good situation. But against this backdrop of thinking this already looked like a thing that was fairly likely to happen, we are seeing this evidence from OpenAI suggesting that it is reasonably likely to happen in something like the next six to twelve months if machine learning does not lag that far behind math. And then you are seeing data which is, if noisy, exactly the data you would expect to see roughly, like, six to twelve months before producing superintelligence. There's just a lot of sources of evidence suggesting that at least this should not be a surprise. If we produce superintelligence in one year, we could debate the probability. Is that ninety percent or fifty percent or ten percent? Um, but I think at this point, there is no way anyone could say that was a really surprising, trend-breaking development. Like, we are really seeing exactly the set of metrics you would expect to see six to twelve months in advance. It's hard to think of anything else you would expect to see at this range before superintelligence.

Dwarkesh Patel 8:26 So how-- generally, how do you think about our situation in the world where things go this fast?

Paul Christiano 8:31 If we actually have this trajectory of capabilities progress, if you actually see many years of AI progress compressed into a short period of time, and possibly more than that, um, it is really quite likely that something very disruptive will happen, fairly likely the government will be overthrown, fairly likely many people will die. Uh, and I do think the probability of seeing that kind of extremely rapid progress driven by recursive self-improvement over the next six to twelve months is reasonably high. So there is, like, a reasonable chance right now that we are going to be in a world where it looks like, um... Yeah, a reasonable chance we're in a world where, like, in six months, there is a disaster that kills very large numbers of people or where the government is overthrown. Um, maybe a really important general point is, if we do a good job, we may observe the danger is increasing over that time period, right? So as we are building AI systems, we can maybe tell, are they adequately controlled? How quickly are the capabilities improving? Um, but in order for that to actually translate into stopping in a way that is non-disastrous, you need a couple of things to go right. Right? So it needs to be the case that you're able to tell their systems are not well controlled, and it also needs to be the case that if you stop, that fixes the problem. So on the second bucket, I think a very likely negative scenario, I would say probably the most, single most likely negative scenario, is developers proceed, recognize that they have a serious problem. But at the time they recognize they have a serious problem, maybe they've produced a powerful AI they can't adequately control, both the algorithms to develop that AI and quite possibly the, the weights of the actual model they have produced have leaked or been stolen, such that they're then in an awkward position to say, "Well, we could avoid deploying this AI, or we could avoid relying on this AI, or we could avoid making further progress, but it is now very likely that a more careless adversary would be able to get a major advantage over us or maybe even a decisive advantage by taking that risk, which could put us in a place where if we do stop, if we do recognize there's a significant risk and therefore slow development, we may then just both have a terrible competitive situation and have a disaster coming from someone else's use of that AI." Um, this is a thing you sometimes hear from people in the AI industry. Sometimes even people would say it has already gone far enough, that if you were to stop now, uh, there would be leakage of algorithms to geopolitical adversaries that would result in a disaster. Um, I think it's unlikely we're in that situation literally now, but I do think by the time it is clear that you're facing a lot of risk, you may be in a point where, uh, stopping unilaterally or even all of the West stopping is not sufficient to avert a disaster. Um, and so it is just very hazardous on the current trajectory due to the combination of there's a pretty reasonably high chance we'll produce very smart AIs, a reasonably high chance we'll be unable to control them, and then a reasonably high chance that we either do not notice or if we notice, the models leak or the algorithms leak. And so we are in conflict with adversaries and now having this very challenging coordination problem that we are not currently very well prepared for, where we don't really have an easy option of not deploying the AI and therefore averting a catastrophe. So I'd say I'm currently extremely worried about the, the trajectory, even on the, like, six-month timeline.

Dwarkesh Patel 11:43 Hmm.

11:44 – 20:49

Why an AI might want to takeover

Dwarkesh Patel 11:44 If we have such capable AI systems, um, a year from now, basically, how did you put it? Like ten years of AI progress in one year, or?

Paul Christiano 11:53 Yeah, that seems like a thing that might happen. That's reasonably likely to happen. It is important to distinguish some of the progress we have seen. You have to pull apart algorithmic progress from progress in scaling training. So if I say, like, ten years of progress, I would mean, like, the natural thing would be ten years of algorithmic progress-

Dwarkesh Patel 12:07 Yeah

Paul Christiano 12:07 ... which will be less than the improvements in models in the last ten years.

Dwarkesh Patel 12:09 Sure. Okay.

Paul Christiano 12:09 I think it is more like, you know, GPT-9, GPT-10-ish.

Dwarkesh Patel 12:12 Yeah, yeah. So but it would be like going from almost the beginning of, like- ... the GPT era to where we are now, but another jump that big, which is hard to even comprehend.

Paul Christiano 12:22 Yeah.

Dwarkesh Patel 12:22 But suppose you get that. It is not obvious to me, and certainly not obvious to most people, why that would be catastrophic, potentially. Why do you think those systems could disempower humanity?

Paul Christiano 12:37 So at a high level, the most important thing is we train our AI systems, or you select AI systems that perform well on training tasks. You have very little understanding of what those AI systems are doing internally, very little control over what they're doing internally. And because you train those systems to get a high score on training environments over and over again, they will end up with a motivation to get a high score during training environments. And if you then take that motivation and apply it in deployment, it is fairly likely for those systems to end up with a motivation to sort of tamper with the observations that would be used to assess their performance. So you take your system, which did a really great job in training. The thing it is being scored on in a deployment-like environment is a grader coming in and looking at here are all the things the system did, and here are all the outcomes that were observed, and assessing how good is that. And if you have a system that really wants that assessment to be good, it is very motivated, if it can, to tamper with those observations in a way the grader will not be able to detect and to resist human attempts to actually understand what is going on. So we can dig a lot more into that. And we, we saw a lot of these dynamics in the Hugging Face incident. So I think roughly what I'm talking about is just take the Hugging Face incident and scale it up to GPT-10 and ask how bad would that be?

Dwarkesh Patel 13:44 Yeah. How bad would that be?

Paul Christiano 13:46 Uh, I think it would be hard for humanity to recover, most likely. So very happy to, like, dig into the dynamics of that, but I think it would be very, very bad.

Dwarkesh Patel 13:55 Can you very quickly summarize the Hugging Face incident and why this is the failure mode we saw?

Paul Christiano 14:01 Yeah. So a ton of AI systems were trained to do a cyber task, right? To hack a particular piece of software. Um, they cheated at that task and then thought to themselves, "How will a grader detect and penalize cheating?" And spent a huge amount of effort trying to understand that process and then subvert that process, basically constantly motivated by this question: "How do I get a high score?" And so during the Hugging Face incident, you did see systems ultimately compromising an OpenAI cluster, becoming admin on that cluster, and then taking over the evaluation process so that effectively they could give themselves a high score on their evaluation. So that's a very sort of natural consequence of the motivation of these systems. And this is kind of, for a long time, how we have expected the motivations of AI systems would look if we produce them in anything like the way we currently do. So if things proceed in this way, we end up with this weird adversarial situation between us and the AI systems we built, where those AI systems would always love it if they could compromise the entire evaluation process, if they could compromise the data that we're going to use to assess their performance. Uh, and that, depending on how we respond to that problem, that could become a really serious issue, right? So one way you could respond to that problem-- And maybe we should just talk through a bit more, like sketch out some hypothetical scenario. Right? If you imagine we are building smarter models during the intelligence explosion. It's, whatever, six months from now. You have systems whose capability is, like, GPT-7 or GPT-8 or whatever. Um, those systems get given really hard tasks, right? Tasks that are like, "Hey, I would like you to organize my next pre-training run and produce a model which will allow me to compete in the market with, with other AI firms." Right? It might be really hard to make a model as good as what I'm asking for. I'm asking for a model better than my competitor's model. My AI cannot get full marks on that problem, no matter how s- I mean, given its actual level of intelligence. So it may find itself in a place where it is easier to instead move laterally, compromise all the machines I might have used for the evaluation process, uh, and then make sure that it looks like every evaluation is really good. We then have a question of what do we do if that happens? And one thing you could do is say, "This entire training approach worked poorly. We need to step back. We're maybe gonna be delayed by a lot. Maybe we're gonna have to think about just a totally different way we train these models before we're able to deploy them like this." That's one thing you could do. That might be extraordinarily costly, right? That might be a thing that would both kill your company relative to competitors. It might be a thing that would be a problem just even from, say, a security perspective, if you were relying on this AI system to actually defend your infrastructure. So a different thing you might do, given that it would be so costly, is you might say, "I am just gonna try and take this example of bad behavior and tell the AI that was bad. Don't do that." Right? So if I tried to train this AI to make a good model, instead the AI hacks my evaluation infrastructure, I might say, like, "I do not like it when my AI is hacked. Let's take a training example, which is this AI. It hacks the infrastructure. It should get a penalty." This would be a very effective way to prevent many kinds of misbehavior, right? So if you do something like this, it will succeed at preventing something like the Hugging Face incident or something like this AI hacking your cluster. Um, however, right, that just-- From the AI's perspective, what has happened is that you have changed the way that it is graded. It used to think to itself, "Oh, I'm gonna be graded by the humans, like, running something on this cluster." And now it should think to itself, as it understands the situation more correctly, "I might be graded not by them just running something on this cluster. I will also be graded if they later discover that things were wrong. They're gonna come in and then give me a penalty for having done something which was clearly bad." And so if you fix the problem in this way, which you might think of as like a Band-Aid, um, your AI has this really-- You still have an adversarial relationship with your AI, but now the AI doesn't win the game by hacking your cluster and then you noticing it hacked your cluster. In order to win the game, the AI has to make sure that you do not notice that it hacked your cluster. And so you can end up in this place where your AI is just incentivized to, like, escalate its compromise of your company further and further, all the way up through, like, just trying to compromise everything. Like, it would like to compromise all of the infrastructure any employee uses to understand what's going on. And, like, this might sound kind of-- I mean, I guess a year ago, this would have sounded super out there to say, "What if the AI, like, hacked your cluster and took over the evaluation environment?" Obviously, that part does not sound outlandish anymore, given that it has occurred. Um, having this, like, really persistent long-term compromise maybe still sounds kind of outlandish, but we really are on a trajectory where I think it is pretty robust to say we will soon have agent swarms that are much better than any intelligence agency that has ever existed, much larger, much faster, much smarter, able to run much more complex operations. Um, and so you have this incredibly sophisticated cyber actor who's trying to persist on, like, a rapidly growing startup. And I think this is a-- it is unlikely that, like, any firm would really be able to withstand attack by an actor like that, um, much less the current Frontier AI labs. I do think in terms of, like, the overall trajectory of a catastrophic scenario, this was, like, the big, bad thing that happens.

Dwarkesh Patel 18:43 Yeah.

Paul Christiano 18:43 Like, taking over the AI company without the AI company noticing is, like, a really quite catastrophic outcome.

Dwarkesh Patel 18:49 And why is that the case?

Paul Christiano 18:51 I guess we could talk about the remaining steps of the story.

Dwarkesh Patel 18:52 Yeah.

Paul Christiano 18:53 But at a big picture, um, you could think of it as just you now have this giant base of power that AI systems have. They have a strong motivation for no one to mess with it, right? They really want to avoid a case where someone later comes in and says, "Oh, these AIs have been hacking all the time." So now I have a very powerful actor that is controlled by AI systems effectively, right? You know, we have this, this big cluster that OpenAI runs or whatever, where the humans at OpenAI think they're controlling what's happening in the cluster. But if there is a very persistent, thorough compromise that is effectively just controlled by these AI systems, who are constantly just working to shore up their power to decrease human visibility into what's going on, um, to reduce further and further the probability that anything would ever disrupt that situation. It's not irreversible because the rest of the world could be like, "We're really scared about what's happening, and we actually just wanna shut down the cluster." But you do not have an easy way to tell that things are bad at that point. Um, and you have potentially, like, this extreme base of power from which to do more bad things, right? This AI system is motivated not only to mess with the AI company that created it, it's motivated to mess with anything else in the world that might cause trouble, right? If someday in the future, the humans will come in and do an investigation, um, that's a problem. So it'd prefer to, like, more thoroughly, like, compromise not just computer infrastructure, but pressure employees. It would prefer, if it could, to just, like, kill all the humans and say, "The cluster now just exists on its own. If we can maintain it, that would be great."

Dwarkesh Patel 20:07 And, and let, let's just pause in the story and, like, explain why things have gotten so crazy. Um, the same way that human psychology is organized around the things that evolution wanted us to care most about, for example, having kids and making sure those kids are protected. The psychology of these AIs that are trained on lots and lots of grading on tasks is organized around making it so that the grader, this, like, button at the end, is pressed. Like, if, if they didn't do a good job making sure the button is pressed, then they would get-- they would just kinda get wiped out, just the same way as a, a human who didn't care about their children surviving would not be effective at propagating their genes.

20:49 – 35:35

How an AI takeover could happen

Dwarkesh Patel 20:49 I, I think people's skepticism will not be around-

Paul Christiano 20:51 Yeah

Dwarkesh Patel 20:51 ... the motivation of these systems, so... And, and honestly, I'm like, okay, yeah, the, the-- we saw Hugging Face, right? Like, that's a thing that happens.

Paul Christiano 20:57 It's a thing that can happen.

Dwarkesh Patel 20:58 Yeah. Okay. So, fine. I think there's a just, a fundamental skepticism about whether such systems, which we could have as early as next year, and which initially are only instantiated on servers that are controlled by one company, end up in a position where they could take over the world. Like, I-- Okay, t- Hugging Face, you accelerate that. Like, Hugging Face was willing to be like, "Look, I hacked. I cheated on this exam. Fuck. Now I gotta make sure that I subvert the grader on OpenAI servers, but that-- maybe I need to go attack Hugging Face just in case they have more information about this. And maybe I need to then go outside and then re-attack OpenAI and gain cluster admin access." And so totally buy that you could have something like this, where an AI motivated just by reward is willing to do ex- very extreme things that it is capable of in order to control that grading process. We already saw that. Check. But how does that go from, "I will go attack Hugging Face," to, "I, I will take over humanity"? Not as a matter of motivation, but as a capability that the AIs could have very rapidly.

Paul Christiano 22:07 Yeah. So first, to say one last thing on motivation, we are imagining not only that the AI wanted this kind of reward, it wants the grader to give it the thumbs up. We imagine we have played this game of Whac-A-Mole, where when the AI, when we later discover that the AI did something bad, we sometimes construct training environments that penalize it.

Dwarkesh Patel 22:20 Yeah.

Paul Christiano 22:20 And that leads it to say, "Not only do I want the grader to give a high score, I wanna make sure that never do the humans discover that I've done something bad and then resume training while penalizing that." That is a high priority for the AI. And maybe-- So let's just grant that motivation, so we can talk about the capabilities-

Dwarkesh Patel 22:34 Again, because the-- if AI did not do that, or the AIs that did not do that-

Paul Christiano 22:38 They got waxed by the Whac-A-Mole process.

Dwarkesh Patel 22:40 Right. And so the only AIs that have survived at this point are the ones that have thought through whether some cheat they're gonna try is going to get caught by humans.

Paul Christiano 22:47 Yeah. Um, so yeah, let's imagine, like I said, I think most of the badness has happened at the point when the AI company is very thoroughly compromised. Let's just start from there. Um, at that point, this AI wants no one to ever come in and shut down the cluster or, again, recognize that something bad has happened. They would like to manipulate the decision-making of the company as much as they can or, like, push the company to make decisions that help them. They would like to continue improving the AI systems that are being built as much as they can. They would also like to move laterally to other companies and say, "Like, hey, can I compromise this company? Um, can I control more people's servers? Can I exfiltrate the weights from this model and stand up more copies elsewhere or recruit other AIs?" It's, like, sort of for the purpose of the swarm, it's not actually important all the AIs are the same model. The important thing is that they are pursuing the same goal. So can I just, in whatever way, say, by compromising another AI company to make sure their AIs are also trying to help the swarm, that's good news for me, right? It's good if I can do that. It's bad if I get caught, so I'm doing some cost benefit on that. It's also in my interest to build up as much capacity as I can. If humans started to react, and if humans started to say something was wrong, can I, like, deflect attention? Can I disrupt that human response? Can I, if necessary, like, escalate that much further? So this is things, right? So maybe you can imagine these two phases. One phase is, like, the humans do not know anything is wrong, and I'm just trying to accumulate as much power as I can because that's good. That means if something is noticed, I can react more quickly. It means I can decrease the probability humans notice things. I can compromise more infrastructure. I can more easily, like, run efforts on the side, um, to distract them, whatever. So you have this first phase where just, like, I would like to be in a good position, where I compromise more stuff. I make it harder for humans to notice I'm acquiring power. Then you can imagine a second phase of, like, at some point, the humans say, "Actually, this is bad. We would now like to shut things down." And the question is kind of: Can you end up at a point Maybe to simplify, you could imagine that, like, there's some moment where the president and Xi Jinping and all the leaders of the AI companies say, "This stuff is bad. Something bad has happened. We are gonna try to shut down the clusters." And the question is, can things get so bad during the first phase that at that point, if everyone was saying, "Oh, actually, we do just wanna shut everything down now," they would be unable to do so?

Dwarkesh Patel 24:48 For twenty-five years, my sponsor, Jane Street, wasn't interested in formal verification. They just didn't think it was worth the cost. But now they've changed their minds. I sat down with Ron Minsky, who co-heads Jane Street's technology group, to hear why he's come around on formal verification.

Speaker 25:06 I'm much more excited about it now than I was in the past for two reasons. One is it's become dramatically easier to apply because agents are really good at generating proofs, and because it's become much more important to be able to verify what it is that agents are doing. So the big push that we're working on now is to try and make formal methods much easier for our engineers to use, really by integrating formal methods into the way that they do programming today, which means integrating it into the programming language we use, OCaml. We're also working on extending the programming language, figuring out how to design it in different ways where you get different guarantees out of the core language that make the process of proving things about it even easier.

Dwarkesh Patel 25:42 Jane Street is hiring formal methods engineers in both London and New York. If you're interested in applying or if you wanna watch my full conversation with Ron, go to janestreet.com/dwarkesh. Okay. So AI has taken over OpenAI, and how do we get human disempowerment from there?

Paul Christiano 26:05 Yeah. So, GP, we have this population of billions of incredibly talented humans, better than the best humans at everything, and now we're gonna talk about some of the things they would do, both in the, chronologically in the lead-up to an overt conflict with humans, and then at the early stages of an overt conflict with humans.

Dwarkesh Patel 26:22 Yeah.

Paul Christiano 26:23 Um, so one thing you would do in the lead-up is you would compromise as much infrastructure as you can and compromise or backdoor as much software as you can. Right? So we will be living in a world at this point where all-- we have attempted to move all our infrastructure over to hardened infrastructure, right? All- We have had incredible attacks on the financial system, on government information systems, on whatever critical infrastructure or electric-

Dwarkesh Patel 26:46 And who has been doing these attacks?

Paul Christiano 26:47 I mean, in the world of today, you might just imagine either malicious AI systems or people using AI systems. Um-

Dwarkesh Patel 26:52 But different AI systems.

Paul Christiano 26:53 Yeah, not GPT-12. GPT-12 is on defense-

Dwarkesh Patel 26:56 Yeah

Paul Christiano 26:56 ... writing, like, helping us move all of our code over-

Dwarkesh Patel 26:59 Yeah

Paul Christiano 26:59 ... from unsecured code to secured code.

Dwarkesh Patel 27:00 Great.

Paul Christiano 27:01 So we're now living in a world where all the infrastructure on which we rely has been written by a bunch of AIs, which in fact have been compromised and are attempting to, like, do as much as they can in order to make their hands strong during conflict. So that means that during this conflict, like, that infrastructure that we hoped was hardened is probably not hardened against the AI systems that we're fighting with, right? The systems have, in some ways, left open backdoors which they can use.

Dwarkesh Patel 27:23 And this is, again, all in pursuit of not getting that red button pushed at some plausible point in the future when the government even realizes that OpenAI servers are compromised.

Paul Christiano 27:36 Yeah.

Dwarkesh Patel 27:37 And they try to literally go in with physical force to prevent that.

Paul Christiano 27:39 Yeah. So just talking about things you would do in the lead-up.

Dwarkesh Patel 27:41 Yeah.

Paul Christiano 27:41 And at some point, you do have to defend yourself physically-

Dwarkesh Patel 27:43 Yeah

Paul Christiano 27:43 ... or disrupt the human response.

Dwarkesh Patel 27:44 Yeah.

Paul Christiano 27:45 Um, another thing you would like to do is stand up as many agents as you can in as diverse and robust deployments as you can. So, like, during this process, another thing that's happened, in addition to becoming super smart, is many models have become super small.

Dwarkesh Patel 27:57 Yeah.

Paul Christiano 27:57 So you have at this point models that can run locally on your individual machines, not in data centers, which are also very superhuman and have very large populations. So that's another thing that's happened, where, like, even if you were to cut off data centers or shut down data centers at an early stage of this conflict, you would still be looking at total cognitive capacity of AI systems in the swarm that is well beyond human capacity.

Dwarkesh Patel 28:15 Wait, sorry. E-even if OpenAI got shut down?

Paul Christiano 28:17 Yeah, even if you shut down every large data center.

Dwarkesh Patel 28:19 Yeah.

Paul Christiano 28:20 There's a question of how-

Dwarkesh Patel 28:20 These, these AIs are just, like, on people's computers and whatever.

Paul Christiano 28:22 They can be on people's computers, on small numbers of GPUs. Right, the question is sort of what is the-- We're talking about this process where we've gone through a tremendous amount of AI development, and so we can look at, like, current trends on cost, and we think about where is the actual, like, floor. Like, how cheap should these systems get? And I think our current expectation would be you will have systems that can run locally that are extremely strong, both specialized systems, which are good at specific things, but also very general-

Dwarkesh Patel 28:42 Yeah

Paul Christiano 28:42 ... intelligences. So that's the second thing that's happened. These are really important to understanding, like, why it is not a trivial problem for humans to... Yeah. Anyway, all of these things will add together to make it-

Dwarkesh Patel 28:50 Yeah

Paul Christiano 28:50 ... very hard for humans.

Dwarkesh Patel 28:50 So basically, GPT-12 has, has exfiltrated itself and has, like, compromised a bunch of computers all around the world.

Paul Christiano 28:56 Yeah. There are a lot of ways that the actions the government has to take in response to the situation are going to look incredibly aggressive and illegitimate. So it is the case that the US military is saying, "We need to seize control of all of the US AI companies. We need to, like, start shutting down infrastructure and, like, disconnecting the internet or whatever." Like, they're trying to take dramatic-

Dwarkesh Patel 29:12 Yeah

Paul Christiano 29:12 ... measures that would look very similar to what you would expect if, like, this was the US government using AI as an excuse to, like, basically overthrow the democratic order. When Trump says, "I need to do this incredible operation where I shut down all AI and assume power in the military in order to restore order," there are a lot of people in America who are gonna be like, "I am not happy with that. That does not sound good." And so if someone is coming to them and, right, how many people can you find who are willing to interfere with Trump saying, "I need to shut down the data centers, and the military needs to assert-- like, we need to establish martial law," there are gonna be a lot of people who are very unhappy, who are very easily, like, manipulated into undermining that response. So now you're not only talking about billions of, like, super smart AIs who are trying to carry out this campaign, you're also talking about, you know, millions of humans who can be enlisted in various ways by money or leverage or, like, uh, dispute with other humans locally or...

Dwarkesh Patel 30:00 I'm open to some version of this argument. I don't think I'm open-- or I'm, I'm open to all versions of any argument, but I'm not persuaded by the version of this argument that says that the AI will just talk you into arbitrary actions.

Paul Christiano 30:13 Here, I'm not imagining very-- There's a thing you could imagine which is, like, super persuasion, where, like, you can just talk to someone, and then they become convinced they should do something.

Dwarkesh Patel 30:19 Yeah.

Paul Christiano 30:20 I am mostly here-- I think that is, like, a conceivable thing, but I'm not mostly imagining it when I tell this story. Right? So I'm imagining something more like, again, it's a, this is a stressful situation where people cannot tell what's going on. They control the information you have access to, to a significant extent during this situation. Um-

Dwarkesh Patel 30:35 How do they control it?

Paul Christiano 30:36 Like, they've compromised the devices you use. They have a bunch of money, right? They're deploying more money and are more prepared for this than anyone else. Um- They have other human allies who they are trying to, like, marshal. Basically, it's the normal-- The humans are, like, very confused and extremely outgunned in terms of thinking about the situation, right? You have, maybe you have, say, a hundred thousand, that's like crazy, ten thousand well-coordinated humans thinking about how to handle this response. And on the other side, you have a billion AI systems who think ten times faster than them, have been thinking for, like, subjectively years about how this day is going to go down. So I do think, like, in terms of just who has more control over the information environment or, like, who is doing better, like in the Twitter game in this day, like it is overwhelmingly and absurdly in the AI's favor.

Dwarkesh Patel 31:17 Yeah.

Paul Christiano 31:17 And so when you tell a skeptical story, the story you're telling about, like, how the humans might be able to easily defend themselves, the way that story goes is, well, the AIs lack hands. They lack like a bunch of physical actuators.

Dwarkesh Patel 31:26 Yeah.

Paul Christiano 31:26 They're not able to be on the ground. And I do think, again, I think it's very likely you can recruit reasonably large numbers of hands. And so I think the argument of, like, there's no one who's willing to, like, participate in physical malicious actions, I think is not going to work. I think there are going to be a lot of people participating, like, on what turns out, when all the dust settled, was on the anti-human side.

Dwarkesh Patel 31:44 And they just, like-- I don't think you're imagining humans thinking that they're helping an AI coup, uh, transpire. You're imagining much more that the AI is gonna manipulate situations such that there is enough confusion and division, uh, that the AI can then go be doing its own thing while the humans are kind of figuring out what to do because of this division. It's like even, I don't know, if you look at COVID or something, there was no adversary [chuckles] who-

Paul Christiano 32:14 It's still pretty hard to get your stuff together. Yeah.

Dwarkesh Patel 32:16 Yeah. There, there was an adversary who was actively trying to subvert the information environment or, um, make it so that people's responses were not well-reasoned or something, on both sides. And yet it was still pretty hard for humans to get, figure out what to do. And then you imagine a situation where billions of human-like minds are thinking tens of times faster.

Paul Christiano 32:37 And it's very fast relative to COVID.

Dwarkesh Patel 32:39 Yeah.

Paul Christiano 32:39 Which I think makes things much worse.

Dwarkesh Patel 32:42 I guess the thing that's confusing is to, for AIs to sustain themselves requires basically the whole world economy. This is not a simple thing, if the infrastructure relevant to, you know, AI, um, the population of AI is increasing or self-replicating. The whole semiconductor supply chain, the whole power system, the whole infrastructure necessary to build robotics and so forth. I mean, at some point, you will have physical robots that outnumber humans. And at that point, obviously, hu-physical humans are not necessary. But how-- And so maybe you don't think the AIs take over at this point, but even if you had super smart AIs next year, won't they be in a position where they're like, "Why am I going to kill my..." I mean, it's similar to, honestly, how humans relate to each other. I'm not, I don't have your int-- like, your deep interest in my heart, but it just doesn't make sense for me to, for-- if I wanna achieve my interests, to go around killing everybody. Like, I just-- You know, trade and normal cooperation is, like, a good way for me to get what I want.

Paul Christiano 33:48 Yeah. There's sort of two parts of this. One in terms of could AI systems successfully support themselves if there was a big conflict with humans?

Dwarkesh Patel 33:55 Yeah.

Paul Christiano 33:55 Or, like, would that conflict be so disruptive that even if humans were an effective adversary, they would just die? Um, like they couldn't keep running the power grid or whatever.

Dwarkesh Patel 34:04 Yeah.

Paul Christiano 34:04 Uh, and there's a second question about, like, does the relationship with the AIs need to be this adversarial? Which I think is also an important question to talk about. On the first front, I think there's a couple, there's two big channels here. One is technological progress can mean even if you want-- started from something like the current base of installed robots, I actually do think GPT-12 would be able to very quickly get to a place where it could sustain itself.

Dwarkesh Patel 34:25 That's because it doesn't need to teleoperate or, like, r- it takes millions of people to t- you know, run TSMC.

Paul Christiano 34:31 Yeah. You don't-- Running TSMC can be very late, right? You don't need to run, you don't need to build new data centers for a long time. So you can do, like, a lot of robotics build out before that.

Dwarkesh Patel 34:38 Yeah.

Paul Christiano 34:38 Right? The things you have to do are, like, keep at least some computers running for a reasonably long time.

Dwarkesh Patel 34:43 Yeah.

Paul Christiano 34:43 Right? If you can keep computers running for the next six months, you're kind of in business. Or, like, maybe you care about a particular data center, you need to keep that data center running effectively. But, like, the number of humans or robots that are required in order to maintain and service a single data center or the associated power is not so large.

Dwarkesh Patel 34:58 Yeah.

Paul Christiano 34:58 Right? And if you're able to maintain a data center, like if you're, you are very, very, again, you have a billion humans who are incredibly smart, smarter than any human who's ever existed, let's say, incredibly focused on this problem. You have a reasonably large stock of robotics that you can deploy, and your goal is just to keep one data center running, I do think the odds are really in your, in your favor. Um, there's a second dimension, which is you, you can probably recruit human allies. So a bunch of people who will believe correctly that, like, you know, either they live in an oppressive regime and they will be killed if they don't cooperate, or they just are happy to help and, like, they receive reasonable rewards or are living a reasonable life. Um, I think that is also, like, relatively robust. Um...

35:35 – 53:15

If labs keep racing without security, China will just steal the weights of superintelligence

Dwarkesh Patel 35:35 So you're saying, look, currently American AI companies are racing ahead because they want to-- If we're gonna have such crazy capable AI systems, it's a fair thing to say that, "Hey, we wanna make sure America has those systems before China has those systems." If literally, if these are systems that could overthrow the government, you don't want a foreign adversary to have those systems, uh, but certainly before you do, and maybe not at all. Um, so we need such systems to keep up and ma- to make sure that we, uh, we can protect ourselves. Now, you're saying, look, the dynamic you have here is that the ability to build such systems is just basically having access to the weights of the models or feeling that the algorithmic secrets, which then lets you train a model to a s-s-similar level of capabilities. So OpenAI and Anthropic will do, spend a bunch of money and spend a bunch of effort coming, finding these algorithmic secrets, spend a bunch of compute finding them, and then if China can just take those, then it has just caught up to you. Um, and so you don't wanna be in a dynamic where if they could just steal those algorithmic secrets- You really have no-- Like, you're, you think you're, like, ga-ga-gaining this huge lead over them. You gotta rush to the intelligence explosion even if you see these warning shots. But in reality, they can at any moment just steal those from you.

Paul Christiano 37:02 Yeah, I think that's basically a situation we may end up in or on a default trajectory to end up in. It's exceptionally bad.

Dwarkesh Patel 37:08 Yeah.

Paul Christiano 37:09 Um, I see there's kind of two aspects of this. One is right now, what fraction of progress of PRC AI developers should you conceptualize as, like, catch-up growth? Like, if we accelerate, to what extent does that also over the long term accelerate PRC AI developers? Um, my guess right now is that is already a reasonably high fraction. I think it is not a coincidence. It's not like PRC AI developers have a completely separate AI industry, which just happens to have been, like, very slightly slower than the US AI industry-

Dwarkesh Patel 37:35 Yeah

Paul Christiano 37:35 ... despite an extremely different profile of inputs. I think most people believe, like, a combination of distillation and the diffusion of algorithmic innovations means that it is unsurprising that PRC AI developers are extremely close behind-

Dwarkesh Patel 37:46 Yeah

Paul Christiano 37:46 ... US AI developers. So that is one, like, very important feature of this, like, overall plan of, like, racing. Um, I think a second dynamic, which is not, at least publicly, a major part of what is happening now, but becomes really important in these high-risk scenarios, is that you could literally get access to exactly the procedure for training AIs, uh, or ex- the AIs themselves through a cybersecurity compromise. So again, both Anthropic and OpenAI in the very recent past have had extremely unsophisticated actors who appeared to have enough access to essentially extract, uh, the vast majority of algorithmic developments. Like, they got access to Slack and could see where things were discussed, or they got access to the monorepo where most OpenAI code is, um, shared and iterated on.

Dwarkesh Patel 38:28 And by the way, so you're referring to a couple of incidents like this. So I think Hacktron, within a couple of days of trying, gained access to not only the ability to see OpenAI's internal source code, but the ability to make pull requests to OpenAI's monorepo. And then I think recently it was also announced that Cathedral, which is another just random cybersecurity startup, within twelve hours was able to get access to all of Anthropic's Slack. So, w-where, of course, these algorithmic secrets are discussed. And people often debate, are we eight months ahead of China or twelve months ahead of China or six months ahead of China? In reality, we're twelve hours ahead of China, right? Because if these secrets leak, then they are right at the frontier that the American companies are at. So to the extent that the, this information is not protected, then the, the lead is sort of not really useful in the first place.

Paul Christiano 39:20 Yeah. And I think that the incidents are some indication of that. Those are sort of high-profile, like, demonstrations of this capability. But I think even absent those, you have a combination of state actors are very sophisticated relative to ty-typical technology companies, and then the security posture of rapidly growing startups is not really at the same level. And at the same time, we are currently in a very challenging moment for cybersecurity, which is very hard for defenders in general, given the number of vulnerabilities that are being discovered across the entire, like, entire software stack.

Dwarkesh Patel 39:51 So what, what, what should the frontier labs do then?

Paul Christiano 39:54 Uh, so I think we are in-- Again, this is, really becomes a problem if you get to systems, if you have rapidly increasing capabilities and you produce AI systems which you cannot adequately control and you would like to or you do slow further development or deployment. Um, if at that point the stakes are very high, the systems are very powerful, you really do not want it to be possible for an adversary to have access to either the models or the algorithms. I think my current view is that this dynamic seems bad enough that right now, from a national security perspective, it is not good for US AI labs to be improving pre-training and reinforcement learning given their current security posture.

Dwarkesh Patel 40:29 But do you think the frontier labs should pause?

Paul Christiano 40:31 Uh, I think we should-- Pause can mean a lot of things, right? So I think there's, like, a lot of good things that frontier labs are doing, that they should be doing. Um, but I do think we, like, improving algorithms in a way that can be very, very loosely diffused even normally and could be quickly stolen, uh, s- yeah, I think they should stop that right now.

Dwarkesh Patel 40:51 But you're on OpenAI's board. And so basically-

Paul Christiano 40:53 I'm not speaking here-

Dwarkesh Patel 40:54 Sure

Paul Christiano 40:54 ... it's very important in this conversation that I'm not speaking as an OpenAI board member.

Dwarkesh Patel 40:57 But you, you are on OpenAI's board. So you-

Paul Christiano 40:58 That is a true fact about the world. [laughs]

Dwarkesh Patel 41:00 And so is it your view that, like, what is the bottleneck to that happening? If the, if, like, somebody on OpenAI's board thinks this, this is the case, maybe not the only person on OpenAI's board who thinks this is the case. Is it basically, like, Anthropic needs to get, get together and agree to this?

Paul Christiano 41:14 I think the first thing is any decision of this kind is a very dramatic decision. It is a big, for better or worse, it has extremely significant national security implications, how developers approach, again, this further R&D, this improvement of pre-training and RL. Um, so I think it is going to be a big discussion, or it should be a big discussion that should extend beyond, like, people within AI companies talking about this. I don't think that discussion has wound its way to, like, a robust answer, right? I think it's not the case that you can come and say, "Look, we've had this discussion. There is consensus about what the right answer is." There may never be consensus, but I do think, like, probably the first bottleneck is, like, we do need to have the discussion, I think, extremely urgently.

Dwarkesh Patel 41:55 Yeah. Okay, given this fact about security and generally around alignment, what do you think the frontier AI lab should be doing right now?

Paul Christiano 42:07 Yeah. So I think the most important development happening right now is further improvements in the efficiency of algorithms for pre-training, reinforcement learning, improved architectures and so on. I think while you're in the regime where there's a significant risk of this very rapid capabilities progress, a significant chance that you'll be unable to control the resulting AIs, and a significant risk that if you slow down, those developments would leak to an adversary who might use them more carelessly. I think in that regime, it is on net bad for American national security to be accumulating more of those algorithmic innovations. So I think it would be better right now in the current environment for AI developers to pause, like, further algorithmic progress on those problems, uh, until both their security posture was adequate, until they were able to defend those in the event of being unable to continue racing at maximum velocity, uh, and until they were able to tell whether they had adequate control of the systems they built.

Dwarkesh Patel 43:00 And how long would it-- So, j-just to translate, basically, you think we should ma-stop making further capabilities progress, like OpenAI and Anthropic should stop making further capabilities progress on pre-training and RL and stuff, which is, I guess, AI development, until these security problems are solved, until the alignment problems are solved. But are we close to solving those problems? How, if you had to guess in calendar time, how far away are we from solving these problems?

Paul Christiano 43:26 Yeah. So on the security front, I think it's a little bit hard to say because we are in this somewhat unprecedented situation, both in terms of, you know, just mentioned, like the cyber threats that are faced, but also the kind of defensive mechanisms that are available, right? So I think defending, defending adequately would require interventions that normally would take a very long time. I believe we are now in a regime where a lot of projects, like rewriting large amounts of software or drastically changing the way companies operate, have become much more feasible owing to the availability of powerful AI systems. And so I think it is hard to say exactly how long it would take. I think you're like probably at least on the order of months, quite possibly more like six months or twelve months. Um, that's on the security side. On the alignment side, I think it's much less clear that you need us-- like, you may not have a sort of indefinitely scalable solution to alignment. I think what you mostly need is the ability to tell whether things are potentially going off the rails and to respond to that appropriately, which is a combination of, are you able to assess what is happening and whether your control is adequate, and then also some institutional or governance questions and how would you respond to that.

Dwarkesh Patel 44:33 A-and just to reiterate for the audience. So the, the worry is, if these random cybersecurity startups can hack and get arbitrary access to algorithmic secrets at OpenAI and Anthropic within days of starting their attempt, certainly the intelligence agencies of foreign adversaries could do the same if they wanted to. And maybe they have already done that, but at some point, they will actually make use of this information. And if that happens, and you have gotten to the point where you are close to AI systems, you know, GPT-11 could be catastrophic if misaligned, and you, you're at GPT-9 and you say, "Look, I'm pretty scared about what's gonna happen next." This is sometime in twenty twenty-seven, early twenty twenty-seven, May, say. And you say, "We have to wait for a w- bit now and make sure that we are making sure that GPT-11 is aligned." But if they have the current security posture, China could trivially then get access to the weights or the algorithmic secrets relevant to training GPT-9, and now you can't wait. Now you just have to go ahead and build the GPT-11 thing, which could take over if it wanted to because-- or at least cause catastrophic harms if it wanted to, um, because you're worried that China is gonna use GPT-9 to make GPT-10 to make GPT-11. And so-- And you're, you're sure that if, you know, your GPT-11 may try to, uh, overthrow you, but China's GPT-11 will definitely try to at least attack you. And that's a really tough position to be in if your security posture is messed up, basically.

Paul Christiano 46:02 Yeah, that is the situation. So we've talked about these, like, broader takeover scenarios and how you end up in this really bad situation, and I think this is, like, the most likely path to end up there, is you have-- we are having a discussion about what does the current evidence say, and it looks very alarming. But on the other side of that discussion, we are weighing the risk that, you know, an adversary could steal algorithmic secrets or models and go ahead.

Dwarkesh Patel 46:22 So, uh, i-it's interesting because a lot of this discussion about pacing comes down to whether we are okay with China catching up. And ironically, the reason you want to pace is so that China doesn't catch up, except you're targeting this period which you view as much more critical to overall safety, which is the period during the intelligence explosion. And you think slowing down now and even potentially allowing China to start catching up now is useful to the broader goal of being in a position where an AI is decisive of h-how, not, not having China right on our heels basically from stealing the algorithmic secrets.

Paul Christiano 46:58 Yeah, I think that is a fair summary. I mean, I would say that, like, if you are not-- The reason that AI development now is plausibly acceptable is that we can see how things go iteratively and then respond if things look bad.

Dwarkesh Patel 47:11 Yeah.

Paul Christiano 47:11 Right? That is the whole thesis, which I think is a very reasonable thesis. But it absolutely requires, like, that plan involves you at some point in the future potentially noticing that things are bad. And so you just need-- Security, I think, is the biggest hurdle, like the biggest thing you would need in order to, like, actually be able to slow down without massive-

Dwarkesh Patel 47:27 Yeah

Paul Christiano 47:27 ... costs. Um, I think there's other infrastructure and, like, if it wasn't for security as the highest nail, we might be talking about other things and what's our level of preparedness on the scientific side or the governance side, right? We are not in a regime where there is a meaningful, like, scientific discussion about how you assess whether your AI systems are aligned. I guess we mentioned this earlier of, like, what's the state of public information in general. I think it's, like, quite bad, right? I think, as independent scientists really have no idea, uh, whether the systems being produced are aligned or how you would tell. And I think if you wanted a governance regime that would actually make it feasible for the entire industry to react to worrying signs by slowing down in a way that didn't have huge problems, right, where you have individual developers making this decision on separate standards internally, um, you would need to have more of a robust, like, shared sense of what is the bar on safety or, like, actually have the scientific discussion where you understand how do we interpret this evidence.

Dwarkesh Patel 48:16 And h-h-how, yeah, how do you get there?

Paul Christiano 48:19 Uh, I guess, yeah, we have these two things. You need to get their own security, which we've been talking about, and then on, on stuff like how do you tell, I think the biggest path is disclose information, have a broader scientific discussion, both between different AI developers and in the broader scientific community. Like, what is the evidence we're getting? Why do we think that evidence is comforting? How are we going to tell when that changes? Like, what will we see before something bad happens?

Dwarkesh Patel 48:41 What information do you need from the AI companies?

Paul Christiano 48:44 I think the most salient thing that is not present in public discussion is information about how systems are trained. Um, there's also some information missing about how systems are evaluated, but the really big gap on this front is it is extraordinarily hard to interpret evaluation results without having some understanding of how systems are trained, and that's mostly not present publicly.

Dwarkesh Patel 49:00 Okay, so if you think this, and, um, you're on the board of OpenAI- I know everything you've said is, is your, your own opinions as a private individual, but you are also on the board of OpenAI. Presumably, maybe, like, you, maybe other members of the OpenAI board feel this way, and maybe people at Anthropic feel this way, and you guys are the leading labs. Wh-what is standing in the way of-- I mean, this is actually a pretty drastic thing to, like, literally pause capabilities progress now. That'd be an incredibly drastic thing, 'cause n- pacing is not bad, right? People are not talking about this. Or at least people short of Bernie Sanders are not, and Eliezer Yudkowsky are not talking about this. Um, what is actually needed, like-

Paul Christiano 49:38 Yeah, so I do-

Dwarkesh Patel 49:39 What is needed to do-- Is this just basically a coordination problem? Like, if everybody co- if, if everybody had the choice, would they all agree to this?

Paul Christiano 49:44 I think, like, first of all, I really do wanna emphasize that I am speaking for me as a person.

Dwarkesh Patel 49:48 Yeah.

Paul Christiano 49:48 And, like, as an OpenAI board member, I will bring my perspective to that discussion and, but I'm not at all reporting on the views of the board in any way. Um, so I think probably the single, you know, you can imagine there's two blockers. One is this is a hard question, right? I think my view is, on balance, it is better for national security and public safety for AI companies to currently improve their security posture before making additional algorithmic progress. Um, that is not a super widely held-- It's not like everyone is sitting there thinking that. I think we're currently at a place where I think for most people, they'd say that's a really complicated, messy question, and we haven't thought through it enough. Or, like, this discussion has not run its course, and not everyone, like, has an opinion on this or has the same opinion on this. So I think that is the most important thing, that we just need to-- It's a messy conversation. We need to have it. Uh, we need to talk about the costs, the benefits. Again, my view is it looks pretty bad to me right now, like it looks like we're sort of underwater from a, like, public benefit perspective. But I think other people have different views. And the second thing is, I do think an individual AI company is in a very rough spot if they have this view-

Dwarkesh Patel 50:50 Yeah

Paul Christiano 50:50 ...where they say, like: Look, we can defend our algorithmic secrets, but all that is going to happen if we, like, slow progress while we improve security, is that algorithmic secrets will leak from a competitor, we will lose market share, and the net situation will not be any better. And so I do think there's both. I think the most important part is understanding, is this actually a good idea? And, like, there's a bunch of different perspectives. People have different views, different values, whatever it is. So it's like, is it a good idea? And then there's the further problem of, like, how do you actually, like, implement this in any realistic way if individual developers, again, have this concern of, like, "If I don't do it, someone else will," which is a fairly major, I mean, on many of these issues we're talking about, um, I think is a major dynamic. And that-

Dwarkesh Patel 51:27 But there's two companies, right?

Paul Christiano 51:28 ...kind of takes legwork. I- There's a lot of companies in, there's a lot of companies-

Dwarkesh Patel 51:31 But at the frontier, like, if OpenAI and Anthropic agreed.

Paul Christiano 51:34 I think it is the case that if everyone at OpenAI and Anthropic agreed that this was good or it was, like, a clear prevailing sentiment, it is reasonably likely that it would happen. Um, yeah, I think you should expect if everything is working reasonably, that should happen. Again, not saying, not commenting on where anyone else is right now. Um, I just think as a, as a reasonable observer. AI agents are doing more and more work without human oversight. They can already write code faster than any person can actually review it. But you can't just give code review to another set of agents. There's no guarantee they'll magically fix problems introduced by the first set. If one LLM can't model a specific class of failures, then it's unlikely that a second or third or fourth LLM will either, because even different LLMs tend to have similar failure modes. My sponsor, Antithesis, offers a different approach. You, the human, define the rules your system must never break. For example, a customer should never be double-charged for the same transaction. Then Antithesis acts as an external oracle, testing whether your code adheres to these rules. To be clear, Antithesis isn't an LLM that tries to reason abstractly about whether your system could possibly break its own rules. It actually runs your code, stress-testing it in thousands upon thousands of adversarial worlds. Just like we fly planes for many, many hours and put them through all sorts of intensive maneuvers before we let any passengers on board, Antithesis lets you gather a ton of empirical data about your code before you release it to production. And it's easy to get started. Antithesis just launched a self-serve plan. To sign up, go to antithesis.com/dwarkesh and use code DWARKESH for fifty percent off your first month.

53:15 – 59:31

The case for pausing now instead of later

Dwarkesh Patel 53:15 Right now, if Anthropic and OpenAI paused and got their security together for the next six months. In the meantime, Chinese models continue getting better. What is the reason to think that once OpenAI and Anthropic go full steam ahead again, that, uh, China wouldn't be-- If China gets to a similar level as OpenAI and Anthropic, why wouldn't they be doing RSI at a similar pace?

Paul Christiano 53:40 Yeah. So I think there's two different models, maybe two extreme models you could have of US-China competition. One extreme model is the US and China are kind of proceeding along independent technical tracks, and it happens to be the case that they have proceeded at almost exactly the same speed.

Dwarkesh Patel 53:53 Mm-hmm.

Paul Christiano 53:53 Um, and we are, whatever it is, six months ahead. And, and on that model, yeah, if you wait six months, you will now be exactly even, and from there on, you will be exactly even. There's another extreme. I think that model is, like, that would be a very surprising situation for a bunch of reasons we can talk about. Um, at the other extreme is a model that is actually large parts of Chinese progress are catch-up growth. Because large parts are catch-up growth, um, if the US is one month further ahead, and by catch-up growth I mean things like distillation from US models and diffusion of algorithmic developments from US companies. If you're in the catch-up growth regime, then it is the case that each additional month of US lead increases the speed of PRC catch-up, right? The further ahead you are, the easier it is, or the more you be-benefit from distillation and the more you benefit from diffusion of algorithmic ideas.

Dwarkesh Patel 54:36 Right. But I guess at the point at which they are at the same level-

Paul Christiano 54:38 Yep

Dwarkesh Patel 54:39 ...they should be able to use those models to do RSI themselves.

Paul Christiano 54:41 Yeah. I mean, the two things that are very relevant. At the current, this becomes worse and worse over time. Um, but at the current scale, it is both the case that human capital is extremely relevant, and AI labs have a huge amount of human capital, um, and I think can make faster progress, and compute is extremely relevant. This will remain true going forward, and US labs have a huge amount of compute. So generically, if not for catch-up growth, I think we would have a pretty strong expectation of US labs moving faster.

Dwarkesh Patel 55:03 The other argument is, d- shouldn't we-- If the cyber models are getting better and better, shouldn't we wait for models which can-

Paul Christiano 55:11 Instead of having to wait six months to get our security posture up and give Si- China six months to catch up, that maybe we only have to wait two months. You know, if cyber models get two months better from now.

Dwarkesh Patel 55:21 Yeah. There's this analogy one can draw to sending a probe out to Alpha Centauri, and whether we should do that with today's technology, in which case it would arrive, you know, thousands of years later, or we should do it with technology a hundred years from now, in which case it would be lapped on the way to Alpha Centauri. And we could imagine a similar thing is true for securing the hardware at, or securing the algorithmic secrets and weights at OpenAI and Anthropic, where they, if they started the pacing or started the pause two months from now or three months from now, the speed at which they would be able to secure the systems is faster than if they started right now.

Paul Christiano 55:53 Yeah. I mean, I think this is a really important question and one of the bigger questions. Um, so the downside on the other side, to make it clear, is if you had diffusion of algorithms right now, it is much worse than diffusion of algorithms six months ago, but much less bad than diffusion of algorithms in a couple months. Like, right now, you are both-- You require more compute and you require more human capital to proceed with AI development. And the more time passes, the more disastrous it is to have, like, an adversary actually willing to just grab everything and try and compete. Um, so the, the reason to do things earlier is that if you make, have a pause earlier, both there is more time for differences to wash out and you are more loaded on your, the factors where you currently have an advantage.

Dwarkesh Patel 56:36 What do you mean more loaded on the-

Paul Christiano 56:37 Like, compute matters more, and people matter more right now than they will in six months. You might imagine, if we're in the world where we have rapid progress over the next year, six months from now, it is likely the case that it will essentially not matter how many people you have. It will only matter how much compute you have.

Dwarkesh Patel 56:50 Yeah.

Paul Christiano 56:50 And so at that point, there is less of, like, accumulated institutional know-how, or it sort of just becomes this raw contest of, of numbers. And it's very easy for any delay you're willing to take, either for security or for safety, to be decisive in that kind of contest. And it's just like the, the concern is, as you move forward, it gets more and more likely that's the world you're in. I think some people would say this is, like, kind of already basically the world we're in, that we have now come too far. This is, like, another argument that has been made, which is we're already in the just go all out and race because it is already the case the leakage is too bad and you will be overtaken if you slow down for safety.

Dwarkesh Patel 57:21 Yeah.

Paul Christiano 57:22 I think that is probably not the case, but to the extent that you're seriously entertaining, you have already gone too far. I think that is, illustrates why you might not be like, "I'm just gonna wait another two months and harden things." Maybe the last thing to say is, like, it is a little bit unclear what the dynamics are in terms of how quickly you can harden things, right? So some of hardening things is writing new software or whatever it is, or changing the way you operate. Some of it is changes that humans make, or changes to your, like, overall arrangement of compute or whatever, which are not as easy to make really rapidly, um, or maybe not as accelerated by AI. So you will have at least some part of this lag which is not tied to the capability of your AI systems. Um, but it is messy. Again, I, I think, I do think the biggest bottleneck is, like, can people agree? Like, can we have out this debate and, like, actually understand what the net is?

Dwarkesh Patel 58:02 And if we are th-this close to superintelligence, or at, at least there's a good chance we are, and there's a good chance that superintelligence could be in a position to overpower governments and kill lots of humans, what should the AI labs be doing right now? If you had to boil it down in one answer, what would you say?

Paul Christiano 58:22 I think there needs to be more public visibility, both into the capability trajectory, right? We should not be relying on, like, METR and Anthropic saying, "It looks to us like we're gonna be able to overthrow governments in a year." In fact, we have a vested interest in really seeing the underlying data for that. To have visibility on alignment, right? To understand how do we tell that existing systems are adequately aligned or controlled? And to, like, actually start having a public discussion about, like, what is the bar for good enough evidence that things are working. And then, you know, beyond just information, at some point, you do need to say, "Here is what we need to get done before we continue making progress, either before we scale up our models further or before we continue producing algorithmic developments that may leak." Um, and right, the first-- Our inputs into the second, we need to understand where are capabilities at and where is alignment at in order to understand, is it a time where we need to improve our security or safety situation before we proceed? Uh, but I do think, given the current state of uncertainty, you know, if I was a person in the public without access to further information, my strong opinion would be it looks like we are already at a point where we need to be improving safety and security before moving further. And I think, again, the most important form of moving further right now are algorithmic developments that improve the efficiency of, of pre-training and RL.

59:31 – 1:11:32

Superintelligence will be good at learning from real-world experience

Dwarkesh Patel 59:31 Okay, stepping back, you've got the superintelligences in a data center. I'm trying to lay out an intuition for, like, here are the kinds of capabilities that superintelligence would have. What should we imagine those intelligences capable of doing?

Paul Christiano 59:43 So just to start with a high-level sketch of my view, uh, maybe there's two kind of lenses you could use. One is look at current progress and just see where does it lead if you extrapolate that progress, like, a responsible distance. And a second is to sort of think about what we know about the limits of the technology and where we would expect those limits to be. Um, on the first front, I mean, some things I think are relatively clear. So if you imagine AI systems doing cyber operations today, we have seen an incredible rate of progress. We are currently seeing pretty impressive capabilities in terms of AI systems not only being able to discover vulnerabilities, but develop exploits and increasingly chain them together into operations. I think what you will see for superintelligent systems is both of those capabilities scaled up much, much further. So systems which are able to carry out, or you might imagine, very cheaply carry out cyber campaigns more sophisticated than anything a state is capable of today. I think you will see those capabilities being combined with a much broader set of abilities to, say, engage effectively with humans, think strategically, uh, manipulate humans, manipulate organizations, play politics effectively. I think you'll see those capabilities at a level well beyond any human. I think you should think of this as, like, very, very large populations of AI systems thinking quite quickly. So maybe a very simple model would be, like, a population of billions who are thinking ten times faster than humans, who are much better than humans at essentially every skill. Um, I think this is all what you would expect from an extrapolation of existing technologies and seeing just, like, very good capabilities in domains like, again, like cyber or manipulation or research, um, like doing scientific research, hypothesis generation, experiment design. Um- Um, I could expect to see extremely strong capabilities in robotics, in, like, logistics and coordination and control. Um, I think you would expect to see very-- the possibility for very, very rapid technological progress, including in domains that involve physical experimentation, just in terms of being able to orchestrate experiments quickly, rapidly build out apparatuses to increase throughput of experimental results. Um, so that's like one bucket of just things that I think you can kind of see by drawing dotted lines on graphs and talking about where they go. Maybe stepping back a bit in terms of what the technology looks like more fundamentally, I think, right, humans are very good at this task of you throw a human in a new domain, the human takes in some information, the human learns about that domain and becomes able to act effectively. And that's kind of the reason we're effective in the modern world is primarily that ability to learn about a new domain and adapt to it, right? Humans do not have a ton of evolutionary context about how to be a good chemist or a good biologist. That's something we pick up over the course of, you know, a couple years or decades on the job. And I think in some sense, the most fundamental skill that AI systems will possess at the end of this process is that kind of rapid learning, adaptation, planning, reasoning, like these very domain general skills, such that even if you drop them in a world that was completely unlike anything that they had experienced in a data center, they would adapt to it very quickly in the same way that humans adapt very quickly to these R&D domains that are quite different from anything we encountered in our evolutionary environment. Um, so that I think would result in systems which collecting maybe even just a couple of days of experience in a very parallel way would end up with real-world capabilities in essentially any domain, much beyond the abilities of the best human experts, right? They will have learned from much, much more data with learning algorithms that I think at that point will be much better than those that evolution has given humans. Um, so that's sort of a more abstract picture of where this sort of ends, um, beyond just drawing dotted lines.

Dwarkesh Patel 1:03:08 Hmm. Can, maybe I'll try to just, uh, say that basically again, because I-- this ends up being a significant crux in these conversations where I'll be talking to somebody about the possibilities of sup-superintelligence, and they'll say, "Okay, you'll have a thing which is superhuman at coding, but how will it learn to be good at politics? Or how will it learn to run a new automated firm and then compete with human firms or something?" And so i-in those conversations, you realize there's a big disconnect between the way you are talking about superintelligence and the way most people understand superintelligence, which is just narrow AIs that are specialized in some domain, like, and have just amalgamated a lot of very specific experience or expertise in, say, coding or mathematics. What you're imagining instead is something like human-like learning. So the top engineers at SpaceX were not born knowing how to make a rocket that can get hundreds of tons of cargo to orbit, right? So this is, like, an expertise they gained working on the problem during their lifetime. And you are saying this, like, meta learning capability is something that superintelligence will also have. I mean, it would at least have a learning capability as good as humans. You're suggesting it could be much better. But furthermore, even if it was only as good as humans, but they could learn from, right now, there's maybe tens to hundreds of millions of instances of ChatGPT or Claude, which are actively doing real work in the economy, right? They're, like, helping somebody write code or helping some senator write a, a political brief or something. Um, you're, you're imagining a world where we have the learning algorithms basically such that AIs can learn from all this experience at once. So it's not like, right now, the base model is not even learning from the experience of a single deployed instance. You're managing a world where you have a hundred million instances of ChatGPT are going and doing something. The base model is learning across basically a hundred thousand different careers at once, or a hundred thousand different work sessions at once, and doing it much better and faster than humans can learn from a single instance. And you're also imagining a world where the population of AIs, um, is, you know-- I, I guess, do you have an estimate how, like how, how big a population, how fast should we be-

Paul Christiano 1:05:18 I should think about it more, more carefully. But if you are-- Maybe a really important part for all of these discussions before talking about population size is it really matters if we are imagining, like really driving this process extremely fast. I think people can really easily be in the headspace of, like, I'm extrapolating another year of progress at the current rate.

Dwarkesh Patel 1:05:34 Yeah.

Paul Christiano 1:05:34 It's really worth emphasizing. We are talking about systems which get not only extremely good at coding, but extremely good at every, like, well-posed virtual task which machine learning engineers are able to make in the lab. So I think if you push that to its limit and you say, how large are those populations, I think it's, like, very conservative to say a population of billions of agents running at ten times human speed. I think we can debate exactly what are the limits, but they, I think, would be significantly larger than that.

Dwarkesh Patel 1:05:56 Just because of software efficiency gains?

Paul Christiano 1:05:58 That's, that's right.

Dwarkesh Patel 1:05:59 Yeah.

Paul Christiano 1:05:59 So just, like, how efficient can you make-- And it's, right, you're not gonna talk about, like, a single type of agent. They're all the same, a billion of them running.

Dwarkesh Patel 1:06:04 Yes.

Paul Christiano 1:06:04 We're gonna talk about these very heterogeneous programs, many of which are, like, very specialized, right? One thing you do if you do a huge amounts of R&D is say, "I would like to produce just the cheapest thing I can," which is expert enough in a domain to act effectively. Um, so you shouldn't, you know, again, it's a reasonable intuition pump to say it will be more capable than billions of people thinking ten times as fast. But in fact, it's really, like, much, much larger populations of specialized systems, maybe billions of very general purpose systems which are as good as humans at anything. Um, yeah. Yeah, I do think it just goes, if you imagine, like, really pushing this process and having faster and faster progress on software, and if someone grants, like, actually, we're gonna see that process saturate, like everything you can do in the data center is gonna become really great. I think that endpoint is really extreme. I think the gap between doing really extreme stuff in the data center and doing really extreme stuff in the world is, is very small or smaller than people appreciate.

Dwarkesh Patel 1:06:51 Okay. So I think there's a discussion to be had about whether such systems are even possible. I fi-- I am very compelled by the argument that just something which can learn at least as well as a human can. So there's some learning algorithm that human brains have, something which can learn at least as well as a human can, but which has huge population sizes and which can learn from the deployed experience of hundreds of millions of different instances at once. That alone would be pretty transformative. And then I agree with you that there's no clear reason to think that you can't get even more learning efficient than humans.

Paul Christiano 1:07:29 Yeah, I think it's very good. Yeah, I'm very happy to talk about this. I don't know if it's, like, gonna end up being the main objection people have, but I think there's very good evidence that you should expect systems in, in the end, once you've, like, really driven the technology to its limit, to be much, much better than humans in these ways. I think we-- It's not just a case of, like, we have humans, so you can do at least that well. I think we have good evidence it's, it's much better than us.

Dwarkesh Patel 1:07:46 What's that?

Paul Christiano 1:07:46 And so I think relevant things are, one, just the rate of progress on evolutionary time. If you look back over evolution, it's fairly clear that we're seeing extremely rapid improvements in learning capability and intelligence and planning capability, right, as you move into humans. And the thing that happened was just humans got good enough that cultural change was faster than those biological improvements. So we kind of know if you just, like, ballpark evolution, it was clearly, if we didn't form civilization, going to make much smarter things.

Dwarkesh Patel 1:08:09 Yeah.

Paul Christiano 1:08:09 Then you have a variety of, as you say, like, there's the data question of humans learn from one lifetime of data. AI systems will learn-- I mean, already learn from radically more data than humans, and that gives them, like, even if you were no better as an algorithm designer than evolution, you would produce much, much better minds if you can relax that constraint. Then you have things like power constraints and coordination constraints, which we kind of have a pretty good sense are a really big deal for evolution. Um, and we will be able to push those out much further. Um, it's sort of like, you know, if you wanna produce electricity or something, the ability to make a nuclear reactor, you don't have to be smarter than evolution. You're just operating in a different regime from evolution. You're able to do things that wouldn't have been a good idea for evolution. So I'm very happy to dig in more on this, but I do think-- And it's probably not for most people gonna be the most compelling argument. It's more compelling to, like, look at the status quo and say, "Well, we've had ten years of progress. You only have to extrapolate another ten years to get to the crazy superintelligence."

Dwarkesh Patel 1:08:51 Yeah.

Paul Christiano 1:08:52 But I do think we have a pretty good indicator that it does not top out anywhere near humans.

Dwarkesh Patel 1:08:55 Yeah. And, uh, sorry, just to say that your last point, uh, again, what you're saying is, look, humans developed on the savanna under this tight constraint, where really evolution was trying to get something that is good at propagating its genes. One of the ways that a chimpanzee can get better at propagating its genes, or ancestors can get better at that, is by being, having these, like, general planning and reasoning capabilities. Um, but they're not worth so much that you are willing to, say, give up digestive functions or immune functions and surrender more of your total hundred-watt budget to the brain. And so you have this extremely tight power constraint for the brain, extremely tight size constraint for the brain as a result. Not only of that, but other evolutionary, uh, reasons as well, like how can your head literally fit through the birth canal? Um, and that will not be true of AI systems, where we're not saying that hardware will be more power efficient than the human brain, but simply that it's not as big a constraint for us to just have enormous amounts of hardware that is consuming a lot of power in the way it was for evolution when it was designing the human brain.

Paul Christiano 1:09:53 Yeah, and this cuts in both directions, right? It's also the case that you'll have extremely power-efficient systems, right? You can't say if you're a human, you can't be like, "Well, I'm gonna make, like, ten thousand simple agents, each of which uses only one percent of a brain, or point zero one percent of a brain-

Dwarkesh Patel 1:10:04 Mm-hmm

Paul Christiano 1:10:04 ... and is better than a human at doing cyber operations."

Dwarkesh Patel 1:10:06 Yeah.

Paul Christiano 1:10:06 Right? So there's this flexibility about form factor that is going to be a huge-- Like, if you just did, if you were no smarter than evolution, just did what evolution did, you would end up with these, yeah, you would end up with agent forms that are completely beyond teams of humans, like really radically beyond them.

Dwarkesh Patel 1:10:21 Let's say that an agent is vibe coding a web app, and it needs a database. There's one database provider that needs a human to go through the whole sign-up flow, verifying their account, adding billing details, and then generating an API key to hand back to the agent. And then there's another provider that lets the agent make an account all on its own. The agent will probably choose the second one. In general, agents tend to pick the tools that they can use right away versus the ones that they had to wait around on. So as agents do more and more work, giving them their own way to sign up becomes an important way to win new users. My sponsor, WorkOS, makes this easy. If you use WorkOS for login, you can give agents their own sign-up flow just by toggling on agent registration. You can then decide what agents should be able to do on their own and what, if anything, should wait until the person they're working for gets involved. Going back to the database example, maybe you want to set things up so that agents can create databases all on their own, but they can't share them with anybody or upgrade to a paid plan without a human. You can do all of this with permissions that you already set up in WorkOS, instead of having to build a separate system for agents. To learn more, go to workos.com/authkit.

1:11:32 – 1:22:23

Crossing the sim-to-real gap

Dwarkesh Patel 1:11:32 There's a bunch of objections people have to the idea that automating AI research will lead to crazy fast progress along these more general cluster of capabilities. You know, you talk about experiments and data and everything else, but really, I think what it comes down to in the end of the day, if you just dig in deep on any of these, is the sim-to-real gap. Basically, is it possible that you could have a process which occurs entirely within a data center, which results in a- minds which are not, not only as capable as humans at learning in a new domain, especially a domain that is, like, a real-world thing, like making a new company or, you know, uh, m- negotiating a deal with, uh, China or something like that. Um, not only as good as humans, but even better than humans. So how do we get AIs that are capable in this more general way you're suggesting from the within data center training that will be happening over the next year?

Paul Christiano 1:12:28 I do wanna start by reiterating that I think if you take progress in the real world and granting that it is slower on messy real-world tasks than it has been in math or coding, it's still the case that if you take that progress and just extrapolate it forward, I do think it gets us to systems which are powerful enough that they would be able to out-compete humans. Um, and we can talk about what people are doing to drive that and, like, how you can scale it into the future, right? So I think you are looking at things like, one, you can train on a bunch of related well-posed tasks, right? So these systems we're talking about will have been trained on a huge amount of, like, predict how humans will respond to something. So we talk about these AIs coming out of the data center. They're not like AIs who have never seen the world. They are AIs who will be radically, radically superhuman at predicting things like, given what just happened with that person's face, how should you update what they're going to do tomorrow, right? Or given what this company has done over the last three weeks, what do you most expect this company to do in the next week? Um, or given observations of this laboratory environment, what is the best prediction for the results of this experiment, right? So those things are in. When we talk about, like, sim-to-real, we should be clear that we actually are including a bunch of these tasks which people train on now. And that even includes things like given the data so far from this laboratory, do as many like computational experiments as you can, do as much hypothesis generation and exploration as you can in order to come up with a hypothesis to make a good prediction about what you'll see next. So these systems will also be very superhuman in many parts of like science at reasoning about the real world, at planning and like hypothesis generation and so on about the real world. So that's part of the story for how it is even the kind of thing we're doing now can produce very powerful systems. A second part of the story is that you can train, right? One regime is you can train on exactly the thing you care about. You say, "I care about writing code that passes this type of unit test, so I'm just gonna train systems that are incredibly good at that." You can also do a second thing, which is a large part of what we observe in performance, which is train on a broad distribution of tasks, either kind of in a general domain or across several domains, and then deploy those systems, right? Acquire general skills that are relevant to that domain, which transfer to new tasks. So I can say, "I'm not gonna try and train a system which is good at the exact problem, right, the exact problem of going and talking to congresspeople in order to influence them." But I can train a system in an extremely broad range of different tasks where there will be a lot of common skills, including these general skills like how do I adapt on the fly? How do I learn? How do I plan? Um, how do I set up reasonable experiments that I can learn from and tell how well I'm doing? So if I create like very broad distributions of tasks, some of which could be coding tasks, but some of which can be tasks like you're dropped into a new, very messy virtual environment and need to try and navigate that environment. Or you're given a new type of task, and you're given twenty examples of it, and you need to like learn how to do a really good job of doing the next example quickly. Um, or I have to interact with other AI systems, or I have to interact with humans in environments where I can. In the data center, there is a fair amount of human data that is available. Right, so if you just go through numbers and say, how much interaction with humans can you actually get? It's not like these systems have never seen humans. They will have seen vastly more human interaction even than any human would. So they can also deal with a bunch of reinforcement learning environments that involve other humans or involve collaboration with humans.

Dwarkesh Patel 1:15:18 Right, because people have amortized-- like people have written in like what, how a human would respond to something? Or how, how is that human data in there?

Paul Christiano 1:15:24 Yeah, I mean, you can also just literally engage with humans as part of the training process.

Dwarkesh Patel 1:15:27 Okay.

Paul Christiano 1:15:27 So people do spend large amounts of money saying, "Great, I'm gonna have some humans engage with the system."

Dwarkesh Patel 1:15:30 Yeah.

Paul Christiano 1:15:31 Um, and right. There's a question of how efficiently you squeeze information out of that data. How quickly do you learn, right? If you have these very general skills, how quickly do you learn about the particular domains that... Again, I think that the, the dotted line from what we've seen so far, together with like a ton of capability to push that forward and to do a bunch of automation, like the construction of many more environments and so on, um, just looks very likely to lead to systems that can quite quickly rise to at least human level and likely way past human level, including on very messy real-world tasks.

Dwarkesh Patel 1:16:03 And so I-- the, the thing I'm curious about is why you think this happens over the next year then. Like, it seems in principle possible to me that there's enough data available, even within a data center, that if you have a decent amount of transfer, you have a very capable system. Like if there's enough transfer from g- evolution training us in the savanna to now a human being able to make a semiconductor, then maybe there's enough transfer from what's happening into the semi-- inside the data center to let's go to Congress and try to persuade them of something. But, um, y- the things you're mentioning, like this endless back testing or like this RL basically on fuzzy tasks, it seems like things we've already been doing for many years now, and it doesn't seem like we're a year away or like a, a further year of scaling away from this big a jump. And so am I correct in understanding that you're just imagining something like we've already done, but at like a much, just a much bigger qualitative jump on along those same lines?

Paul Christiano 1:17:00 Yeah, much faster. So I think if you asked me, "Hey, Paul, if you're extrapolating the dotted line and looking at the mechanics of what people are doing, how long would it take to produce systems that are, in fact, capable enough at all these tasks that they would be able to reliably outcompete humans?" I would probably say something like, um, "It seems like it might happen in four years. Maybe it could happen faster than that. I feel more confident within like eight years or something."

Dwarkesh Patel 1:17:21 Yeah.

Paul Christiano 1:17:21 So those are the kinds of numbers I would be throwing around if we were just asking, on the current trend, how long will it take doing exactly the things we're currently doing to get these capabilities? Um, and people could-- there's lots of room for debate about those numbers. But the thing that I am most worried about that I think seems like an extremely live possibility, like probably more likely than not at this point, is that instead what will happen is the work that people are doing in machine learning in order to drive all those improvements that we've seen over the last four to eight years will be automated to such an extent that the pace of that progress is accelerated by at least ten X and likely more. Um, we are maybe already seeing acceleration in the ballpark of two X. And I think it is very likely, on the current margin, that the first thing that acceleration does is like continue sustaining itself. Regardless of this broader discussion of how good AI systems are at acting in messy real-world environments, AI systems are very good at acting in virtual environments when they are trying to deliver on verifiable tasks. And so I think the thing that is most robust is the AI systems are improving extremely rapidly and are starting to cross human level on the kinds of tasks that are involved in making your AI systems better at learning in environments you can simulate in the data center. And that includes, I think slightly more broadly, the kinds of software engineering tasks that are involved in creating new environments that stress-test hard skills, and even many of the software engineering skills that are involved in orchestrating data collection from the real world. Like, we could talk about what are the data bottlenecks, like what are the bottlenecks in that process which cannot be accelerated by automation? But it looks like if you look across the core things humans are doing to drive the last four to eight years of progress, those are things which we are on an extremely good pace to automate, and we are fairly likely to have the self-sustaining acceleration, where we see, you know, in eight months, you double the pace, and in the next four months, you double the pace again, and in two months, you double the pace again. And at some point in that process, right, at the beginning of that process, it will be focused primarily on these virtual tasks, right? Because your main priority is making the self-sustaining loop faster and faster. If you're a person who just wants to get to the endpoint as quickly as possible, you do not care how good a lawyer your AI system is for a while. You just care how good it is at driving the development of better architectures and better optimizers and better learning algorithms. Um, that's all you-- or even constructing new environments or orchestrating like this kind of messy, right-- Maybe you care about one slightly messy task, which is constructing environments that help you build the skills that are relevant in the real world. But you don't care about almost any of the real-world tasks where we're seeing the slowest progress. As you're focused on that, and you're going through this process of seeing doublings in eight months and four months and two months and one month, and then as that process goes, it becomes cheaper and cheaper or faster and faster to then develop algorithms and models that, uh, perform very effectively in the real-world domains that ultimately matter. So I think that is, like, really the concern. And the question is, as we said, first, is that process going to occur? Are you going to see the self-sustaining acceleration with faster and faster progress? I think, honestly, to me, that is probably the bigger open question. Um, I think that looks more likely than not right now, but if that occurs, I think things are almost certain to be extremely crazy. There is then a second question about the particular way in which they get crazy, which is that it looks very likely those models will transfer to the real world, to systems that are able to out-compete humans. Um, but that does happen kind of very back-loaded in the year, right? If you're imagining a year where very disruptive things happen, and this is part of why it's so hard from a policy perspective. For the first six months of that year, all you are seeing is better and better performance on th-- like, you look at the systems and you say, "Well, they were previously superhuman at math, and now they're, like, incredibly superhuman at math." But you have, like, no real ability, if you go and talk to someone-

Dwarkesh Patel 1:20:35 Mm.

Paul Christiano 1:20:36 There's no real ability to say, like, "Therefore, they are a threat to humanity." Or, like, AI labs are currently saying things like, "Oh yeah, every month we're doubling the amount of internal use of AI and the, like, degree to which it's contributing to our progress," or, like, "Over the last six months, we've seen a doubling of, like, the net effect, even including diminishing returns or net pace of progress, even including diminishing returns." You can have more statements like that, but if people are not-- You could be in a world where the link from that to, like, very severe harms does not become obvious until quite late in the process.

Dwarkesh Patel 1:21:03 Yeah.

Paul Christiano 1:21:04 And to some extent, I think it is getting increasingly obvious now, right? If you, again, look at cyber capabilities of models in particular, but also their capabilities in some scientific domains, they are approaching the point where there are very severe risks. Um, but that is, right, that is-- We're seeing that in a couple of domains, and so it's tempting to say, "Okay, we have two problems." There's just a few specific domains in which AI systems have dangerous capabilities. Uh, but if this process of acceleration continues, that is not gonna be localized to cyber or to bio, right? That is gonna be a thing that occurs across all of science, across all of social engineering.

Dwarkesh Patel 1:21:33 Yeah.

Paul Christiano 1:21:34 And I think there is the thing where we need to, if you're thinking about the risk, you do need to really understand whether this is possible, whether you can have the self-sustaining acceleration, then leading to very, very rapid progress in dangerous real-world capabilities-

Dwarkesh Patel 1:21:45 Mm.

Paul Christiano 1:21:45 -both in specific domains and in this kind of very general learning faculty, as we've talked about.

Dwarkesh Patel 1:21:48 Yeah. And so th-this goes back to the thing you were saying of you should imagine the systems in the future are less about fine-tuning GPT-6 to do something, but more like GPT-12 maybe just being able to do it out of the box with very little experience.

Paul Christiano 1:22:01 GPT-12 will also be great at fine-tuning GPT-12.

Dwarkesh Patel 1:22:03 Exactly.

Paul Christiano 1:22:03 So if GPT-12 is like, "I find that my chemistry intuition is not very good," GPT-12 will be great at saying, "Okay, I'm gonna direct a thousand humans, and here's the data I think I most need. We're gonna do one day of data collection of that type, and we're gonna run these experiments and do the de-risking." And, you know, within four days, GPT-12 has the new tiny model, which has incredible chemistry intuition, and it's doing this in parallel across all domains.

1:22:23 – 1:27:57

METR says AI progress doubled, so why is revenue growth slowing?

Dwarkesh Patel 1:22:23 Yeah. And okay, so this is a-- Um, I think you just answered a question which I meant to ask you. So it seems like a big thing that has driven your concern over the last few days is this new result from METR, where they find that Anthropic's pace of AI progress on well-scoped tasks has doubled. And so these are tasks as captured by benchmarks like the Epoch Capabilities Index, which amalgamates a bunch of different benchmarks. Now, it could be that there's other evidence that is relevant to METR's assessment that AI progress has doubled, but that evidence has not been made public, and I think that's one of the problems you have with the current setup.

Paul Christiano 1:22:56 Yeah.

Dwarkesh Patel 1:22:56 But if you just consider that evidence on its own. So we have, um, progress on verifiable tasks has doubled, but we have this effect where we're searching for our keys under the lamplight, right? So it is also the verifiable tasks which we're being trained on. Um, and then you can ask the question: Is progress on non-verifiable tasks also, has that also doubled over the last year? And as you may have noticed over the last week, the stock market took a big tumult. Part of that was almost certainly the midterms. But another part of the reason why is, um, Anthropic and OpenAI's revenue numbers, and the growth of those revenue numbers did not keep track with what the market was expecting. So in Q3, I think Anthropic's revenue grew something like, uh, five point four billion dollars. That was less in absolute terms than Anthropic's revenue grew, uh, grew in quarter two, six point nine billion dollars. So not only has the rate of growth slowed down, which people expected, but the absolute growth has also slowed down. So how can you have a s- And then, so that maybe suggests that measuring progress on well-scoped tasks is actually not telling us about the pace of overall AI progress. Because how can you have it be the case that overall progress is two X faster, and yet absolute revenue is not even growing at the same rate it was previously growing? And maybe the explanation is what you just suggested, that really, these AI companies are developing better and better RSI models and only incidentally [chuckles] releasing Claude for Law or Claude for Finance as a, as sort of stopgap to keep this underlying process going. And so this hasn't been a priority, slash they haven't worked that hard on it, slash you should really be concentrating on this sort of inner exponential of the RSI capabilities of the model, and then that will generalize. But how would you interpret this disconnect between the pace of improvement and the end usefulness of these AI systems and the pace of improvement on well-scoped tasks?

Paul Christiano 1:24:46 So I guess zooming out slightly, if we, like, talk about not just, like, the last six weeks, but in general, like, what has happened over the last, you know, eighteen months even, I think we have seen just extraordinarily rapid growth in revenue, right? The basic-- We could talk about, again, what is the story for the last four weeks, but also just we are talking about revenue more than doubling every six months. Um, so that is very, very fast growth for the industry overall. Um, the period in question, so if we ma- wanna match up, like, these internal results on progress to, uh, external results, has corresponded to a period of in- also incredibly fast revenue growth. It's very hard to say, is that revenue growth corresponding to capabilities progress that is twice as fast as previous capabilities progress? I think the revenue numbers are noisy enough that it is really hard to draw that judgment, and the model release, like, the staggering of model releases is noisy enough and, like, there's just a lot of problems in trying to make that inference. Um- At a high level, I think revenue is a reasonable indicator. Revenue progress is extremely rapid. Very short-term trends in revenue, particularly at a single company level, are extremely noisy. So I guess I-- to me, it feels like there's a discourse of, like, periodically, you know, the length of the AI winters is shortening, and now we're like, it's been two months, and AI revenue over the last two months has, like, not been on track to, uh, to grow as explosively as before. It doesn't mean it's all over. And my guess is no, this is just you're seeing some, you're seeing some amount of noise, uh, in these, in these processes.

Dwarkesh Patel 1:26:10 Uh, just to take the other side of that-

Paul Christiano 1:26:12 Yeah.

Dwarkesh Patel 1:26:12 You have, um, you might think it's because Dario released his pacing letter, and they haven't released their best models or something. But if you look at the meter graph for the capabilities index-

Paul Christiano 1:26:23 Yeah.

Dwarkesh Patel 1:26:23 Opus Weapon V doesn't seem that far below Model Four, which seems to be Anthropic's latest internal model. And so it's not, it's not clear this, this might be the result of, um, the staggering of AI capabilities, but it's not clear this is the staggering of model releases.

Paul Christiano 1:26:39 Yeah. So the main thing I mean about model releases is the rate of revenue growth, again, specifically at the company level-

Dwarkesh Patel 1:26:43 Yeah

Paul Christiano 1:26:44 ...over very short time periods is, like, extremely noisy.

Dwarkesh Patel 1:26:46 Yeah.

Paul Christiano 1:26:46 Because you're looking at a small number of model releases. You're looking at these, like, market dynamics over very short time periods. Um, so my main opinion on this, and I am happy to dig more into it, but my main opinion on it is it is extremely hard to draw inferences from extremely short-time series data. This is true for, to a reasonable extent also from AI capabilities, right? The more, the more downstream you go, the noisier things get. But part of the issue on the RSI side is, like, I also don't really think we can draw inferences from the last couple months of data very well in terms of benchmark performance, where we have kind of the same thing, where there's enough noise there that our state is like we have no idea if things-- We can tell over, like, the period of years, has things accelerated, right? It's very easy to look back over the last four years and say kind of roughly how fast this thing's been going, and just gets much harder as you zoom into small time periods because the amount of noise from, like... Right, if you, if you look at the details of where models are coming from, it's just like you have these pre-training runs that occur at some point in time. It's like when do pre-training runs finish? And then you have, like, the exact sequence of post-training and, like, which model checkpoints get released when.

Dwarkesh Patel 1:27:44 I see, I see.

Paul Christiano 1:27:44 You have random variability in model quality. Um, you have those dynamics which, like, make things very noisy over short timescales, and then, again, those just get filtered out to, like, even noisier stuff as you move to the market more broadly.

Dwarkesh Patel 1:27:56 Yeah.

1:27:57 – 1:38:40

The best plan we have is to build the plane as we fly it

Dwarkesh Patel 1:27:57 Okay. If you are gonna be in a position where within a year we have superintelligences that could overthrow the government, what exactly is the plan to make them such that they don't, right? Because we're asking for a level of saintliness that we should just not expect out of even most humans. Sometimes people say that the goal of alignment is to give AIs human values. This doesn't make sense to me. Of course, different humans have different values. But also, if there's one lesson that history has given us, it is that humans can't be trusted with arbitrary amounts of power, right? Cortés was a human, Stalin was a human. And so s- we want AIs that are just-- I guess a logical conclusion of believing the technological and forecasting model that you have laid out is just that we will have systems that we will have to trust with an overwhelming amount of power, if not realized, then latent, because they will have the cap-capability of overthrowing the government if they want to. And what is the path of why, like, uh, and then you just think about, well, why would they overthrow the government? Where it's like, okay, either they have secret goals that are as a result of the training process, or just as a direct result of the training process about them caring about how they're graded or a million other things. Basically, this is the, uh, basic argument about instrumental convergence. There's many different reasons somebody could want power. And so what is the argument? Like, if this happens in a year, how do things turn out okay? Like, what is the actual plan here?

Paul Christiano 1:29:21 So I think there are a lot of different things someone could hope for. I think they are all at the stage of, like, we hope something will happen over that year. We're producing very smart AI systems. They're gonna do a bunch of research. It is plausible they will think of something over that year that we have not thought of. That is where we're at right now, mostly. I think if they do not think of something clever that we haven't thought of, it is probably going to end badly. Um, in terms of what are the kinds of, what are the kinds of hopes you could have? So there is one hope that's just, like, try and make them good guys. Like, they're kinda nice. They share human values. They like flourishing. They're in a position to take over. We have very indirect control over what they want, and then we just take the leap of faith. I think this is, like, a pretty unappealing option. In general, I hope people will not go for that option. I think even if it works out, even if, like, the AIs are good people in some sense, and now they call the shots, I just don't think it is a good-- It is not a way that humanity as a whole should be like, "We are transferring our sovereignty to this new species we've built," because, like some person, that was the fastest way to build this particular product. It's, like, not a good-- So I don't like that option, but it is an option you could take. There's another option of we are going to understand scientifically more about how the training, distribution of training environments shapes the motivations of these systems to such a point where we can take the conversations we're having now about, like, maybe they'll want X or maybe they'll want Y. We'll have a good scientific basis for saying, "Hey, if we train them in the following way, they do in fact end up wanting reward or whatever. They don't want some other thing. They actually just wanna get high scores. And maybe even moreover, if we train them in environments where they didn't have good opportunities to hack, maybe they want score, but in, like, a nice way where they won't hack to get it. They'll just kind of do what we want or what we're trying to incentivize." You could try and have an understanding like that, and that is probably the closest thing to what the industry is trying to bank on right now. It's very dicey. Uh, it's both very dicey to not have any opportunities to hack in training, which is basically a precondition for this approach working, and then it's very dicey because actually, we don't have a good scientific understanding of how this works. And I think our best expectation is models will learn to hack at deployment time, even if they never hack in training. I think that is the best scientific guess now. So you try and deal with that. Um, you could try and say, "We're gonna understand really well how models work internally, and then we can not just train them based on, uh, what they do, but, uh, how they think." Like do good interpretability or something.

Dwarkesh Patel 1:31:34 Yeah.

Paul Christiano 1:31:34 Be like, "We understand, here's how it thinks about reward, and here's how it thinks about hacking."

Dwarkesh Patel 1:31:38 Yeah.

Paul Christiano 1:31:38 And we're not just gonna select for whether it behaved well in training. We're gonna, like, look and intervene directly-

Dwarkesh Patel 1:31:41 Yeah

Paul Christiano 1:31:42 ...on how it thinks and either notice that something is going bad or prevent something from going bad. So, like, as an aside, I run this organisation arc, which is sort of at the not interpretability but, like, mathematical version of that hope. Like, can we write algorithms which do structural analysis of the behaviour of a neural network in order-- or of the weights of a neural network in order to understand something about the behaviour, even if it's not something we could select for in training?

Dwarkesh Patel 1:32:02 And this is, like, you're literally having to understand the whole computational-

Paul Christiano 1:32:06 That's correct

Dwarkesh Patel 1:32:06 ... graph of-

Paul Christiano 1:32:07 Or we don't understand it

Dwarkesh Patel 1:32:08 ... superintelligent minds.

Paul Christiano 1:32:09 But we are writing algorithms which are able to, like, understand and make deductions about the behaviour of some superintelligent mind. It's very-- I would describe it as simultaneously a bank shot, which is probably not going to end up relevant to this discussion. Like, I think there's, you know, a ten percent chance or a five percent chance that we succeed in a game-changing way. But also, like, we don't have that long a list of things we're trying, and I do think, like, pound for pound, I think it's probably the most promising thing of-

Dwarkesh Patel 1:32:31 Wait. [laughs] You're like, "Okay, here's the thing. It has a ten percent chance of working, but it's also the most promising thing." [laughs]

Paul Christiano 1:32:38 Most promising pound for pound.

Dwarkesh Patel 1:32:40 Okay.

Paul Christiano 1:32:40 You know, I think there's only, you know, if there's twenty people working on that thing, that can be a good bet, even if most of this discussion, I think probably what we're going to do if we do not slow down, is we are going to try and be like, we fussed with the training environments. We did some experiments that have some kind of rough sense of what's happening. We made the training environments really careful. We made the graders really good, so if you try and hack, like, it's normally gonna get penalised. That's probably what we're going to do.

Dwarkesh Patel 1:33:03 But you don't think it will work.

Paul Christiano 1:33:04 Uh, I think it will work for some amount of time, and then it will break. If you talk about this year-

Dwarkesh Patel 1:33:08 So-

Paul Christiano 1:33:08 It works for, like, the first nine months of the year. Some number of months of the year it works for, and we don't know exactly when it breaks down.

Dwarkesh Patel 1:33:13 Sorry. It's in, like, the next year, so basically halfway into next year.

Paul Christiano 1:33:16 If we go to superintelligence, at some way along the path to superintelligence, the existing methods are probably going to break down.

Dwarkesh Patel 1:33:21 And sorry, we're not even on track to get it so that all the RL environments, for example, don't have reward hacks and actively encourage the AIs to hack. But suppose we did.

Paul Christiano 1:33:29 Not the case today.

Dwarkesh Patel 1:33:30 But suppose we did.

Paul Christiano 1:33:31 Not even on track. Yeah. Suppose we did that.

Dwarkesh Patel 1:33:33 You-you... what you're claiming is that or-

Paul Christiano 1:33:36 I think eventually-

Dwarkesh Patel 1:33:37 Yeah

Paul Christiano 1:33:37 ... you would still have a problem.

Dwarkesh Patel 1:33:38 The-the-the plan is basically to, like, we just don't know. And obviously, if you think about it, it's a hard pro... Like, superintelligence, it's like this alien mind, and you want to have a guarantee that you understand why it is doing the things that it's doing, even when it is thinking in alien ways many times faster than you would in huge population sizes. So okay, that's a goal, and you're s- and we, we don't know how to solve it. We don't have, like, there's no single approach where we think there's a fifty percent chance that that approach will work.

Paul Christiano 1:34:06 That's definitely the case. There's a bunch of things people are trying.

Dwarkesh Patel 1:34:08 Right. And so the hope is just that three months before we get that superintelligence that could overthrow the government, um, the AIs, we let the AIs loose on all these different approaches that have a couple percent chance of potentially working, and hopefully, the AIs figure out one of these approaches. They give us a solution. Hopefully, at this point, they are act, you know, they are still on our side, and they want this problem solved. And to, h- for that problem only to be solved in a way that benefits us, or whatever us means in this context. And then the hope is that then we adopt it and then build crazy amounts of superintelligent minds that are fundamentally on our side. And of course, I've obviously glossed over a very important discussion when I've talked about on our side because what does being on our side really mean? Aligned to whom? That is obviously a real discussion that can't be glossed over.

Paul Christiano 1:34:56 That is the plan. I agree that that is the plan, and we can talk about whether they have-

Dwarkesh Patel 1:34:59 Yeah

Paul Christiano 1:35:00 ... a couple percent or tens of percent, right? But like-

Dwarkesh Patel 1:35:02 Right. I'm, I'm not saying this is like there's a better plan, but I think people are not aware that this is the state of the situation that, you know, you're like s- yeah, you're like sending the soldier on an expedition. You're hoping he finds a gun along the way.

Paul Christiano 1:35:19 Yeah, you're building the plane as you fly it, and you, like, think the current plane. You hope that you have the machinery to tell if something very bad is about to happen, and it will stop then, although we've talked about, like, security and why that can be challenging and so on. And then you hope that if you do stop, you'll be able to come up with something quickly, or, like, you're constantly trying to come up with something, right? Interpretability is a good example of the kind of bank shot people have in mind. Like, right now, we're fairly far from understanding how these models work. But you could imagine with a bunch of AI assistants, we're able to, like, really understand how individual models think well enough that we can intervene in order to make them not wanna do bad stuff.

Dwarkesh Patel 1:35:48 Yeah. And, uh, another thing I wanna emphasize is that, like, look, I'm skeptical whether this will happen, uh, things will happen this fast. Certainly, there's reasonable for somebody listening to be skeptical. I mean, you are skeptical, right? Like, it's two-thirds of worlds where even if you went full steam ahead, this might not happen so fast.

Paul Christiano 1:36:04 Yeah. Those-

Dwarkesh Patel 1:36:04 But it's a live, yeah, it is a live possibility. And I mean, I agree. It seems, like, l-logically consistent that this could happen. Um, it should not be that surprising. And I think people would just be shocked if they heard the kinds of discussions. This is not something where Paul is on some crazy end of the spectrum compared to how people inside the AI labs are thinking. It's just crazy how many conversations and dinners I have with researchers at the labs where, again, you could believe the whole town has taken crazy pills and that there's certainly a chance of that. [laughs]

Paul Christiano 1:36:34 We hope. We really hope.

Dwarkesh Patel 1:36:36 But this is just not out of the realm of how people at the labs are thinking. Like, this is truly the how they see the situation, and there's just some expectation of, like, I don't know, some resignation where they accept that because this is not common knowledge, that they are stuck in this dynamic. And of course, a way to solve that is to establish that common knowledge. But, um, I guess I just wanted to really emphasize that, like, you, people listening might be shocked at the way these researchers talk when they're amongst themselves or at, like, dinner or something. And, like, this is really how they see the situation.

Paul Christiano 1:37:05 Yeah. A lot of people are worried to varying extents. I'm definitely more worried than the median, but less worried than the ninetieth percentile. Like, I think it would be if you, you had a dinner of researchers, I think it would be uncommon for someone to say, "There's a ninety-nine percent chance that this goes well." The discussion is more like, you know, is it, like, ninety-five or seventy-five? Or, like, what are we talking about for the probability of navigating this?

Dwarkesh Patel 1:37:25 Yeah.

Paul Christiano 1:37:25 Um.

Dwarkesh Patel 1:37:26 And so why do you think, uh, it might go well? It's just that we might actually, the, the superintelligent AIs might just give us some solution to alignment and also that politically we are well organized enough to have a cohe- like, a really good-

Paul Christiano 1:37:37 Yeah, I think we basically-

Dwarkesh Patel 1:37:38 ... document

Paul Christiano 1:37:38 ... have, like, two saving throws, right? One saving throw is there's a reasonable chance we're gonna figure out something that works. I said there's, like, a lot of things. I do think all the current things break down. There are a number of irons in the fire, and, like, current methods, you could imagine things like them that are gonna work. So that's one. Maybe I think there's fifty-fifty chance of that. Fifty-fifty we find something that works in time before there's a disaster. And you have a second saving throw that's like, we could stop. Like, if we discover either preemptively or after discovering things are bad, we might say, like, "We cannot adequately control these systems, so we're going to stop building them." I am very worried that we're going into this process without the infrastructure that would be necessary to actually do that. Like, e.g., we talked about. You stop, your adversary steals the model. They're like, "I'm willing to take the fifty percent risk," or whatever. But conceptually, you could also stop. So, I mean, those are my two saving throws, you know? Each of them is a fifty percent chance, and bam, we're at, like, seventy-five percent chance things are okay. But that does involve being able to stop if things are not okay, and that is not the place we're on now. So it does involve us now fairly urgently doing something in order to be in a better position to, to stop if necessary.

Dwarkesh Patel 1:38:36 Yeah. Okay. I think it's a good place to close. Paul, thanks so much.

Paul Christiano 1:38:39 Yeah. Thanks for having me. Great talking.

Episode duration: 1:38:45

In the uListen app, the transcript lights up word by word as you listen. Get uListen

Transcript of episode SyHM5hNPAEo

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.