Sholto Douglas & Trenton Bricken — How LLMs actually think

Narrator 0:00 (laughs) It's right after this, and you ruin it. (laughs)

Dwarkesh Patel 0:05 (laughs)

Sholto Douglas 0:05 Oh, my God. (laughs)

Trenton Bricken 0:08 You're failing the line test right now, really badly. This is like...

Narrator 0:12 Yeah, it is. It is. (laughs)

Trenton Bricken 0:12 I'm like, "Wait, really?"

Dwarkesh Patel 0:15 "Can we drink on our glasses?"

Narrator 0:20 That's funny. (laughs)

Dwarkesh Patel 0:21 The glass go? (laughs)

Sholto Douglas 0:21 Yeah, let's go. Uh... (laughs)

Dwarkesh Patel 0:23 Oh my God, dude. I'm like, I feel like leaving the house.

Narrator 0:26 (laughs)

Dwarkesh Patel 0:27 My backpack is like, launching...

Narrator 0:32 (laughs)

Trenton Bricken 0:32 (laughs) Uh...

Dwarkesh Patel 0:33 Let's get like no context on the chair.

Narrator 0:35 (laughs)

Trenton Bricken 0:37 (laughs)

Dwarkesh Patel 0:38 Let's go. (laughs)

Sholto Douglas 0:39 Dude, it is literally falling over.

Narrator 0:41 Yeah. It's like... (laughs)

Dwarkesh Patel 0:41 Have you seen the videos?

Sholto Douglas 0:43 Yeah.

Narrator 0:43 (laughs)

Dwarkesh Patel 0:45 I think the video has shown it enough that we can almost live it out.

Sholto Douglas 0:48 Let's do it.

Narrator 0:48 Like, don't want to collapse it.

Sholto Douglas 0:50 (laughs)

Narrator 0:51 (laughs)

Dwarkesh Patel 0:52 Okay. Today I have, uh, the pleasure to talk with two of my good friends, Sholto and Trenton. Um, Sholto-

Sholto Douglas 0:59 You just got us mixed up. (laughs)

Narrator 1:02 (laughs)

Trenton Bricken 1:02 (laughs)

Dwarkesh Patel 1:02 I knew we did. (laughs) I wasn't going to say anything.

Narrator 1:05 So let's do this in reverse.

Dwarkesh Patel 1:09 (laughs)

Narrator 1:09 How about I started with "my good friends"? (laughs)

Dwarkesh Patel 1:13 Yeah, Gemini 1.5, the context length, just wow.

Narrator 1:17 (laughs)

Dwarkesh Patel 1:18 (laughs) Oh shit. Anyways, um, Sholto, uh, Noam Brown... (laughs)

Narrator 1:26 (laughs)

Dwarkesh Patel 1:26 Noam Brown, the guy who wrote the Diplomacy paper, he said this about Sholto. He said, "He's only been in the field for 1.5 years, but people in AI know that he was one of the most important people behind Gemini's success." Um, and Trenton, who's at Anthropic, uh, works on mechanistic interpretability, and it was widely reported that he has solved alignment.

Narrator 1:46 (laughs)

Sholto Douglas 1:47 (laughs)

Dwarkesh Patel 1:47 With his recent paper.

Trenton Bricken 1:47 He read random Twitter.

Narrator 1:49 Oh. (laughs)

Dwarkesh Patel 1:49 On, uh... Um, so this will be a capabilities only podcast. Alignment is already solved, so no need to discuss further. Um, okay, so let's start by talking about context lengths.

Trenton Bricken 2:00 Yep.

Dwarkesh Patel 2:02 G- it seemed to be under hyped given how important it seems to me to be that you can just put a million tokens into context. There's apparently some other news that, you know, got pushed to the front for some reason. But, um, yeah, I- uh, is, tell me about how you see the future of long context lengths and what that implies for these models?

Sholto Douglas 2:18 Yeah. So I think it's really under hyped because until I started working on it, I didn't really appreciate how much of a step up in intelligence it was for the model to be, have the onboarding problem basically instantly solved. Um, and you can see that a little bit in the perplexity graphs in the paper, where just throwing millions of tokens' worth of context about a code base allows it to become dramatically better at predicting the next token in a way that you'd normally associate with huge increments in model's scale. But you don't need that. All you need is like a new context. Um, so under hyped, uh, and yeah, buried by some other news.

Dwarkesh Patel 2:49 In context, are they as sample efficient and smart as humans?

Sholto Douglas 2:54 I think that's really worth exploring, because, for example, one of the evals that we did in the paper, uh, has it learning a language in context better than a human expert, uh, could learn that new language over the course of a couple months. And this is only like a pretty small demonstration. But I'd be really interested to see things like Atari games or something like that, where you like throw in a couple 100 like or 1,000 frames, labeled actions, and then in the same way that you'd like show your friend how to play a game-

Dwarkesh Patel 3:17 Right.

Sholto Douglas 3:18 ... and see if it's able to reason through. It might, at the moment, you know, with the infrastructure and stuff, it's still a little bit slow at like-

Dwarkesh Patel 3:23 Yeah.

Sholto Douglas 3:23 ... uh, doing that. But I would actually, I would guess that might just work out of the box in a way that would be pretty mind-blowing.

Trenton Bricken 3:30 And, and crucially, I think this language was esoteric enough that it wasn't in the training data.

Sholto Douglas 3:33 Right. Exactly. Yeah, if you look at the model before, uh, it has that context thrown in, it just, it doesn't know the language at all and it can't get any translations yet.

Dwarkesh Patel 3:40 And this is like an actual, like human language, not just-

Sholto Douglas 3:42 Yeah.

Dwarkesh Patel 3:43 Okay.

Sholto Douglas 3:43 Exactly. An actual human language.

Dwarkesh Patel 3:44 So if this is true, it seems to me that these models are already in an important sense superhuman. Um, not in the sense that they're smarter than us, but I can't keep a million tokens in my context when I'm trying to solve a problem, remembering and integrating all the information in a entire code base. Am, am I wrong in thinking this is like a huge unlock?

Sholto Douglas 4:05 I actually genuinely think that's true. Uh, like previously, I've been frustrated when models aren't as smart. Like you ask them a question, and you want it to be smarter than you or to know things that you don't.

Trenton Bricken 4:15 Right.

Sholto Douglas 4:15 And this allows them to know things that you don't in a way that like, it just ingests a huge amount of information in a way you just can't.

Dwarkesh Patel 4:21 Mm-hmm.

Sholto Douglas 4:21 Um, so yeah. It's extremely important.

Dwarkesh Patel 4:24 Well, how do we explain in-context learning?

Sholto Douglas 4:26 Yeah. So there's a piece of... There's a line of work I quite like, uh, where it looks at in-context learning, uh, as basically like very similar to gradient descent but, um-

Trenton Bricken 4:35 Mm-hmm.

Sholto Douglas 4:35 ... like they're sort of, the attention operation can be viewed as gradient descent on the in-context data. That paper had some cool plots where it basically showed like we take N steps of gradient descent, and that looks like N layers of in-context learning. Um, it looks very similar. So I think like that's one way of viewing it, um, and trying to understand what's going on.

Dwarkesh Patel 4:50 Yeah.

Trenton Bricken 4:50 Yeah. And, uh, you can ignore what I'm about to say because given the introduction-

Dwarkesh Patel 4:54 (laughs)

Sholto Douglas 4:54 (laughs)

Trenton Bricken 4:55 ... alignment is solved, and now safety isn't a problem. But uh, I think the context stuff does get problematic, um, but also interesting here. Um, I think there'll be more work coming out in the not too distant future, um, uh, around what happens if you give 100 shot prompt, um, for jailbreaks, adversarial attacks.

Dwarkesh Patel 5:15 Mm-hmm.

Trenton Bricken 5:16 Um, it's also interesting in the sense of if your model is doing gradient descent and learning on the fly, uh, even if it's been trained to be-

Sholto Douglas 5:24 Right.

Trenton Bricken 5:24 ... uh, harmless-

Sholto Douglas 5:24 Yeah.

Trenton Bricken 5:24 ... um, it, you're dealing with a totally new model in a way. Um, you're like fine-tuning but in a way where you can't control what's going on.

Dwarkesh Patel 5:32 Can, can you explain w- what do you mean by gradient descent is happening in the forward pass and attention?

Sholto Douglas 5:37 You probably have a better memory than I.

Trenton Bricken 5:37 Yeah. There was... No, no, no. There was something in the paper about trying to ch- teach the model to do linear regression-

Sholto Douglas 5:41 Right.

Trenton Bricken 5:41 ... but like just through the number of samples they gave in the context.

Sholto Douglas 5:44 Yeah.

Trenton Bricken 5:44 And you can see, if you plot on the X axis like number of, um, shots that it has, or examples, and then like the loss it gets on just like ordinary least squares regression-

Sholto Douglas 5:53 Yeah.

Trenton Bricken 5:54 ... that will go down with time.

Sholto Douglas 5:55 Yeah. And it goes down exactly matched with, uh, number of gradient descent steps.

Trenton Bricken 5:59 Yep.

Sholto Douglas 5:59 Yeah.

Trenton Bricken 5:59 Yeah. Exactly.

Dwarkesh Patel 6:00 And, uh, okay. Um, I, I only read the intro and discussion section of that paper. But in the discussion, the way they framed it is that w-... the model, in order to get better at long context, uh, uh, tasks, the model has to get better at learning to learn-

Sholto Douglas 6:15 Yeah.

Dwarkesh Patel 6:15 ... from these examples or from the context that is already within the window. And that, the implication of that is the model learn, if like meta-learning happens because it's, has to learn how to get better at long context tasks, then in some important sense, the pa- the t- the task of intelligence is like, requires long context examples and long context training.

Sholto Douglas 6:36 Like, meta learn, like you have to induce meta-learning.

Dwarkesh Patel 6:39 Right.

Sholto Douglas 6:39 And like understanding how to better induce meta-learning in your pre-training process is like a very important thing to actually, about give it flexible or adaptive intelligence.

Dwarkesh Patel 6:45 Right. But you can proxy for that just by getting better at doing long context tasks. Um, one of the bottlenecks for AI progress that many people identify is the inability of these models to perform tasks, uh, uh, tasks on long horizons, which means engaging with a task for many hours or even many weeks or months, where like if I have, I don't know, an assistant or an employee or something, they can just do a thing I tell them for a while. Um, and AI agents haven't taken off for this reason from what I understand. So how linked are long context windows and the ability to perform well on them, and the ability to do these kinds of long horizon tasks that require you to engage with, uh, an assignment for many hours, or are these unrelated concepts?

Sholto Douglas 7:27 Uh, I mean, I would actually take issue with the, that being the reason that agents haven't taken off.

Dwarkesh Patel 7:32 Mm-hmm.

Sholto Douglas 7:32 Um, where I think that's more about like nines of reliability and the model actually successfully doing things. And if you just can't chain tasks successively with high enough probability, then you won't get something that looks like an agent. And that's why something like an agent might fall in more of a step function in sort of like GPT-4 class models, Gemini ultra class models had not enough. Um, but maybe like the next incremental model scale means that you get that extra nine, even though the, like the loss isn't going down that dramatically, that like small amount of extra ability gives you the like extra...

Dwarkesh Patel 7:59 Yeah.

Sholto Douglas 8:00 And, and like yeah, obviously you need some amount of context to-

Dwarkesh Patel 8:02 Right.

Sholto Douglas 8:03 ... fit long horizon tasks. But I don't think that's been the limiting factor up to now. Mm-hmm.

Trenton Bricken 8:07 Yeah, um, the NeurIPS best paper this year by Ryland Schafer was the lead author, points to this as like the emergence of mirage, where people will have a task and you get the right or wrong answer depending on if you've sampled the last five tokens correctly. And so naturally that's, you're multiplying the probability of sampling all of those. Um, and, and if you don't have enough nines for reliability, then you're not gonna get emergence.

Sholto Douglas 8:31 Exactly.

Trenton Bricken 8:31 And all of a sudden you, you do.

Sholto Douglas 8:33 Yeah.

Trenton Bricken 8:34 And it's like, "Oh my gosh, this ability is emergent." When, when actually it was kind of almost there-

Sholto Douglas 8:38 Yeah.

Trenton Bricken 8:38 ... to begin with.

Sholto Douglas 8:39 And there are ways that you can find like a smooth metric for that.

Dwarkesh Patel 8:42 Uh, is yeah, human eval or whatever the, in the GPT-4 paper, the, the coding problems they ha- they measured it by-

Sholto Douglas 8:47 Log pass rate and stuff.

Dwarkesh Patel 8:48 Exactly. Yeah. Uh, for the audience, the context on this is, uh, you, it's, basically the idea is you wanna, uh, when you're measuring how much progress there has been on a specific task like solving coding problems, you, um, you up-weighted when it gets it right only one in a thousand times, you don't like give it a one in a thousand score, 'cause it's like, "Oh, like got it right some of the time." And so the curve you see is like it gets it right one in a thousand, then one in a hundred, then one in 10, and so forth. So actually I wanna follow up on this. So if your claim is that the AI agents haven't taken off because of reliability rather than long horizon task performance, isn't the lack of reliability when a task is chained on top of another task, on top of another task, isn't that exactly the difficulty with long horizon tasks is that like you have to do 10 things in a row or 100 things in a row and diminishing the reliability of any one of them, uh, w- we or the, yeah, the probability goes down from 99.99 to 99.9, then like the whole thing gets multiplied together and the whole thing's becomes much less likely to happen?

Sholto Douglas 9:50 Right. That, that is exactly the problem. But the, the key issue you're pointing out there is that your base pass, like task solve rate is 90%.

Dwarkesh Patel 9:56 Mm-hmm.

Sholto Douglas 9:57 Um, and if it was 99%, then chaining them doesn't become a problem.

Dwarkesh Patel 9:59 Mm-hmm.

Sholto Douglas 10:00 Um, but also-

Trenton Bricken 10:01 It's like a se-

Sholto Douglas 10:01 Yeah, exactly. And I, I think this is also something that just like hasn't been properly studied enough. If you look at all of the evals that are commonly, like the academic evals, uh, a single problem, right? Um, you know, like the math problem, it's like one like typical math problem or MMOU, it's like one university level like, like problem from across different topics.

Dwarkesh Patel 10:18 Mm-hmm.

Sholto Douglas 10:19 You are beginning to start to see evals looking at this properly, uh, via more complex tasks like SuiteBench-

Dwarkesh Patel 10:24 Yeah.

Sholto Douglas 10:24 ... uh, where they take a whole bunch of GitHub issues and that's like, that is like a reasonably long horizon task.

Dwarkesh Patel 10:28 Right.

Sholto Douglas 10:28 But it's still not a multi-

Dwarkesh Patel 10:30 Yeah.

Sholto Douglas 10:30 It's like a multi, you know, sub-hour as opposed to like multi-hour or multi-day task. Um, and so I think one of the things that will be really important to do over the next however long is understand better what does success rate of a long horizon task look like. Um, and I think that's even important to understand what the economic impact of these models might be, and like actually properly judge increasing capabilities-

Dwarkesh Patel 10:50 Right.

Sholto Douglas 10:50 ... by like cutting down the tasks that we do and the inputs and outputs involved into minutes or hours or, or days and seeing how good it is successively like chaining and completing tasks-

Dwarkesh Patel 11:01 Right.

Sholto Douglas 11:01 ... at those different resolutions of time. Because then that tells you like how automatable a job family or task family is-

Dwarkesh Patel 11:06 Mm-hmm.

Sholto Douglas 11:06 ... um, in a way that like MMOU scores don't.

Dwarkesh Patel 11:09 Mm-hmm.

Trenton Bricken 11:10 I mean, it was less than a year ago that we introduced 100K context windows and I think everyone was pretty surprised by that.

Sholto Douglas 11:16 Yeah. Yeah.

Trenton Bricken 11:16 So yeah, uh, everyone would just kind of had this soundbite of quadratic attention costs and-

Sholto Douglas 11:21 Yeah.

Trenton Bricken 11:21 ... we can't have long context windows.

Sholto Douglas 11:22 Yeah.

Trenton Bricken 11:22 And uh, here we are. So yeah, like the benchmarks are being actively-

Sholto Douglas 11:26 Yeah.

Trenton Bricken 11:26 ... made.

Dwarkesh Patel 11:27 Wait, wait. So doesn't the fact that there's these companies, Google and I don't know, Magic, maybe others who have million token attention imply that the quadra... Y- you shouldn't say anything 'cause (laughs) you're not allowed.

Sholto Douglas 11:38 (laughs)

Dwarkesh Patel 11:38 But d- wouldn't, doesn't that like imply that it's not quadratic anymore or are they just eating the cost? Interesting.

Sholto Douglas 11:42 Well, like who knows what Google is doing for its long context team, right?

Dwarkesh Patel 11:45 (laughs) Yeah. I'm not making any, making any claims here.

Sholto Douglas 11:47 I'm not saying, I'm not saying it's either .

Dwarkesh Patel 11:48 Right. Right.

Sholto Douglas 11:48 It's just one of the things that's frustrated me about, uh, in like the general research fields approach to attention is that there's an important way in which the quadratic cost of attention is actually dominated in typical dense transformers by the MLP block.

Dwarkesh Patel 12:02 Right.

Sholto Douglas 12:03 Um, so-You have this N squared term that's associated with attention, but you also have an N squared term that's associated with the D model, the residual stream dimension of, uh, the model. And if you look, uh, I think Sasha Rush has a great, uh, tweet where he, it looks like, basically plots the curve of the cost of attention, respective like the cost of like really large models and attention actually trails off. Um, and you actually need to be doing pretty long context before that, uh, that term becomes, uh, like really important. Uh, and the second thing is that people often talk about how attention at inference time is, uh, such a huge cost, right? And if you think about when you're actually generating tokens, the operation is not N squared. It is one Q, like one set of Q vectors looks up a whole bunch of KV vectors, and that's linear with respect to, uh, the amount of like context that the model has. Um, and so I think this drives a lot of the like recurrence and state space research where people have this meme of oh, like linear attention and all this stuff. Um, and as Trenton said, there's like a graveyard of ideas around attention. Um, and not to think, I don't think it's worth exploring, but, uh, I think it's important to consider where the actual like strengths and weaknesses of it are.

Dwarkesh Patel 13:12 Uh, okay. So, uh, wh- what do you make of this take? Um, as we move forward through the takeoff, more and more of the learning happens in the forward pass. So originally, like all the learning happens in the backward, uh, you know, d- during like this like bottom up sort of hill climbing evolutionary p- process. Um, if you think in the limit during the intelligence explosion, it just like the AI is like maybe wri- like hand writing the weights or like doing GoFi or something, and we're in like the middle step where like a lot of learning happens in context now with these models, uh, a lot of it happens within the backward pass. Does this seem like a meaningful, uh, gradient along which progress is happening of like how much... Because the, the broader thing being the, um, if you're learning in the forward pass, it's like much more sample efficient 'cause you can kind of like basically think as you're learning, like when humans, when you read a textbook, you're not just skimming it and trying to absorb what, you know, what inductive, these- th- these words follow these words. You like read it and you think about it and then you read some more, you think about it. Um, I don't know. Does this seem like a sensible way to think about the progress?

Sholto Douglas 14:14 Yeah. It may just be one of the ways in which like, you know, birds and planes like fly, but they fly slightly differently. And like the virtue of technology allows us to do that, like, like

Trenton Bricken 14:22 Mm-hmm.

Sholto Douglas 14:22 ... I guess you could accomplish things that birds can't.

Trenton Bricken 14:25 Mm-hmm.

Sholto Douglas 14:25 Um, it might be that context length is similar in that, uh, it allows it to have a working memory that we can't. Uh, but functionally is not like the key thing towards actual reasoning. The key step between GPT-2 and GPT-3 was that all of a sudden, like there was this meta learning behavior that was observed-

Trenton Bricken 14:41 Yeah.

Sholto Douglas 14:41 ... in training, uh, like, like in the pre-training of the model.

Trenton Bricken 14:44 Yeah.

Sholto Douglas 14:44 Um, and that's a- as you said, like it's something to do with you give it some amount of context, it's able to adapt to that context. And that was a behavior that wasn't really observed before that, um, at all. And maybe that's a mixture property of context and scale and this kind of stuff, but it wouldn't have occurred in a... to model tiny context, I would say.

Dwarkesh Patel 15:00 Mm-hmm. Um, th- this is actually an interesting point. So when we talk about scaling up these models, how much of it comes from just making the models themselves bigger and how much comes from the fact that during any single call, uh, you are using more compute? So if you think of diffusion, you can just iteratively a- keep adding more compute and if adaptive compute is solved, you can keep doing that. Um, and in this case, if there's a quadratic penalty for attention, but you're doing long context anyways, then you're still dumping in more compute during, not during training, not during having bigger models, but just like... Yeah.

Trenton Bricken 15:36 Yeah. It's, it's interesting 'cause you do get more forward passes by having more tokens.

Dwarkesh Patel 15:40 Right.

Trenton Bricken 15:41 Um, my one gripe, I guess I have two gripes with this though.

Dwarkesh Patel 15:45 Mm-hmm.

Trenton Bricken 15:45 Or maybe three. So one, like in the AlphaFold...

Sholto Douglas 15:48 That's a lot of gripes (laughs) .

Dwarkesh Patel 15:48 (laughs)

Trenton Bricken 15:48 In the, in the AlphaFold paper, um, one of the transformer modules, they have a few, and the architecture's like very intricate.

Dwarkesh Patel 15:54 Yeah.

Trenton Bricken 15:54 Um, but they do, I think five forwards passes through it and will gradually like refine their solution as a result. Um, you can also kind of think of the residual stream. I mean, Asholto alluded to kind of the read-write operations as like a poor man's adaptive compute, um, where it's like, I'm just gonna give you all these layers and like if you want to use them, great. If you don't, then that's also fine. Um, and then people will be like, "Oh, well the brain is, is recurrent and you can like do however many loops through it you want." And I think to a certain extent that's right, right? Like if I ask you a hard question, you'll spend more time thinking about it and that would correspond to more forward passes. But, um, I think there's a finite number of forward passes that you can do. Um, it's kind of with language as well. People are like, "Oh, well human language can have like infinite recursion in it, like infinite nested statements of like, the boy jumped over the bear that was doing this, that had done this, that had done that." But like empirically you'll only see five to seven levels of recursion, um, which kind of relates to, um, whatever that magic number of like how many things you can hold in working memory at any given time is. Um, and so yeah, it's not infinitely recursive, but like does that matter in the regime of human intelligence and like, can you not just add more layers?

Dwarkesh Patel 17:03 Break down

Dwarkesh Patel 17:04 for me, w- you're referring to this in some of your previous answers of, listen, you have these long contexts and you can hold m- more things in memory, but like ultimately comes down to your ability to mix concepts together to do some kind of reasoning. Uh, and these models aren't necessarily human level at that, even in context. Break down for me how you see storing just raw information versus reasoning and what's in between. Like where's the reasoning happening? Is that, uh, where's, where's just like storing raw information happening? What's different between them I- in these models?

Trenton Bricken 17:37 Yeah. I don't have a super crisp answer for you here. Um, I mean, obviously with the input and output of the model, you're, you're mapping back to actual tokens, right? And then in between that you're, you're doing higher level processing. Um-

Dwarkesh Patel 17:52 Um, uh, b- before we get deeper into this, we should explain to the audience, you referred earlier to Anthropics way of thinking about transformers as these read right operations that layers do. One of you should just kind of explain at a high level what you mean by that.

Trenton Bricken 18:06 So the residual stream, y- imagine you're in a boat going down a river, and, um, the boat is kind of the current query, uh, where you're trying to predict the next token.

Dwarkesh Patel 18:15 Yeah.

Trenton Bricken 18:16 So it's the cat sat on the blank.

Dwarkesh Patel 18:18 Right.

Trenton Bricken 18:18 And, and, uh, y- then you have these little, like, streams that are coming off the river, where you can get extra passengers or collect extra information if you want, and those correspond to the attention heads and MLPs-

Dwarkesh Patel 18:29 Yeah.

Trenton Bricken 18:30 ... that are, that are part of the model.

Dwarkesh Patel 18:31 Right. And- Okay, so you, you-

Sholto Douglas 18:33 I, I was just gonna-

Dwarkesh Patel 18:33 Yeah, please chime in.

Sholto Douglas 18:34 I almost think of it like the working memory-

Dwarkesh Patel 18:35 Right.

Sholto Douglas 18:36 ... of the model.

Dwarkesh Patel 18:36 Yeah.

Sholto Douglas 18:37 Like, the, the RAM of the computer where you're, like, choosing what information to read in-

Dwarkesh Patel 18:40 Yeah. Exactly.

Sholto Douglas 18:41 ... so you can do something with it, and then maybe read, like, read something else in later on. Yeah.

Trenton Bricken 18:45 And you can operate on sub-spaces of that high dimensional vector.

Sholto Douglas 18:48 Exactly.

Trenton Bricken 18:48 Um, a ton of things are, uh, I mean, at this point I think it's, it's a, a- almost given that, like-

Sholto Douglas 18:53 Yeah.

Trenton Bricken 18:53 ... are encoded in super position.

Sholto Douglas 18:54 Yeah.

Trenton Bricken 18:55 Right? So it's like, yeah, the residual stream is just one high dimensional vector, but actually there's a ton of different vectors that are packed into it.

Dwarkesh Patel 19:03 Yeah. I, I might, like, just, like, dumb it down, like, w- as the way that would have made sense to me a few months ago of, okay, so you have, you know, whatever words are in the input you put into the model. All those words get converted into these, uh, tokens, and those tokens get converted into these vectors. And basically it just, like, the, this small amount of information that's moving through the model and w- the way you explained it to me, Sholto, and, uh, this paper talks about is early on in the model maybe it's just doing some very basic things about, like, what do these tokens mean? Like, if it says, like, ten plus five, just, like, moving information about to have the, have that, um-

Sholto Douglas 19:40 A good representation.

Dwarkesh Patel 19:40 Exactly. Just a represent- And in the middle, maybe, like, the deeper thinking is happening about, like, how to think, yeah, how to solve this. At the end you're converting it back into the output token, because the end product is you're trying to predict the probability of the next token from the last of those residual streams. Um, and so yeah, it is interesting to think about, like, just, like, the small compressed amount of information moving through the model and it's, like, getting modified in different ways. Uh, Trenton, so you're- it's interesting, uh, y- you're one of the few people who have, like, background from neuroscience, so you can think about the analogies here, uh, to, yeah, to the brain. And in fact I have o- one of our friends, Doahe, he wrote a paper in grad school about thinking about attention in the brain, and he said this is the only or first, uh, w- like, ex- neural explanation of why attention works. Whereas we have evidence from why the CNNs work b- convolutional neural network- networks work based on the visual cortex or something. Um, yeah. I'm, I'm curious h- uh, do you think in the brain there's something like a residual stream of this compressed amount of information that's moving through and it's getting modified, uh, as you're thinking about something? Even if that's not what l- is literally happening, do you think that's a good metaphor for what's happening in the brain?

Trenton Bricken 20:55 Yeah. Yeah. So at least in the cerebellum you basically do have a residual stream.

Dwarkesh Patel 20:59 Mm.

Trenton Bricken 21:00 Where, um, the whole what we'll call the attention module for now, and I can go into whatever amount of detail you want for that, um, you have inputs that route through it, but they'll also just go directly to the, like, end point that, that that module-

Dwarkesh Patel 21:15 Mm.

Trenton Bricken 21:15 ... will contribute to. So there's a direct path and an indirect path. Um, and, and so the model can, like, pick up whatever information it wants and then add that back in. Um-

Dwarkesh Patel 21:26 What, what happens to the cerebellum?

Trenton Bricken 21:27 Uh, so the cerebellum nominally just does fine motor control, um, but I analogize this to the, um, person who's lost their keys and is just looking under the streetlight, where it's very easily to observe this behavior. Um, one leading cognitive neuroscientist said to me that a dirty little secret of any fMRI study where you're looking at brain activity for a given task is that the cerebellum is almost always active and lighting up for it. Um, if you have a damaged cerebellum, you also are much more likely to have autism, um, so it, it's associated with, like, um, social skills, um, in one of these particular studies where I think they use PET instead of fMRI. But, um, when you're doing next token prediction, (laughs) the cerebellum lights up a lot.

Dwarkesh Patel 22:10 Mm.

Trenton Bricken 22:10 Um, also, 70% of your neurons in the brain are in the cerebellum.

Dwarkesh Patel 22:14 Mm.

Trenton Bricken 22:14 They're small, um, but they're, they're there, and they're taking up real metabolic cost.

Dwarkesh Patel 22:21 Th- this was one of Gern's points that, like, what changed with humans was not just that we have more neurons, or he says, uh, he shared this article, um, but specifically there's more neurons in the cerebral cortex and the cerebellum, and y- y- you should say more about this. But, like, they're, they're more metabolically expensive and they're more involved in signaling and sending information back and forth.

Trenton Bricken 22:41 Yeah.

Dwarkesh Patel 22:41 And what we get is that attention to what's going on.

Trenton Bricken 22:43 Yeah. Yeah. So, so I guess the, the main thing I want to communicate here, um, so back in the 1980s, uh, Pentti Kanerva, uh, came up with a associative memory algorithm for, um, I have a bunch of memories, I want to store them, um, there's some amount of noise or corruption that's going on, and I want to query or retrieve the best match. And so he writes this equation for how to do it, and a few years later realizes that if you implemented this as an electrical engineering circuit, it actually looks identical to the core cerebellar circuit. And that circuit, and the cerebellum more broadly, is not just in us, it's in basically every organism. There's active debate on whether or not cephalopods have it. They kind of have a different evolutionary trajectory. But, um, even fruit flies with the drosophila mushroom body, uh, that is the same cerebellar architecture. Um, and so th- that convergence and then my paper which shows that actually this operation is to a very close approximation the same as the attention operation, including implementing the Softmax and having this sort of, like, nominal quadratic cost that we've been talking about. Um, and so the three-way convergence here and the takeoff and success of transformers, um, seems pretty striking to me.

Dwarkesh Patel 23:56 Yeah. I, I, okay, I want to zoom out and ask, I think what motivated this discussion in the beginning was...We were talking about like, wait, what, what is the reasoning? What is the memory? Um, when you think about th- the analogy you found to attention and this, um, do you think of this as more just looking up the relevant memories or the relevant facts? And if that is the case, like where is the reasoning happening in the brain? Li- uh, how, how do we think about like how that builds up into the reasoning?

Trenton Bricken 24:22 Yeah. So maybe my hot take here, I don't know how hot it is, is that like most, most intelligence is pattern matching.

Dwarkesh Patel 24:32 Mm-hmm.

Trenton Bricken 24:32 And you can do a lot of really good pattern matching if you have a hierarchy of associative memories. Um, so you, you have, you start with your very basic associations between just like objects in the real world. Um, but you can then chain those and have more abstract associations such as like a wedding ring symbolizes like so many other associations that are, are downstream. Uh, and so I, a- a- and you could even generalize, uh, the attention operation and this associative memory as the MLP layer as well, in it's in a long-term setting where you don't have like tokens in your current context. Um, but I, I think this is an argument that like associat- association is all you need. Um, and associative memory in general as well, it's not... So, so you can do two things with it. Um, you can both de-noise or retrieve a current memory. So like if I see your face but it's like raining and cloudy, I can, I can de-noise and kind of like gradually update my query towards my memory of your face.

Dwarkesh Patel 25:35 Right.

Trenton Bricken 25:35 But I can also, um, access that memory and then the, the value that I get out actually points to some other totally different part of the space. And so y- so a very simple instance of this would be, um, if you learn the alphabet, right? And so I, I query for A and it returns B. I query for B and it returns C. And, and you can traverse the whole thing. Um, yeah.

Dwarkesh Patel 25:56 Yeah. I, uh, d- that, one of the things I talked to Dennis about was he had a paper in 2008 that memory and imagination are very linked because of this very thing that you mentioned of memory is reconstructive. And so you are in some sense imagining every time you're thinking of a memory 'cause you're only storing a condensed version of it and you're like have to... And this is famously why human memory is terrible and like why people in the witness box or whatever will just make shit up. Um, okay, so, uh, l- l- let me ask a stupid question. So you like reads Sherlock Holmes, right? And like the guy's incredibly sample efficient. You'll, he'll like see a few observations and he'll like, uh, uh, basically figure out who committed the crime 'cause there's a series of deductive steps that leads from somebody's tattoo and what's on the wall to, uh, w- the implications of that. How does that fit into this picture? 'Cause like crucially what makes him smart is that there's not like an association, but there's a sort of deductive connection between different pieces of information. Um, would you just explain it as there, that, that's just like high, higher level association? Like w- yeah.

Trenton Bricken 27:02 I think so. Yeah. So, so I think learning these higher level associations to be able to then map patterns to each other as a, as kind of like a meta learning. I think in this case he would also just have a really long context length.

Dwarkesh Patel 27:12 (laughs)

Trenton Bricken 27:12 Or really long, a really long working memory, right? Where he can like have all of these bits and continuously query them as he's coming up with whatever theory so that the theory is moving through the residual stream.

Dwarkesh Patel 27:25 (laughs)

Trenton Bricken 27:25 And, and then he's, he's, has, his attention heads are querying his, his context.

Dwarkesh Patel 27:30 Right.

Trenton Bricken 27:30 But then, um, how he's projecting his query and keys in the space and how his MLPs are then retrieving like longer term facts or, or modifying that information is allowing him to then in later layers do even more sophisticated queries and slowly be able to reason through and come to a meaningful conclusion.

Sholto Douglas 27:51 That feels right to me in terms of like you're looking back in the past, you're selectively reading in certain pieces of information, comparing them, maybe that informs your next step of like what piece of information you now need to pull in and then you build this representation which I like progressively looks closer and closer and closer-

Trenton Bricken 28:04 Right.

Sholto Douglas 28:04 ... to like the suspect in your case.

Trenton Bricken 28:06 Yeah. Yeah.

Sholto Douglas 28:07 Um, and that, yeah, doesn't feel at all outlandish.

Dwarkesh Patel 28:10 Do the low lens, lens on like (laughs) -

Trenton Bricken 28:14 Yeah.

Sholto Douglas 28:14 ... suspects.

Dwarkesh Patel 28:14 (laughs)

Trenton Bricken 28:14 Well, well, something I think that, that people who aren't doing this research, um, can overlook is after your first layer of the model-

Sholto Douglas 28:22 Yeah.

Trenton Bricken 28:22 ... every carry, query, key and value that you're using for attention comes from the combination of all the previous tokens. So like my first layer, I'll query my previous tokens and just extract information from them. But all of a sudden let's say that I attended to tokens one, two, and four in equal amounts.

Sholto Douglas 28:39 Yeah.

Trenton Bricken 28:39 Then the vector in my residual stream, a- assuming that they just, that they wrote out the same thing to the value vectors, but, but ignore that for a second, um, is a third of each of those. And so when I'm querying in the future, my query is actually a third of, of each of those things. And so-

Sholto Douglas 28:54 But they might be written to different subspaces.

Trenton Bricken 28:56 That's, that's right. Hypothetically, but they, they wouldn't have to.

Sholto Douglas 28:59 They wouldn't have to. Yeah.

Trenton Bricken 28:59 And so you can, you can recombine and immediately even by layer two and certainly by the deeper layers, just have like these very rich vectors that are packing in a ton of information and the, and the causal graph is like literally over every single layer that happened in the past.

Sholto Douglas 29:14 Okay.

Trenton Bricken 29:14 And, and that's what you're operating on. Um, yeah.

Sholto Douglas 29:17 It does bring to mind like a very funny eval to do would be like a Sherlock Holmes eval. Let's say you put the entire book into context and then you have like a sentence which is like, "The suspect is like X." Then you have like a logic probability distribution over like the different characters in the book.

Trenton Bricken 29:29 Yeah, yeah.

Dwarkesh Patel 29:30 Yeah.

Sholto Douglas 29:30 Um, and then like as you put more like book in-

Dwarkesh Patel 29:32 That would be super cool. Yeah, yeah, yeah.

Trenton Bricken 29:34 Yeah, yeah.

Sholto Douglas 29:34 It would be super interesting. I wonder if you'd get anything at all.

Dwarkesh Patel 29:37 (laughs)

Sholto Douglas 29:37 But it'd be cool.

Dwarkesh Patel 29:38 Um, the Sherlock Holmes is probably already in the training data.

Sholto Douglas 29:40 Yeah.

Trenton Bricken 29:40 Right, right.

Dwarkesh Patel 29:41 We've got to get like a mystery novel that was written in the-

Trenton Bricken 29:43 You could get an LLM to write it.

Dwarkesh Patel 29:44 (laughs)

Sholto Douglas 29:44 Or we could like, or you could purposely exclude it, right?

Dwarkesh Patel 29:47 Oh, we can? How do you...

Trenton Bricken 29:48 Well, you need to scrape any discussion of it from Reddit or any other thing. Right?

Sholto Douglas 29:51 Right. It's hard. Um, but that, that's like one of the challenges that goes into things like long context evals is to get a good one, you need to know that it's not in your training data, um, that you've like put in the effort to exclude it.

Dwarkesh Patel 30:01 What, um, so, uh, I-I, I, I actually want to... there's two different threads I want to follow up on. Let, let's go to the long context one and then we'll come back to, um, this. So, in the Gemini 1.5 paper-

Sholto Douglas 30:14 Yeah.

Dwarkesh Patel 30:14 ... the eval that was used was, can it like... wa- something with Pogram essays.

Sholto Douglas 30:19 Yeah.

Dwarkesh Patel 30:19 Can it like remember something from Pogram?

Sholto Douglas 30:19 Yeah, yeah the needle in a haystack thing.

Dwarkesh Patel 30:20 Right. Um, which yeah, I mean, there's like, the... we don't necessarily just care about its ability to recall one specific fact from the context.

Sholto Douglas 30:30 Yeah.

Dwarkesh Patel 30:30 I'll step back and ask the question, um, uh, like the loss function for these models is unsupervised, you don't have to like come up with these bespoke things that you keep out of the training data. You know, is there a way you can do a benchmark that's also unsupervised, where... I don't know, another LLM is rating it in some way or something like that? And maybe the answer is like, well if you could do this, like reinforcement learning would work, because then you have this like unsupervised...

Sholto Douglas 30:56 Yeah, I mean, I think people have explored that kind of stuff. Like, for example, Anthropic has the Constitutional RL paper-

Dwarkesh Patel 31:01 Yeah.

Sholto Douglas 31:01 ... where they take another language model-

Dwarkesh Patel 31:02 Yeah.

Sholto Douglas 31:02 ... and they point it and say, like, "H- how, you know, helpful or harmless was that, uh, response?" And then they get it to update and try and, you know, improve along the 3D frontier-

Dwarkesh Patel 31:11 Right.

Sholto Douglas 31:11 ... of helpfulness and harmfulness. Um, so y- you can like point language models at each other and create evils in this way. It's obviously an imperfect art form at the moment-

Dwarkesh Patel 31:19 Yeah.

Sholto Douglas 31:20 ... um, because, uh, you know, you get reward function hacking basically, um, and the language what like, uh, if you try and match up to what... even humans are imperfect here. Like, if you try and match up what humans will say, humans typically prefer longer answers which aren't necessarily better answers. And you get that same behavior with models.

Dwarkesh Patel 31:38 Mm. Um, on the other... okay, so the other thread, going back to the Sherlock Holmes thing. If it's all associations all the way down, this is a sort of like naive dinner party question, if I just like met you, oh, you're... I'm working on AI. Um, but okay, does that mean we should be less worried about superintelligence? Because there's not this sense in which it's like Sherlock Holmes plus plus. It'll still need to just like find these associations like humans find associations and like... You know what I mean? It's not just like, it, it sees a frame of the world and it's like figured out all the laws of physics.

Trenton Bricken 32:11 So, so for me, because this is, this is a very legitimate response, right? It's like, well, artificial general intelligence aren't... if you say humans are generally intelligent, then they're no more capable or competent. I, I'm just worried that you have that level of general intelligence in Silicon, where you can then immediately clone hundreds of thousands of agents and they don't need to sleep and they can have super long context windows and then they can start recursively improving and then things get really scary. Um, so, so I think to answer your original question, yes, you're right, they would still need to learn associations, but-

Dwarkesh Patel 32:43 Wait, wait, but the recursive self-improvement would still have to be them... Like, if intelligence is fundamentally about these associations, like the improvement is just them getting better at association, there's not like another thing that's happening and... So then it seems like you might disagree with the intuition that, well, they can't be that much more powerful if they're just doing associations.

Trenton Bricken 33:02 Well, I think then you can get into really interesting cases of meta-learning. Um, like when you play a new video game or like study a new textbook, uh, you're bringing a whole bunch of skills to the table to form those associations much more quickly. And like, because everything in some way ties back to the physical world, I think there are like general features that you can pick up and then, and then apply in novel circumstances. Um...

Dwarkesh Patel 33:26 Should we talk

Dwarkesh Patel 33:27 about intell- intelligence explosion then? I don't know if-

Sholto Douglas 33:28 Okay. (laughs)

Trenton Bricken 33:29 That's a good lead in.

Dwarkesh Patel 33:29 That was a totally good segue.

Sholto Douglas 33:30 Let's jump in. (laughs)

Trenton Bricken 33:31 (laughs)

Dwarkesh Patel 33:31 Yeah. I mentioned multiple agents and I'm like, "Oh, here we go."

Sholto Douglas 33:35 (laughs)

Dwarkesh Patel 33:35 Okay. So, um, the reason I'm interested in discussing this is, with you guys in particular, is the models we have of the intelligence explosion so far come from economists. Which is fine, but I think we can do better because the very... like the... in the model of the intelligent explosion, what happens is you replace the AI researchers and then there's like a, a bunch of A- automated AI researchers who can speed up progress, make more AI researchers, make further progress. And so I feel like if that's the metric or that's the mechanism, we should just ask the AI researchers about whether they think this is plausible.

Sholto Douglas 34:11 Yeah.

Dwarkesh Patel 34:11 So let me just ask you, like if I have a thousand Asian Shotas or Asian Trentons, are they just... do you think that you get an intelligence explosion? Is that... Yeah, wh- what does that look like to you?

Sholto Douglas 34:23 I think one of the important bounding constraints here is compute. Um, like I do think you could dramatically speed up AI research, right? Uh, like it seems very clear to me that in the next couple of years we'll have things that can do many of the software engineering tasks that I do on a day-to-day basis, um, and therefore dramatically speed up my work, um, and therefore speed up like the rate of progress, right? Um, at the moment, I think most of the labs are somewhat compute bound in that they, they're always... there are more experiments you could run and more pieces of information that you could gain in the same way that like scientific research on biology is also somewhat experimentally, uh, like throughput bound, like you need to be able to run and culture the cells in order to get the information. I think that will be at least a short term planning constraint. Obviously, you know, Sam's trying to raise seven trillion dollars to run, to buy...

Dwarkesh Patel 35:13 (laughs)

Sholto Douglas 35:13 ... get chips and so, um, like it does seem like there's going to be a lot more compute in future as everyone is heavily ramping. You know, NVIDIA's stock price sort of represents the relative, uh, (laughs) compute increase. Um, but any thoughts?

Trenton Bricken 35:27 I think we need a few more nines of reliability, um, in order for it to really be useful and trustworthy.

Sholto Douglas 35:34 Yeah.

Trenton Bricken 35:34 Right now it's like... And, and just having context lengths that are super long and it's like very cheap to have. Uh, like if, if I'm working on our code base, um, it's really only small modules that I can get Claude to write for me right now. Um, but it's very plausible that within the next few years, um, it... or even sooner, uh, it can automate most of my task. The only other thing here that I will note is, uh, the research that at least, uh, our sub-team and interpretability is working on is so early stage, um, that you really have to be able to make sure everything is, is like done correctly in a bug-free way and contextualize the results with everything else in the model. And if something isn't going right-... be able to enumerate all of the possible things-

Sholto Douglas 36:25 Yeah.

Trenton Bricken 36:25 ... and then, and then slowly work on those. Um, like an example that we've publicly talked about in previous papers is dealing with LayerNorm, right? And it's like, if I'm trying to get an early result or look at like the logit effects of the model, right? So it's like if I activate this feature that we've identified to a really large degree, how does that change the output of the model? Um, am I using LayerNorm or not? How is that changing the feature that's being learned? Um, there, there... Yeah, there... And, and that will take even more context or reasoning abilities for the model.

Sholto Douglas 36:55 I... So you used a couple of concepts together, and it's not self-evident to me that they're the same, but you... It seemed like you were using them inter- interchangeably. So I just want to... Um, like, uh, one was, well, to work on the Claude codebase and make more modules based on that, they need more context or something, where, like, it seems like they might already be able to fit in the context. Or do you mean like actual... Do you mean like the context window context or like more...

Trenton Bricken 37:21 Yeah, the context window context.

Sholto Douglas 37:22 Um, so yeah-

Trenton Bricken 37:23 Yeah.

Sholto Douglas 37:23 ... it seems like now it might just be able to fit. The, the thing that's preventing it from making good modules is not, uh, the lack of being able to put the codebase in there.

Trenton Bricken 37:31 I, I think that will be there soon. Yeah.

Sholto Douglas 37:32 But like, it's not going to be as good at you... as you at, like, coming up with papers because it can, like, fit the codebase in there.

Trenton Bricken 37:37 No, but it will speed up a lot of the engineering.

Sholto Douglas 37:40 Hmm. In a way that causes an intelligence explosion?

Trenton Bricken 37:43 Um, no. That accelerates research. But, but I think these things compound. So like, the faster I can do my engineering, the more experiments I can run. And then the more experiments I can run, the faster we can... I mean, my, my work isn't actually accelerating capabilities at all.

Sholto Douglas 37:57 Right, right. But just like-

Trenton Bricken 37:57 It's interpreting the models, but, but we have a lot more work to do on that.

Sholto Douglas 38:00 (laughs)

Trenton Bricken 38:00 Um, uh, surprise to the Twitter... (laughs)

Sholto Douglas 38:04 (laughs)

Trenton Bricken 38:04 Twitter guy. (laughs)

Sholto Douglas 38:05 Yeah, I mean for, for context though, like when you released your paper, there was a lot of talk on Twitter about-

Trenton Bricken 38:10 (laughs)

Sholto Douglas 38:10 ... "Alignment is solved, guys. Close the curtains." (laughs)

Trenton Bricken 38:15 (laughs) Yeah, yeah. No, it's... It keeps me up at night how quickly the models are becoming more capable and, like, just how poor our understanding still is of what's going on.

Sholto Douglas 38:24 Um, yeah. I, I guess, I'm still... Okay, so l- l- let's think through the specifics here. By the time this is happening, we have bigger models that are two to four orders of magnitude bigger, right? Uh, or at least in effective compute are two to four orders of magnitude bigger. And so, um, th- this, like, idea that, well, you can run experiments faster or something, you're having to retrain that model in this version of the intelligence explosion. Um, like if... The recursive self-improvement is different from what might have been imagined 20 years ago where you just rewrite the code. You actually have to train a new model, and that's really expensive. Not only now, but especially in the future as you keep like making these models orders of magnitude bigger. Doesn't that dampen the possibility of a sort of recursive self-improvement type intelligence explosion? It's definitely gonna act as a breaking mechanism.

Trenton Bricken 39:19 Mm-hmm.

Sholto Douglas 39:20 Um, like def- like... I agree that the world of, like, what we're making today looks very different to what people imagined it would look like 20 years ago. Like, it's not going to be able to write the same code to be like really smart, um, because actually it needs to train itself. Like the, the code itself is typically quite simple. Um, typically fairly small and self-contained. Um, I think John Carmack had this nice phrase where it's like, it's like the first time in history where like you can actually plausibly imagine writing AI with like 10,000 lines of code. Um, and that, like, actually does seem plausible if... when you pare most training codebases down, uh, to the limit. Uh, but it doesn't take away from the fact that this is something we should really strive to measure and estimate, like how progress might occur. Like, we should be trying very, very hard right now to, uh, measure exactly how much of a software engineer's job is automatable and what the trend line looks like, um, and be trying our hardest to project out those trend lines.

Trenton Bricken 40:12 But, but with all due respect to software engineers, like you are not like writing a React front end, right?

Sholto Douglas 40:18 Right.

Trenton Bricken 40:18 So it's like I- I don't know how this... Like, what is concretely happening? And maybe you can walk me through... Walk me through like a day in the life of Sho... Like you're working on an experiment-

Sholto Douglas 40:29 Yeah.

Trenton Bricken 40:29 ... or project that's going to make the model "better".

Sholto Douglas 40:32 Right.

Trenton Bricken 40:33 Like, what is happening from observation, to experiment, to theory, to like writing the code? What is happening?

Sholto Douglas 40:39 Um, and so I think important to contextualize here is that like I've primarily worked on inference so far. So a lot of what I've been doing is just taking, um, or helping guide the pre-training process, um, such that we design a good model for inference-

Trenton Bricken 40:52 Yeah.

Sholto Douglas 40:52 ... and then making the model, and like the surrounding system faster. I've also done some pre-training work around that, but it hasn't been like my 100% focus. But I can still describe what I do when I do that work.

Trenton Bricken 41:00 I know, but sorry, let me interrupt and say-

Sholto Douglas 41:02 So there's like two types of work. Yeah.

Trenton Bricken 41:03 In Carl Shulman's in- uh, like when he was talking about it on the podcast, he did say that things like improving inference or even literally having like better ma- help, helping it make you... help make better chips or GPUs, that's like part of the intelligence explosion.

Sholto Douglas 41:16 Yeah. Yeah.

Trenton Bricken 41:17 Because like obviously if the inference code runs faster, like it- it happens better or faster or whatever.

Sholto Douglas 41:21 Right.

Trenton Bricken 41:21 Anyway, sorry, go ahead.

Sholto Douglas 41:22 Yeah. Um... Okay, so what does, what does concretely a day look like? Um, I think the most important, like, part to illustrate is this cycle of coming up with an idea, proving it out at different points in scale, um, and, uh, the... and like interpreting and understanding what goes wrong. And I think most people would be surprised to learn just how much goes into interpret- like interpreting and understanding-

Trenton Bricken 41:47 Mm-hmm.

Sholto Douglas 41:48 ... what goes, what goes wrong. Because the ideas... People have long lists of ideas that they want to try, and not every idea that you think should work will work, and trying to understand why that is is quite difficult, and like working out what exactly you need to do to interrogate it. Um, so, so much of it is like introspection about what's going on. It's not pumping out thousands and thousands and thousands of line of code. It's not, um... Like the difficulty in coming up with ideas even. I think many people have a long list of ideas that they want to try, but paring that down and shot calling under very imperfect information what the right ideas to explore further is really hard.

Trenton Bricken 42:23 Uh, uh, tell me more about...

Dwarkesh Patel 42:25 Wha- what do you mean by imperfect information? Are these early experiments? Are these-

Sholto Douglas 42:29 Yeah.

Dwarkesh Patel 42:29 ... like, what is the information that you're...

Sholto Douglas 42:31 Um, so, so Demis mentioned this in his podcast and also, like, you obviously, it's like, the GPT-4 paper where you have, like, scaling law increments. And you can see, like, in the GPT-4 paper, they have, like, a bunch of, like, dots, right? Where they say, "We can estimate the performance of our final model, like, using all of these dots." And there's a nice curve that, like, flows through them. And Demis mentioned, yeah, that, uh, we do this process of scaling up. Um, concretely, why does- why is that imperfect information? Is you never actually know if the trend will hold. For certain architectures, the trend has held really well.

Dwarkesh Patel 43:02 Oh.

Sholto Douglas 43:02 And for certain changes it's held really well. But that isn't always the case. And things which can help at smaller scales can actually hurt at larger scales.

Dwarkesh Patel 43:11 Interesting.

Sholto Douglas 43:11 Um, so making guesses based on what the trend lines look like and based on, like, your intuitive feeling of, "Okay, this- this is actually something that's gonna matter." Um, particularly for those ones which help at the small scale.

Dwarkesh Patel 43:26 Th- that's interesting to consider that for every chart you see in a release paper-

Sholto Douglas 43:30 Yeah.

Dwarkesh Patel 43:30 ... or technical report that shows that smooth curve-

Sholto Douglas 43:32 Yeah.

Dwarkesh Patel 43:32 ... there's a graveyard of, like-

Sholto Douglas 43:34 Exactly.

Dwarkesh Patel 43:34 ... first few runs and then it's, like, flat or something.

Sholto Douglas 43:36 Yeah, yeah. There's all these, like, other lines that go in, like, different directions-

Dwarkesh Patel 43:38 Yeah.

Sholto Douglas 43:38 ... or f- like, tail off and then-

Dwarkesh Patel 43:40 And you just, like, didn't pursue that.

Trenton Bricken 43:40 Yeah.

Sholto Douglas 43:40 That's-

Trenton Bricken 43:41 It, yeah, it's crazy. Both, like, as a grad student and then also here, like, the number of experiments that you have to run-

Sholto Douglas 43:46 Yeah.

Trenton Bricken 43:46 ... before getting, like, a meaningful result.

Sholto Douglas 43:48 I, um... Tell me, okay, so you... But, but presumably it's not just like you run it until it stops and then, like, let's go to the next thing. Um, there's some process by which to interpret the early data and also to look at your, like, you... I don't know. I could, like, put a Google Doc in front of you- Yeah.

Dwarkesh Patel 44:04 (laughs)

Sholto Douglas 44:04 And I'm pretty sure you could just, like, keep typing for a while on, like, different ideas you have. Um, and there's some bottleneck between that and just, like, making the models better immediately. Right. Um, yeah. Walk me through, like, what is the- what is the inference you're making from the- for- first early steps that makes you have better experiments and better ideas? I think one thing that I didn't fully convey before was that I- I think a lot of, like, good research comes from working backwards from the actual problems that you wanna solve. And there's a couple of, like, grand problems I suppose in, like, making the models better today, um, that you would identify as issues and then, like, work back from, "Okay, how could I, like, change it to achieve this?" Um, there's also a bunch of when you scale, you run into things and you wanna, like, fix behaviors or, uh, or, like, issues at scale. And that, like, informs a lot of the research for the next increment and this kind of stuff. Um, so concretely, the barrier is a little bit software engineering. Like, often having a code base that's large and, uh, sort of capable enough that it can support many people doing research at the same time makes it complex. If you're doing everything by yourself, your iteration pace is gonna be much faster. I've heard that, like, Alec Radford, for example, who, like, famously d- did much of the pioneering work at OpenAI, he, like, mostly works out of, like, a Jupyter Notebook and then-

Dwarkesh Patel 45:14 (laughs)

Sholto Douglas 45:14 ... like, has someone else who, like, writes and productionizes that code for him.

Dwarkesh Patel 45:18 (laughs)

Sholto Douglas 45:18 I don't know if that's true or not. Um, but, like, that kind of stuff, like, actually operating with other people makes it, the c- raises the complexity a lot, um, because eh- eh n- from natural reasons, like, uh, familiar to, like, every software engineer. Um, and then, uh, the- the inherent running those, like, running and launching those experiments is easy, but there's inherent time, like, slows downs induced by that. So you often want to be parallelizing multiple different streams 'cause-

Dwarkesh Patel 45:45 Mm-hmm.

Sholto Douglas 45:46 ... one, you can't, like, be totally focused on one thing necessarily. Um, you might not have, like, fast enough feedback cycles.

Dwarkesh Patel 45:51 Yeah.

Sholto Douglas 45:52 Um, and then intuiting what went wrong is actually really hard. Like, working out what... Like, this is, in many respects, the problem that the team that Trenton is on is trying to better understand is, like, what is going on inside these models? We have inferences and understanding and, like, head canon for why certain things work, but it's not an exact science. Um, and so you have to constantly be making guesses about why something might have happened, what experiment might reveal whether that is or isn't true, and that's probably the most complex part.

Dwarkesh Patel 46:19 Mm-hmm.

Sholto Douglas 46:19 Um, the performance work, by comparatively, is easier but harder in other respects. It's just a lot of low level and, like, difficult engineering work.

Dwarkesh Patel 46:28 Mm-hmm.

Trenton Bricken 46:29 Yeah. I- I agree with a lot of that. But even on the interpretability team, I mean, especially with Chris Olah leading it, there are just so many ideas that we want to test. And it's really just having the engineering skill, but I'll put engineering in quotes because a lot of it is research, to, like, very quickly iterate on an experiment, look at the results, interpret it, try the next thing, communicate them, um, and then just ruthlessly prioritizing what the highest priority things to do are.

Dwarkesh Patel 46:57 Mm-hmm.

Sholto Douglas 46:57 Um... And this is really important, like, the ruthless prioritization is something which I think separates, uh, a lot of, like, quality research from, um, research that doesn't n- necessarily succeed as much. We're in this funny field where, uh, so many of our theoretical, initial theoretical understanding is, like, broken down basically. Um, and so you need to have this simplicity bias and, like, ruthless prioritization over what's actually going wrong. And I think that's one of the things that separates the most effective people is they don't necessarily get, like, too attached to solving or using a given s- like, uh, solution that they're necessarily familiar with. Um, but rather they attack the problem directly. Um, you see this a lot, uh, in, like, maybe people come in with a specific academic background. They try and solve problems with that toolbox. Um, and the best people are people who expand the toolbox dramatically. They're, you know, they're running around in there and taking ideas from reinforcement learning, but also from optimization theory. And also they have a great understanding of systems and so they know what the sort of constraints that bound the problem are. And they're good engineers, they can iterate and try ideas fast. Like, by far the best researchers I've seen, uh, they all have the ability to try experiments really, really, really, really fast. Um, and that is, that's cycle time, at- at smaller scales. Cycle time separates people.

Trenton Bricken 48:11 I mean, m- machine learning research is just so empirical.

Sholto Douglas 48:14 Yeah.

Trenton Bricken 48:15 And, and this is honestly one reason why I think, uh, our solutions might end up looking more brain-like than otherwise.

Dwarkesh Patel 48:21 Mm-hmm.

Trenton Bricken 48:21 Uh, it's like even though we wouldn't want to admit it, the whole community is kind of doing, like, greedy evolutionary optimization over the landscape-

Dwarkesh Patel 48:30 (laughs)

Trenton Bricken 48:31 ... of, like, possible AI architectures and everything else. Uh, it's like no better than evolution and that's not even necessarily a slight against evolution. Like-

Dwarkesh Patel 48:37 That's such an interesting idea. I'm still confused on what will be the bottleneck for these... What would be, have to be true of an agent such that it's like sped up your research? So in the Alec Radford example you gave where he apparently already has the equivalent of like-

Sholto Douglas 48:53 Right.

Dwarkesh Patel 48:53 ... co-pilot for his Jupyter Notebook experiments, um, is it just that if he had enough of those, he would be a mo- dramatically faster researcher, and so you just need Alec Radford? So it's like you're not automating the humans, you're just making the most effective researchers who have great taste more effective and, like, running the experiments for them and so forth? Or like, um, that you're still working at the point where, at which the intelligence explosion is happening, you know what I mean? Like is, is that what you're saying or-

Sholto Douglas 49:18 Right. Um, and, and if that were like directly true, why can't we scale-

Dwarkesh Patel 49:23 Mm-hmm.

Sholto Douglas 49:23 ... our current research teams better-

Dwarkesh Patel 49:25 Right.

Sholto Douglas 49:25 ... for example, is an, I think an interesting question to ask.

Dwarkesh Patel 49:27 Yeah.

Sholto Douglas 49:27 Like why... If this work is so valuable-

Dwarkesh Patel 49:29 Yeah.

Sholto Douglas 49:29 ... why can't we take hundreds or thousands of, you know, people who are, like, they're definitely out there, um, and like scale our organizations-

Dwarkesh Patel 49:38 Yeah.

Sholto Douglas 49:38 ... better? Um, it's... (sighs) I think we are less, at the moment, bound by the sheer engineering work of, of making these things than we are by compute to run and get signal and, uh, and taste, i- in terms of what the actual, like, right thing to do with, and that making those difficult inferences on imperfect information. Um... For, for the Gemini team, 'cause I think for interpretability- Right. ... we actually really want to keep hiring talented engineers. Yeah. And I think that's a big bottleneck for us to just keep- Yeah. ... making a lot of progress. And, and like obviously, more pe- like more people is, like, better. Um, but I do think, like it's interesting to consider, I think like one of the biggest challenges that, uh, like I've thought a lot about is, how do we scale better? Like Google is an enormous organization. It has 200,000-ish people, right? Or like 100 and maybe 80,000 or something like that. Um, and one has to imagine if there were like ways of scaling out Gemini's research program to all those fantastically talented software engineers, um, this seems like a, a key advantage that y- you would wanna be able to take advantage of, you'd want to be able to use, but like how do you effectively do that? It's a very complex organizational problem.

Dwarkesh Patel 50:53 Mm. So compute and taste, that's interesting to think about because at least the compute part is not bottlenecked on more intelligence, it just bottlenecked on SAM 7 trillion or whatever, right?

Sholto Douglas 51:05 (laughs) Yeah, yeah.

Dwarkesh Patel 51:06 So if I gave you 10X the H100s to run your experiments, how much more effective a researcher are you?

Sholto Douglas 51:11 TPUs please. (laughs)

Dwarkesh Patel 51:13 (laughs) Uh, how much more effective a researcher are you?

Sholto Douglas 51:16 Uh, I think the Gemini program would probably be like maybe five times faster with 10 times more compute or something like that.

Dwarkesh Patel 51:26 So that's pretty good elasticity, uh, like 0.5.

Sholto Douglas 51:29 Yeah.

Dwarkesh Patel 51:29 Wait, that's insane.

Sholto Douglas 51:31 Yeah, I think like more compute would just like directly convert into progress.

Dwarkesh Patel 51:34 So you have some allo- um, some fixed size of compute and some of it goes to inference, uh, some of, I guess like, and also, um, like to clients-

Sholto Douglas 51:44 Yeah.

Dwarkesh Patel 51:44 ... of GCP.

Sholto Douglas 51:45 Yep.

Dwarkesh Patel 51:46 Some of it goes to-

Sholto Douglas 51:47 Usefulness.

Dwarkesh Patel 51:47 Huh? (laughs)

Sholto Douglas 51:48 (laughs)

Dwarkesh Patel 51:48 Some of it goes to training and there-

Sholto Douglas 51:51 Yeah.

Dwarkesh Patel 51:51 I guess as a fraction of it, some of it goes to running the experiments for the full model?

Sholto Douglas 51:55 Yeah, that's right.

Dwarkesh Patel 51:56 Shouldn't then the fraction who does experiments be higher given that you would just be like-

Sholto Douglas 52:01 Right and-

Dwarkesh Patel 52:01 If like the bottleneck is research and research is bottlenecked by compute...

Sholto Douglas 52:04 And so one of the strategic decisions that every pre-training team has to make is, like, exactly what amount of compute you allocate to your different training runs. Uh, just like to your, your research program versus like scaling the last best, I, like, you know, thing that you landed on. Um, and I think, uh, they, they're, like, they're all trying to arrive at like a sort of pre-optimal point here. One of the reasons why you need to still keep training big models is that you get information there that you don't get otherwise. Um, so scale has all these emergent properties, uh, which you want to understand better.

Dwarkesh Patel 52:40 Mm-hmm.

Sholto Douglas 52:40 And if you, like, are always doing research and never... Like, remember what I said before about, like, you, you're not sure what's gonna, like-

Dwarkesh Patel 52:46 Mm-hmm.

Sholto Douglas 52:47 ... fall off the curve, right?

Dwarkesh Patel 52:48 Yeah.

Sholto Douglas 52:49 Um, if you, like, keep doing research in this regime-

Dwarkesh Patel 52:51 Yeah.

Sholto Douglas 52:52 ... uh, and like keep on getting more and more compute efficient, you may never... You, you may have actually like gone off-

Dwarkesh Patel 52:59 Sure.

Sholto Douglas 52:59 ... the path that actually eventually scales. So you need to constantly be investing in doing big runs too at the frontier of, of what you sort of expect to work.

Dwarkesh Patel 53:08 Okay, so then tell me what it looks like to be in the world where AI has significantly sped up AI research 'cause from this it doesn't really sound like the AIs are going off and r- writing the code from scratch and that's leading to faster output. It sounds like they're really augmenting the top researchers in some way. Like, yeah, tell me concretely, are they doing the experiments? Are they coming up with the ideas? Are they just, like, evaluating the outputs of the experiments? What's happening?

Sholto Douglas 53:31 So I think there's, there's like two worlds you need to consider here. One is where AI has meaningfully sped up our ability to make algorithmic progress.

Dwarkesh Patel 53:38 Right.

Sholto Douglas 53:39 And one is where the output of the AI itself is the thing that's like the crucial ingredient towards, uh, like, model capability progress.

Dwarkesh Patel 53:47 Mm-hmm.

Sholto Douglas 53:47 And like specifically what I mean there is-

Dwarkesh Patel 53:49 Synthetic data.

Sholto Douglas 53:49 ... like synthetic data.

Dwarkesh Patel 53:50 Yeah.

Sholto Douglas 53:50 Right? Um, and in the first world where it's meaningfully speeding up algorithmic progress, I think a necessary component of that is more compute. The, and, and you've probably like reached this elasticity point where like AIs maybe at some point are easier to speed up and get onto context than yourself, uh, sorry, than other people. Um, and so AIs meaningfully speed up your work because they're like a fantastic co-pilot basically that helps you code, like, multiple times faster. Um, and that seems like actually quite reasonable. Uh, super long context, super smart model, um, it's onboarded immediately and you can like send them off, uh, and to, like-

Dwarkesh Patel 54:30 Right.

Sholto Douglas 54:30 ... complete sub-tasks and sub-goals for you. And that actually, like, feels very plausible but again we don't know because there are no great evals about that kind of thing. Um, like the best one is, as I said before, SuiteBench which-

Dwarkesh Patel 54:41 Oh, oh, oh, in that one, uh, somebody was mentioning to me, like the problem is that if you-When a human is trying to do a full request, they'll like type something out and they'll like run it and see if it works. And if it doesn't, they'll rewrite it.

Sholto Douglas 54:54 Yeah.

Dwarkesh Patel 54:54 None of this was part of the, the, um, the opportunities that the LLM was given when run, run on this. Like it just like output it and if it runs and like checks all the boxes then you know, it passed, right? So may- it might have been an unfair test in that way.

Sholto Douglas 55:07 So you can imagine that is, uh, like if you were able to use that, that would be a effective training source for having like the, the key thing that's missing from a lot of training data is, uh, is like the, the reasoning traces, right? And I think this would be if I wanted to try and automate a specific field with like job family, um, or like understand how, how like at risk of automation that is.

Dwarkesh Patel 55:35 Yeah.

Sholto Douglas 55:36 Then having reasoning traces feels to me like a really important part of that.

Dwarkesh Patel 55:42 Mm-hmm. There's so many thread... Yeah. Th- There are so many different threads in that I want to follow up on. Let's begin with the data versus, um, like, uh, yeah, compute thing of like-

Sholto Douglas 55:54 Yeah.

Dwarkesh Patel 55:54 ... is, is the output of these AIs the thing that's causing the intelligence explosion or something?

Sholto Douglas 55:58 Yeah.

Dwarkesh Patel 55:58 Um, people talk about how these models are really a reflection on their data.

Sholto Douglas 56:04 Yeah.

Dwarkesh Patel 56:04 I think there was a, I forgot his name, but there was a, there's a great blog by this OpenAI engineer and he was talking about at the end of the day, as these models get better and better, it just like, they're just going to be really effective pro- like maps of the dataset.

Sholto Douglas 56:19 Yeah.

Dwarkesh Patel 56:19 And so it's like at, at the end of the day, like you gotta stop thinking about architectures. It's like the most effective architecture is just like doing an amazing job of mapping the data.

Sholto Douglas 56:27 Right.

Dwarkesh Patel 56:27 Um, so that implies that future AI progress comes from the AI just making really awesome data, right? Like that, that you, you're mapping to it.

Sholto Douglas 56:36 I think that's clearly a very important part. Yeah.

Dwarkesh Patel 56:39 Yeah. That's really interesting. Um, does that look to you like, I don't know, like things that look like chain of thought or what, what do you imagine as these models get better, as these model get smarter, what does the synthetic data look like?

Sholto Douglas 56:51 When I think of really good data, uh, to me that, that reasons something which involved a lot of reasoning to create.

Dwarkesh Patel 56:57 Yeah.

Sholto Douglas 56:58 So in modeling that, um, is this similar to like Ilia's perspective on, on trying, on, on achieving like super intelligence via effectively like perfectly modeling the human textual output, right?

Dwarkesh Patel 57:09 Yeah.

Sholto Douglas 57:09 Um, but even in the near term, in order to model something like the archive papers or Wikipedia, you have to have an incredible amount of reasoning behind you in order to understand what next token might be-

Dwarkesh Patel 57:22 Yeah.

Sholto Douglas 57:22 ... uh, being, being output. Um, and so for me, what I imagine as good data is like mode- like data where you can similarly, uh, at least like where, where it had to do reasoning to produce something. And then like the trick of course is how do you verify that that reasoning was correct?

Dwarkesh Patel 57:38 Yeah.

Sholto Douglas 57:38 Um, and this is why you saw like DeepMind do that, uh, geometry, uh, like self, like the sort of like self play for geometry basically. Or like the sort of tree search for your geometry, 'cause geometry is a really, it's an easily formalizable, easily verifiable, uh, field. So you can, you can check if its reasoning was correct. Uh, and you can generate heaps of data of correct, uh, of like trick-

Dwarkesh Patel 57:58 Yeah.

Sholto Douglas 57:58 ... of verified geometric proofs, train on that, and you know that that's good data.

Dwarkesh Patel 58:02 Um, this is actually funny 'cause I had a conversation with Grant Sanderson-

Sholto Douglas 58:05 Yeah.

Dwarkesh Patel 58:05 ... like last year where we were debating this and I was like, "Fuck dude, by the time they get the goal of the Matholypiad, of course they're gonna automate all the jobs." Um.

Sholto Douglas 58:13 (laughs)

Dwarkesh Patel 58:14 (laughs) Yikes. Um, yeah, on the synthetic data thing, uh, I, one of the things I speculated about in my scaling post, which was heavily informed with discussions with you two. (laughs)

Sholto Douglas 58:27 (laughs)

Dwarkesh Patel 58:27 And you especially Sholto, was, um, you can think of like human evolution through the perspective of like, we get language and so we're like generating the synthetic data, which-

Sholto Douglas 58:37 Right.

Dwarkesh Patel 58:38 ... you know, like our copies are generating the synthetic data which we're trained on.

Sholto Douglas 58:41 Yeah.

Dwarkesh Patel 58:41 And it's like this really effective, uh, genetics cultural like co-evolutionary loop.

Sholto Douglas 58:45 A- And there's a verifier there too, right? Like there's the real world. You might generate a theory about-

Dwarkesh Patel 58:51 Yeah.

Sholto Douglas 58:51 ... you know, the gods cause the storms.

Dwarkesh Patel 58:53 Right.

Sholto Douglas 58:53 Right? And then like someone else finds cases where that isn't true and-

Dwarkesh Patel 58:56 Yeah.

Sholto Douglas 58:56 ... so you like know that that like that what sort of didn't match your verification function. And now like actually instead you have like some, uh, weather simulation which required a lot of reasoning to produce and like accurately matches reality. Uh, and like you can train on that as a, as a better model of the world. Like we are training on, on that-

Dwarkesh Patel 59:14 Yeah.

Sholto Douglas 59:14 ... and like stories and like scientific theories.

Dwarkesh Patel 59:16 Yeah.

Sholto Douglas 59:17 So to speak.

Dwarkesh Patel 59:17 Uh, um, I want to go back. I'm just remembering something you mentioned, uh, a little while ago of given how sort of like empirical ML is, it really is evolutionary process as resulting in better performance and not necessarily an individual coming up with a breakthrough in like a top-down way. Um, that has interesting implications. First being that there really is... People, when people like are concerned about capabilities increasing because more people are going into the field, I've somewhat been skeptical of that way of thinking. But from this perspective of just like more input, it really does... Yeah. It feels more like, oh, actually by like the fact that more people are going to ICML means that there's like faster progress towards GPT-5.

Sholto Douglas 1:00:04 Right. Yeah. You just have more genetic recombination-

Dwarkesh Patel 1:00:06 Right.

Sholto Douglas 1:00:06 ... and like shots on target.

Dwarkesh Patel 1:00:08 Yeah.

Sholto Douglas 1:00:08 And I mean, aren't all fields kind of like that? Like th- there's this sort of scientific frame rate of like discovery versus invention, right?

Dwarkesh Patel 1:00:16 Yeah.

Sholto Douglas 1:00:16 And discovery almost involves like whenever there's been a massive scientific breakthrough in the past, typically there are multiple people co-discovering that at like roughly the same time.

Dwarkesh Patel 1:00:26 Right.

Sholto Douglas 1:00:26 Um, and that feels to me at least a little bit like the mixing and trying of ideas.

Dwarkesh Patel 1:00:30 Yeah.

Sholto Douglas 1:00:30 Like you can't try an idea that's so far out of scope that you, you have no way of verifying it or with the tools you have available.

Dwarkesh Patel 1:00:37 Yeah. I think physics and math might be slightly different in this regard.

Sholto Douglas 1:00:41 In this regard. Yeah.

Dwarkesh Patel 1:00:41 Um, but especially for biology or any sort of wetware and to the extent we want to analogize-

Sholto Douglas 1:00:46 Yeah.

Dwarkesh Patel 1:00:46 ... neural networks here, it's just, it's comical how, how serendipitous a lot of the discoveries.

Sholto Douglas 1:00:50 Yeah.

Dwarkesh Patel 1:00:51 Like penicillin, for example. Um, another implication of this is...This idea that, like, AGI is just gonna come tomorrow, like somebody's just going to discover a new algorithm and we have AGI, that seems less plausible. Like, it will just be a matter of more and more ML researchers finding these marginal things that all add up together to make models better, right?

Sholto Douglas 1:01:10 Like, yeah. That feels like the correct story to me, yeah.

Trenton Bricken 1:01:14 E- especially while we're still hardware constrained.

Dwarkesh Patel 1:01:17 Right. Do you buy this, uh, narrow window framing of the intelligence explosion of you have to each ... You know, GPT-3 to GPT-4 is two ooms, orders of magnitude more compute, or at least more effective compute, um, in the sense that if you didn't have any algorithmic progress, it would have to be two orders of magnitude bigger, like the raw form to be as good. Um, do you buy the framing that given that you have to be two orders of magnitude bigger at every generation, if you don't get AGI by GPT-7, that can help you catapult in an intelligence explosion? Like you're kind of just fucked as far as, like, much smarter intelligences go and you're kind of stuck with GPT-7 level models for a long time, because at that point you're just like consuming significant fractions of the economy to make that model. And we just don't have the wherewithal to like make GPT-8.

Trenton Bricken 1:02:10 Th- this is the Carl Shulman sort of argument of like, we're going to race through the orders of magnitude in the near term, but then longer term it would- it would be harder.

Dwarkesh Patel 1:02:19 Um, I think, like he's probably talked about it a lot. But yeah, but do you buy- do buy that framing? Um...

Sholto Douglas 1:02:24 Yeah. I mean, I generally buy that increases in order of magnitude of compute by like in an absolute terms almost like diminishing returns on that capability. Right. Like we've seen over a couple of orders of magnitude models go from being unable to do anything to being able to do like huge amounts. And it feels to me like each incremental order of magnitude like is more nines of reliability of things and so unlocks things like agents. But at least at the moment I haven't seen like transformatively tr- like there's- it doesn't feel like reasoning improves like linearly, so to speak, but rather like somewhat sublinearly.

Dwarkesh Patel 1:02:55 That's actually a very bearish sign because one of the things we were- we were chatting with one of our friends and he made the point, um, that if you look at what new applications are unlocked by GPT-4 or relative to GPT 3.5, it's not clear that's like that much. Like GPT-3 can do perplexity or whatever. So if there is this diminishing increase in capabilities and um, and that increased cost exponentially more to get, that's actually a bearish sign on like what 4.5 will be able to do or five will unlock in terms of economic impact.

Sholto Douglas 1:03:28 That being said, for me the jump between 3.5 and four is like pretty huge. And so like even if I was like another 3.5 to 4 jump is like ridiculous, right? Like if you imagine five as being a 3.5 to 4 jump like straight off the bat in terms of like ability to do SATs and this kind of stuff. Yeah.

Dwarkesh Patel 1:03:44 If the LSAT performance was particularly striking.

Sholto Douglas 1:03:46 Exactly. You go from like, uh, you know, like very smart, like, from like, you know, not super smart to like very smart to, like, utter genius in the next generation instantly. And it doesn't at least like to me, feel like we're going to sort of jump to utter genius in the next generation, but it does feel like we'll get very smart plus lots of reliability and then like we'll see TBD what that continues to look like. Um...

Dwarkesh Patel 1:04:13 Um, will GOFAI be part of the intelligence explosion where, like you say, synthetic data, but like in fact it will be like it writing its own source code in some important way. There's an interesting paper that you can use diffusion to, like come up with model weights. Um, I don't know how like legit that was or whatever, but like, I don't know, something like that.

Trenton Bricken 1:04:32 Can you... So- so GOFAI is good old fashioned AI, right? And can you define that? Because when I hear it, I think like if else statements for like symbolic logic.

Sholto Douglas 1:04:41 Sure. (laughs)

Dwarkesh Patel 1:04:42 (laughs)

Sholto Douglas 1:04:43 Um, um, I actually want to make sure we like, don't li- like we, like, fully unpack the whole like model improvement increments.

Dwarkesh Patel 1:04:49 Yeah.

Sholto Douglas 1:04:49 Because I don't want people to come away with the perspective that like, actually this is super bearish and like models aren't going to get much better and stuff.

Dwarkesh Patel 1:04:55 Okay.

Sholto Douglas 1:04:55 More what I want to emphasize is like the jumps that we've seen so far are huge. Um, and even if those like continue on like a smaller scale, we're still in for extremely smart, uh, like very reliable agents like over the next couple of orders of magnitude. And so like we didn't sort of fully close the threat on the narrower window thing. Um, but when you, you think of like, let's say GPT-4 costs, I don't know, let's call it $100 million or whatever. Um, you have, what, the 1B run, the 10B run, the 100B run all seem very plausible by, uh, you know, private company standards. Um, and then the...

Trenton Bricken 1:05:32 You mean in terms of dollar.

Sholto Douglas 1:05:33 In terms of dollar amounts yeah. Um, and then you can also imagine even like a 1T run being part of like a national consortium or like a, you know, a national level, uh, thing, but much harder on the behalf of an individual company. But Sam is out there trying to raise $7 trillion, right? (laughs)

Dwarkesh Patel 1:05:49 (laughs) Like he's already preparing for, like, a whole order of magnitude more than the, uh...

Trenton Bricken 1:05:53 He shifted the Overton Window.

Sholto Douglas 1:05:54 He shifted the orders of magnitude here beyond the national level. Um, so I want to point out that, one, we have a lot more jumps. And even if those jumps are- are relatively smaller, that's still a pretty stark improvement in capability.

Trenton Bricken 1:06:10 Not only that, but if you believe claims that GPT-4 is around 1 trillion parameter count, uh, I mean the human brain is between 30 and 300,000,000,000,000 synapses. And so that's obviously not a 1 to 1 mapping. And- and we can debate the numbers, but, uh, it seems pretty plausible that we're below brain scale still.

Dwarkesh Patel 1:06:28 Uh, uh, so crucially, the point being that the algorithmic overhead is really high in the sense that and maybe this is something we should touch on explicitly of even if you can't keep dumping more compute beyond the models that cost $1,000,000,000,000 or something, the fact that the brain is so much more data efficient implies that if you can... We have the compute, if we had like the brain's algorithm to train, um, uh, train-

Trenton Bricken 1:06:55 Yeah.

Dwarkesh Patel 1:06:56 If you had, if you could like train as a sample efficient as humans train from birth, we could make the AGI.

Trenton Bricken 1:07:01 Yeah, but the sample efficiency stuff, I never know exactly how to think about it because...... obviously, a lot of things are, are hardwired in certain ways, right? And like the co-evolution of language and-

Dwarkesh Patel 1:07:10 Sure.

Trenton Bricken 1:07:10 ... like the brain structure. Um, so i- i- it's hard to say. Also, there are some results that, uh, if you make your model bigger, it becomes more sample efficient.

Dwarkesh Patel 1:07:19 Mm-hmm.

Sholto Douglas 1:07:19 Mm-hmm.

Trenton Bricken 1:07:20 Um-

Sholto Douglas 1:07:20 Yeah.

Trenton Bricken 1:07:20 ... and so...

Sholto Douglas 1:07:21 Yeah, yeah, the original scaling was paper.

Trenton Bricken 1:07:22 Right.

Sholto Douglas 1:07:22 You had that, right?

Trenton Bricken 1:07:23 Yeah.

Sholto Douglas 1:07:23 Like larger models are more

Narrator 1:07:24 (music plays)

Trenton Bricken 1:07:24 Right. So, so maybe that also just solves it. Um, like you don't have to be more data efficient, but if your model's bigger, then you also just are more data efficient, like...

Sholto Douglas 1:07:33 Yeah, maybe.

Dwarkesh Patel 1:07:33 Well, how do we think about... Yeah, how do, what is like the explanation on why that would be the case? Like w- a bigger model just sees the exact same data, at the end of seeing that data, it's learned more from it.

Dwarkesh Patel 1:07:44 Does it have much space to represent it?

Trenton Bricken 1:07:45 I mean, I mean, my, my like very naive take here would just be that like, like, like so, so one thing that the superposition hypothesis, that interpretability has pushed, uh, is that your model is dramatically under-parameterized. A- and that's typically not the narratives that deep learning has pursued, right? But if you're, if you're trying to train a model on like the entire internet and have it predict it with incredible fidelity, uh, you are in the under-parameterized regime. And you're having to compress a ton of things and take on a lot of noisy interference in doing so. And so having a bigger model, you can just have cleaner representations that you can work with.

Dwarkesh Patel 1:08:16 Yeah. C- uh, uh, for the audience, you should unpack why that, first of all, what superposition is and why that is the implication of superposition.

Trenton Bricken 1:08:23 Sure. Yeah, so the, the fundamental result, and this was before I joined Anthropic, but the paper's titled Toy Models of Superposition, finds that even for small models, if you are in a regime where your data is high dimensional and sparse, and by sparse, I mean any given data point doesn't appear very often, um, your model will learn a compression strategy, which we call superposition, so that it can pack more features of the world into it than it has parameters. And, um, so, so the sparsity here is like... and I think, I think both of these constraints apply to the real world and modeling internet data is, is a good enough proxy for that, of like there's only one door cache. Like there's only one shirt you're wearing, there's like this liquid death can here. And so these are all objects or features, and how you define a feature is tricky. Um, and so, so you're in a really high dimensional space 'cause there are so many of them-

Dwarkesh Patel 1:09:18 Right.

Trenton Bricken 1:09:18 ... and they appear very infrequently.

Dwarkesh Patel 1:09:20 Yeah.

Trenton Bricken 1:09:21 And, and in that regime, your model will learn compression. Um, to, to riff a little bit more on this, um, I, I, I think it's becoming increasingly clear. I will say I, I believe that the reason, um, networks are so hard to interpret is because in a large part this superposition. So if you take a model and you look at a given neuron in it, right? A given unit of computation, and you ask, "How is this neuron contributing to the output of the model when it fires?" And you look at the data that it fires for, it's very confusing. It'll be like 10% of every possible input or like Chinese, but also fish and trees and the word, the, a full stop in URLs, right? Um, but uh, the paper that we put out towards mono-semanticity last year shows that if you project the activations into a higher dimensional space and provide a sparsity penalty, so you can think of this as undoing the compression in the same way that you assumed your data was originally high dimensional and sparse, you return it to that high dimensional and sparse regime, you get out very clean features.

Dwarkesh Patel 1:10:22 Mm-hmm.

Trenton Bricken 1:10:22 And things all of a sudden start to make a lot more sense.

Dwarkesh Patel 1:10:26 Mm-hmm. Okay. Um, uh, uh, there's so many interesting threads there. Uh, the first thing I wanna ask is the, the thing you mentioned about these models are trained in a regime where they're over-parameterized. Isn't that when you have generalization, like grokking happens in that regime, right? So, um, like-

Sholto Douglas 1:10:50 Um, under-parameterized.

Dwarkesh Patel 1:10:50 ... isn't that what you want?

Trenton Bricken 1:10:51 So, so, so I would say the models were under-parameterized.

Dwarkesh Patel 1:10:53 Oh, I see what you're saying. Yeah.

Trenton Bricken 1:10:54 Yeah, yeah. Like, like typically people talk about deep learning as if the model is over-parameterized.

Dwarkesh Patel 1:10:58 Mm-hmm.

Trenton Bricken 1:10:58 Um, but, but actually the claim here is that they're dramatically under-parameterized-

Dwarkesh Patel 1:11:01 I see.

Trenton Bricken 1:11:01 ... given the complexity of the task that they're trying to perform.

Dwarkesh Patel 1:11:04 Okay. Um, another question. So the distilled models, like it, it... first of all, okay, so what is happening there? 'Cause earlier, the earlier claims we're talking about is, um, the smaller models are worse at learning than bigger models, but like GPT-4 Turbo, you could say, make the claim that actually GPT-4 Turbo is worse at reasoning style stuff than GPT-4, um, but probably knows the same facts, like the distillation got rid of like some of the reasoning things. Um-

Sholto Douglas 1:11:35 Do, do we have any evidence that GPT Turbo is a distilled version of four? It might just be new architecture.

Dwarkesh Patel 1:11:40 Oh, okay.

Sholto Douglas 1:11:40 Yeah.

Dwarkesh Patel 1:11:41 All right. I'm with you.

Sholto Douglas 1:11:41 Like it could just be like a faster, more efficient neuro architecture.

Dwarkesh Patel 1:11:44 Okay, interesting.

Sholto Douglas 1:11:45 So that's cheaper. Yeah.

Dwarkesh Patel 1:11:46 Um, y- what, what is the... ho- how do you like interpret what's happening in distillation? I think Grant had one of these questions on his website of why can't you train the distilled model directly? Why does it have to go through... and is it a, is the picture like you had to project it from this bigger space to a smaller space? How, how?

Trenton Bricken 1:12:03 Um, I mean, I think both models will still be using superposition. Um, but, but the, the claim here is that you get a very different model if you distill versus if you train from scratch.

Dwarkesh Patel 1:12:13 Yeah.

Trenton Bricken 1:12:14 Um, and, and, uh, it's just more efficient or