Adam Marblestone on Dwarkesh Patel: Why AI Uses Blunt Losses

Dwarkesh Patel 0:00 The big million-dollar question that I have that, um, I've been trying to get the answer to through all these interviews with AI researchers, how does the brain do it, right? Like, we're throwing way more data at these LLMs, and they still have a small fraction of the total capabilities that a human does. So what's going on?

Adam Marblestone 0:15 Yeah. I mean, this might be the quadrillion-dollar question-

Dwarkesh Patel 0:17 (laughs)

Adam Marblestone 0:17 ... or something like that. It's, it's, it's arguably, you could make an argument this is the most important, you know, question in science. I don't claim to know the answer. I, I also don't really think that the answer will necessarily come even from a lot of smart people thinking about it as much as they are. I, my, my overall, like, meta-level take is that we have to empower the field of neuroscience to just make neuroscience a, a more powerful, uh, field, technologically and other otherwise, to actually be able to crack a question like this. But maybe the, the way that we would think about this now with, like, modern AI, neural nets, deep learning, is that there are sort of these, these cer- certain key components of that. There's the architecture. Um, there's maybe hyperparameters of the architecture. How many layers do you have or sort of properties of that architecture? There is the learning algorithm itself. How do you train it? You know, backprop, gradient descent, um, is it something else? There is how is it initialized, okay? So if we take the learning part of the system, it still may have some initialization of, of the weights. Um, and then there are also cost functions. There's like, what is it being trained to do?

Dwarkesh Patel 1:28 Yeah.

Adam Marblestone 1:28 What's the reward signal? What are the loss functions? Supervision signals. My personal hunch within that framework is that the, the field has neglected, uh, the role of this very specific loss functions, very specific cost functions. Uh, machine learning tends to like mathematically simple loss functions, right? Predict the next token. Um, you know, cross-entropy. The, you know-

Dwarkesh Patel 1:53 Right.

Adam Marblestone 1:53 ... these, these, these, these, uh, s- simple kind of computer scientist loss functions. I think evolution may have built a lot of complexity into the loss functions. Actually, many different loss functions for different areas, turned on at different stages of development. A lot of Python code, basically, uh, generating, uh, a specific curriculum for what different parts of the brain need to learn. 'Cause evolution has seen many times what was successful and unsuccessful, and e- evolution could encode the knowledge of, of the learning curriculum. So, so in the, in the machine learning framework, maybe we can come back and we can talk about, yeah, where do the loss functions of the brain come from. Can that, can loss fun- different loss functions lead to different efficiency of learning?

Dwarkesh Patel 2:31 You know people say, like, the cortex has got the universal human learning algorithm, the special sauce-

Adam Marblestone 2:35 Right, right.

Dwarkesh Patel 2:36 ... that humans have. What's up with that?

Adam Marblestone 2:37 Well, this is a huge question. Uh, and we don't know. I've seen models where what the cortex, uh, you know, the cortex has typically this, like, six layered structure.

Dwarkesh Patel 2:45 Yeah.

Adam Marblestone 2:45 Layers in a slightly different sense than layers of a neural net.

Dwarkesh Patel 2:48 Yeah.

Adam Marblestone 2:48 It's like any one location in the cortex has six physical layers of tissue as you go in layers of the sheet, and then those areas then connect to each other, and that's more like the layers of a network. Um, I've seen versions of that where what you're trying to explain is actually just how does it approximate backprop.

Dwarkesh Patel 3:07 Yeah.

Adam Marblestone 3:07 And what is the cost function for that? What is the network being asked to do? If you were sort of, are trying to say it's something like backprop, is it doing backprop on next token prediction or is it doing backprop on-

Dwarkesh Patel 3:16 Right, exactly.

Adam Marblestone 3:17 ... uh, classifying images or, or what is it doing? Um, and, uh, no one, no one knows. (laughs) Um, but I think, I think one, one thought about it, one possibility about it is that, um, it's just this incredibly general, um, prediction engine. So, so any one area of cortex is just trying to predict any... basically, can it learn to predict any subset of all the variables it sees from any other subset? So, like, omnidirectional inference, um, or omnidirectional prediction. Um, whereas an LLM is just you see everything in the context window, and then it, it computes a very particular-

Dwarkesh Patel 3:56 Yeah.

Adam Marblestone 3:56 ... conditional probability, which is given all the last thousands of things, what is the very probabilities for all the, all the, the next token?

Dwarkesh Patel 4:03 Yeah.

Adam Marblestone 4:03 Um, but it would be weird for a large language model to say, you know, um, you know, the quick brown fox, blank, blank, the lazy dog, um, and fill in the middle.

Dwarkesh Patel 4:15 Yeah.

Adam Marblestone 4:15 Um, uh, versus do the next token. It, it, it, if it's, if it's doing just forward, it can learn how to do that stuff in this emergent level of in-context learning, but natively, it's just predicting the next token.

Dwarkesh Patel 4:27 Yeah.

Adam Marblestone 4:27 What if the cortex is just natively made so that it can, you know, any area of cortex can predict any pattern in any subset of its inputs given any other missing subset? Um, that is a little bit more like, quote-unquote, probabilistic AI. Um, I think a lot of things I'm saying, by the way, are extremely similar to, like, what Yann LeCun would say.

Dwarkesh Patel 4:48 Yeah.

Adam Marblestone 4:48 Um, he's really interested in these energy-based models. Um, and something like that is like the joint distribution of all the variables. What is the, what is the likelihood or unlikelihood of just any combination of variables? And if I, if I clamp some of them, I say, "Well, definitely these variables are in these states," then I can compute with probabilistic sampling, for example, I can compute, okay, conditioned on these being set in this state, what are... And these could be any arbitrary subset of, of, of variables in the model. Uh, can I predict what any other subset is gonna do and sample from any other subset given clamping this subset? And I could choose a totally different subset and sample from that subset. Um, so it's omnidirectional inference. And so, you know, i- that could be, there are some parts of er- of cortex that might be like association areas of cortex that may, you know, predict vision from audition.

Dwarkesh Patel 5:39 Yeah.

Adam Marblestone 5:39 There might be areas that predict things that the more innate part of the brain is gonna do. Because remember, this whole thing is basically riding on top of the sort of a lizard brain and lizard body, if you will. Um, and that thing is a thing that's worth predicting, too. So you're not just predicting do I see this or do I see that, but I, is this muscle about to tense? Am I about to have a reflex where I laugh?... you know, is my heart rate about to go up? Um, am I about to activate this instinctive behavior?

Dwarkesh Patel 6:06 Ba- based on my higher level of understanding of... Like, I can match, uh, somebody has told me there's a spider on my back-

Adam Marblestone 6:14 Right.

Dwarkesh Patel 6:14 ... to this lizard part that would activate if I was, like, literally seeing a spider in front of me.

Adam Marblestone 6:21 Yes.

Dwarkesh Patel 6:21 And that you- you learn to associate the two, so that even just from h- somebody hearing you say, "There's a spider on your back," you-

Adam Marblestone 6:25 Yeah. Let's, well, let's come back to this. And this, this is partly having to do with, with Steve Berne's theories, which I'm recently obsessed about.

Dwarkesh Patel 6:31 Yeah.

Adam Marblestone 6:32 But, but on your podcast with Ilya, um, he said, "Look, I'm not aware of any, any good theory of how evolution encodes high level desires or intentions." And I think this is, like, this is, like, very connected, uh, to, to, to all of these questions about the loss functions and the cost functions, um, that the brain would use, and it's a really profound question, right? Like, like, let's say that, um, I am embarrassed for saying the wrong thing on your podcast, because I'm imagining that Yann LeCun is listening-

Dwarkesh Patel 7:07 (laughs)

Adam Marblestone 7:07 ... and he says, "That's not my theory." (laughs) That you described energy-based models really badly. That's gonna enact- activate, in me, innate embarrassment and shame, and I'm gonna wanna go hide and, and whatever. And tha- that's gonna activate these innate reflexes. Um, and that's important, because I might otherwise get, get killed by Yann LeCun's-

Dwarkesh Patel 7:25 (laughs)

Adam Marblestone 7:25 ... you know, uh, marauding army of, of other-

Dwarkesh Patel 7:28 The French AI researcher's coming for you, Adam. (laughs)

Adam Marblestone 7:30 And so it's important that I have that instinctual response. But of course, evolution has never seen Yann, Yann LeCun or known about energy-based models or known the, what a, a, a, a, a, a important scientist or a podcast is. And so somehow, the brain has to encode this desire to, you know, uh, not, not piss off really important, you know, people in the tribe or something like this, um, in a very robust way, without knowing in advance all the things that the, the learning subsystem, okay, of the brain, the part that is learning, the cortex and other parts. Uh, the cortex is gonna learn this world model, and it's gonna include things like Yann LeCun (laughs) and, and podcasts. And, uh, evolution has to make sure that that, those neurons, whatever the Yann LeCun being upset with me neurons get properly wired up to the shame response, or this part of the reward function. Um, and this is important, right? 'Cause if we're gonna be able to seek status in the tribe or learn from knowledgeable people, as you said, or things like that, exchange knowledge and skills with friends but not with enemies. I mean, we have to learn all this stuff. So it has to be able to robustly wire these learned features of the world, um, learned parts of the world model up to, uh, these innate reward functions, and then actually use that to then learn more, right? 'Cause next time, I'm not gonna try to piss off Yann LeCun if he emails me that-

Dwarkesh Patel 8:47 (laughs)

Adam Marblestone 8:47 ... that I got this wrong. Um, and so, uh, well, I'm gonna do further learning based on that. So it, in constructing the reward function, it has to use, uh, learned information. But how can evolution... You know, evolution didn't know about Li- Yann LeCun, so how can, how can it, how can it do that? And so, uh, the basic idea, um, that Steve Berne is proposing is that, well, part of the cortex, uh, or, or other areas like the amygdala that learn, um, what they're doing is they're modeling the steering subsystem. The steering subsystem is the part with these more innate- innately programmed responses and the innate programming of these series of reward functions, cost functions, bootstrapping, uh, functions that exist. So there are parts of the amygdala, for example, that are able to monitor what, what those parts do and predict what those parts do. So, um, so how do you find the neurons, um, that are important for social status? Well, you have some innate heuristics of social status, for example. Or you have some innate, uh, innate, uh, heuristics of friendliness that, um, that the steering subsystem can use. And the steering subsystem has s- actually has its own sensory system, which is kind of crazy. So we think of, you know, vision as being something that the cortex does.

Dwarkesh Patel 9:54 Mm-hmm.

Adam Marblestone 9:54 But there's also a steering subsystem subcortical visual system called the superior colliculus with innate ability to detect faces, for example, or threats. Um, so it, and so th- there's a visual system that, uh, has innate heuristics, um, and that, the steering subsystem has its own responses. So there'll be part of the amygdala or part of the cortex that is learning to predict those responses. And so what are the neurons that are, that matter in the cortex for, um, social status or for friendship? Where they're the ones that predict those innate heuristics for friendship, right? So you train a predictor in the cortex, and you say, "Which neurons are part of the predictor?" Uh, those are the ones that are... Now it's, now we've actually managed to wire it up. Yeah.

Dwarkesh Patel 10:36 Th- this is fascinating. Um, I, I feel like I still don't understand. I understand how the cortex could learn how this primitive part of the brain would respond to... Um, so it can obviously, it has these labels on, here's literally a picture of a spider-

Adam Marblestone 10:55 Yeah.

Dwarkesh Patel 10:55 ... and this is bad. Like, be scared of this.

Adam Marblestone 10:56 Right.

Dwarkesh Patel 10:56 And then it, the cortex learns that this is bad, because the innate part t- tells it that.

Adam Marblestone 11:00 Yeah.

Dwarkesh Patel 11:01 But then it has to generalize to, okay, the spider's on my back-

Adam Marblestone 11:04 Yes.

Dwarkesh Patel 11:05 ... and somebody's telling me, "The spider's on your back," that's also bad.

Adam Marblestone 11:07 Yes.

Dwarkesh Patel 11:07 But it never got supervision on that.

Adam Marblestone 11:10 Right.

Dwarkesh Patel 11:10 So how does it...

Adam Marblestone 11:11 Well, it's because the learning subsystem, um, is a powerful learning algorithm that does have generalization, uh, that is capable of generalization. So the steering subsystem, these are the innate responses. So you're going to have some, let's say, um, built into your steering subsystem, uh, these lower brain areas, hypothalamus, brain stem, et cetera. Um, and again, they include, they have d- their own primitive sensory systems. So there may be a innate response, um, if I see something that's kind of moving fast toward my body that I didn't previously see was there and is kind of small and dark and high-contrast, that might be an insect kind of skittering onto my body. Um, I am going to, like, flinch, right?

Dwarkesh Patel 11:51 Yeah.

Adam Marblestone 11:52 Um, and so there are these innate responses. And so there's gonna be some group of neurons, let's say in the hypothalamus, that is the, "I am flinching."

Dwarkesh Patel 11:59 Yeah.

Adam Marblestone 12:00 Uh, or, "I just flinched."

Dwarkesh Patel 12:01 Right.

Adam Marblestone 12:01 Right? They're the, they're the, I, I just flinched, uh, neurons in the hypothalamus.Um, so when you flinch first of all, that negative contribution with the reward function, you didn't want that to happen perhaps. Um, but that's only happen... That's a reward function then that is, it doesn't have any generalization in it, so I'm going to avoid that exact situation of the thing skittering toward me.

Dwarkesh Patel 12:19 Yeah.

Adam Marblestone 12:20 Um, and maybe I'm going to avoid some actions that lead to the thing skittering. Uh, so that's, that's something, a generalization you can get, what Steve calls it is downstream of the reward function. Um, so I'm going to avoid the situation where the spider was skittering toward me. Um, but you're also going to do something else, so there's going to be, like, a part of, of your amygdala, say, that is saying, "Okay, um, a few, you know, a few milliseconds, you know, hundreds and hundreds of milliseconds or seconds earlier, um, could I have predicted that flinching response?" It's going to be s- it's going to be a group of neurons that is e- essentially a classifier-

Dwarkesh Patel 12:54 Mm-hmm.

Adam Marblestone 12:54 ... of, "Am I about to flinch?" Um, and I'm going to have classifiers for that for every important steering subsystem variable that evolution needs to take care of. Am I about to flinch? Am I talking to a friend? Should I laugh now? Is the friend high status? Whatever variables the hypothalamus brainstem contain, um, am I about to taste salt? Um, so that it's going to have, uh, all these variables, and for each one, it's going to have a predictor. It's going to train that predictor. Now, the predictor that it trains, that can have some generalization. And the reason it can have some generalization is because it just has a totally different input. So its input data might be things like the word spider, right? But the word spider can activate in all sorts of situations-

Dwarkesh Patel 13:30 Yeah.

Adam Marblestone 13:30 ... that lead to the world, word spider activating in your word m- world model. Um, so, you know, if you have a, a complex world model with really complex features, that inherently gives you some generalization. It's not just the thing skittering toward me. It's even the word spider, uh, or the concept of spider is going to cause that to trigger. And this predictor can learn that. So whatever spider neurons are in my world model, um, which could even be a book about spiders or somewhere, a room where there are spiders or whatever that is, um-

Dwarkesh Patel 13:58 The, the amount of heebie-jeebies that this, this conversation is eliciting-

Adam Marblestone 14:01 Yeah. (laughs)

Dwarkesh Patel 14:01 ... in the audience is like... (laughs)

Adam Marblestone 14:03 So now I'm activating your steering subsystem.

Dwarkesh Patel 14:05 (laughs)

Adam Marblestone 14:05 Your, your, uh, steering subsystem spider hypothalamus, uh, subgroup of neurons o- of, of skittering insects-

Dwarkesh Patel 14:11 (laughs)

Adam Marblestone 14:11 ... are activating based on these very abstract concepts in the conversation.

Dwarkesh Patel 14:13 If we keep going, I'm gonna have to put in a trigger warning. (laughs)

Adam Marblestone 14:16 That's because, that's because you learned this, and the, the b- because the cortex inherently has the ability to generalize, because it's just predicting based on these very abstract variables and all these integrated information that it has. Whereas the, the steering subsystem only can use whatever the superior colliculus and a few other sensors can spit out.

Dwarkesh Patel 14:32 Yeah.

Adam Marblestone 14:32 So...

Dwarkesh Patel 14:32 By the way, it's remarkable that the person who's made this, this connection between different pieces of neuroscience, Stephen Burns, like former physicist-

Adam Marblestone 14:38 Yeah.

Dwarkesh Patel 14:39 ... has for the last few years has been trying to synthesize.

Adam Marblestone 14:41 He's an AI safety researcher. He's just synthesizing. This comes back to the academic incentives thing.

Dwarkesh Patel 14:44 Right.

Adam Marblestone 14:44 I think that this is, it's, this is a little bit hard to say what is the exact next experiment? How am I going to publish a paper on this?

Dwarkesh Patel 14:50 Yeah.

Adam Marblestone 14:50 How am I going to train my grad student to do this? Very qu- very speculative. But there's a lot in the neuroscience literature, and Stephen has been able to pull this together. And I think that Steve has an answer to Ilya's question essentially.

Dwarkesh Patel 14:58 That's great, yeah.

Adam Marblestone 14:58 ... which is, which is how, how does the brain ultimately code for these higher level desires and link them up to the more primitive rewards? Yeah.

Dwarkesh Patel 15:05 Very naive question, but why can't we achieve this omnidirectional inference by just training the model to not just map from t- token to next token, but remove the masks-

Adam Marblestone 15:18 Right.

Dwarkesh Patel 15:19 ... in the training so that it maps every token to every token? Or, um, come up with more labels between video and audio and text so that it, uh, it's forced to map one to each one?

Adam Marblestone 15:30 I mean, that may be, that may be the way. So it's, it's not s- clear to me. Some people think that there's sort of a different way that it does probabilistic inference or a different learning algorithm that isn't backprop. Um, there might be, like, other ways of learning energy-based models or other things like that that you can imagine. But, uh, that is involved in being able to do this and that the brain has that. But I think there's a version of it where, you know, the, what the brain does is like crappy versions of backprop to learn to predict, you know, through a few layers. And that, yeah, it's k- it's kind of like a multimodal foundation model.

Dwarkesh Patel 16:04 Right.

Adam Marblestone 16:04 Yeah, so maybe the cortex is just kind of like a, a, certain kinds of foundation models there. You know, s- LLMs are maybe just predicting the next token, but, you know, vision models are, maybe are trained learning to fill in the blanks or reconstruct different pieces or combinations.

Dwarkesh Patel 16:17 Yeah.

Adam Marblestone 16:17 But h- but I think that, uh, it does it in an extremely flexible way. So it's, you know, if you train a model to just fill, to fill in this blank at the center, okay, that's great. But what if you didn't train it to s- to fill in this other blank over to the left? Um, then it doesn't know how to do that.

Dwarkesh Patel 16:30 Yeah.

Adam Marblestone 16:30 It's not part of its, like, uh, repertoire of predictions that are, like, more ties into the network. Whereas with a really powerful inference system, you could choose that test time. You know, what is the, the sub, you know, the s- the s- the subset of variables-

Dwarkesh Patel 16:44 I see.

Adam Marblestone 16:44 ... it needs to infer and what, which ones are clamped.

Dwarkesh Patel 16:46 Okay, s- two sub-questions. One, I, wha- m- it makes me wonder whether the thing that is lacking in artificial neural networks is less about the reward function and more about the encoder or the embedding, which, um, like maybe the issue is that you're not representing video and audio and text in the right latent abstraction-

Adam Marblestone 17:11 Mm-hmm.

Dwarkesh Patel 17:11 ... such that they could intermingle and, um, conflict. Maybe this is also related to-

Adam Marblestone 17:17 Yeah.

Dwarkesh Patel 17:18 ... why LLMs seem bad at drawing connections between different ideas. Like, it's like are the ideas represented at a level of generality at which-

Adam Marblestone 17:24 Right.

Dwarkesh Patel 17:24 ... you could, which you could notice different connections?

Adam Marblestone 17:26 Well, the problem is these questions are all co-mingled. So if we don't know if it's doing a backprop-like learning and we don't know if it's doing energy-based models and we don't know how these areas are even connected in the first place, it's, like, very hard to, like, really get to the, the ground truth of this. But yeah, it's possible. I mean, I think that people have done some work. My friend Joel DiPello actually did something of, some years ago where, um, I think he put a model... I think it was a model of V1, um, of sort of specifically how the, the early visual cortex represents images, and put that as, like, an input into, like, a convnet, and that, like, improved something. So it could be, it could be, like, differences. The retina is also doing, you know, motion detection and certain things are kind of getting filtered out. So there, eh, there may be some pre-processing of the sensory data. There may be some clever combinations of which modalities are predicting which or so on that-... um, that lead to better representation. There may be much more clever things than that. Some people certainly do think that there's inductive biases built into the architecture that will shape the representations, you know, differently or that there are clever things that you can do. Um, so as Astera, which was the- the- the same organization that em- employs Steve Berens just launched this neuroscience project based on Dora So's, uh, work. And she has some ideas about how you can build, um, vision systems that basically require less training. They put in- they in-build into the assumptions of the design of the architecture, um, that, uh, things like objects are bounded by surfaces, and this- you know, surfaces have certain types of shapes and relationships of how they occlude each other and stuff like that. So y- it may be possible to build more assumptions into the network. Um, evolution may have also put some changes of architecture. Um, it's just I think that also the cost functions and so on may be a- a key- a key thing that it does.

Dwarkesh Patel 19:14 So, Andy Jones has this amazing 2021 paper, where he uses AlphaZero to show that you can trade off test time compute and training compute. And while that might seem obvious now, this was three years before people were talking about inference scaling. So this got me thinking, is there an experiment you could run today, even if it's a toy experiment, which would help you anticipate the next scaling paradigm? One idea I had was to see if there was anything to multi-agent scaling. Basically, if you have a fixed budget of training compute, are you gonna get the smartest agent by dumping all of it into training one single agent, or by splitting that compute up amongst a bunch of models, resulting in a diversity of strategies that get to play off each other? I didn't know how to turn this question into a concrete experiment though, so I started brainstorming with Gemini 3 Pro in the Gemini app. Gemini helped me think through a bunch of different judgment calls. For example, how do you turn the training loop from self-play to this kind of co-evolutionary leaked training? How do you initialize and then maintain diversity amongst different AlphaZero agents? How do you even split up the compute between these agents in the first place? I found this clean implementation of AlphaGo Zero, which I then forked and opened up in Antigravity, which is Google's agent-first IDE. The code was originally written in 2017, and it was meant to be trained on a single GPU of that time. But I needed to train multiple whole separate populations of AlphaZero agents, so I needed to speed things up. I rented a beefcake of a GPU node, but I needed to refactor the whole implementation to take advantage of all this scale and parallelism. Gemini suggested two different ways to parallelize self-play, one which would involve higher GPU contact switching, and the other would involve higher communication overhead. I wasn't sure which one to pick, so I just asked Gemini. And not only did it get both of them working in minutes, but it autonomously created and then ran a benchmark to see which one was best. It would have taken me a week to implement either one of these options. Think about how many judgment calls a software engineer working on an actually complex project has to make. If they have to spend weeks architecting some optimization or feature before they can see whether it will work out, they will just get to test out so many fewer ideas. Anyways, with all this help from Gemini, I actually ran the experiment and got some results. Now, please keep in mind that I'm running this experiment on an anemic budget of compute, and it's very possible I made some mistakes in implementation. But it looks like there can be gains from splitting up a fixed budget of training compute amongst multiple agents, rather than just dumping it all into one. Just to reiterate how surprising this is, the best agent in the population of 16 is getting 1/16 the amount of training compute as the agent trained on self-play alone, and yet it still outperforms the agent that is hogging all of the compute. The whole process of vibe coding this experiment with Gemini was really absorbing and fun. It gave me the chance to actually understand how AlphaZero works, and to understand the design space around decisions about the hyperparameters, and how search is done, and how you do this kind of co-evolutionary training, rather than getting bogged down in my very novice abilities as an engineer. Go to gemini.google.com to try it out.

Dwarkesh Patel 22:20 I want to talk about this idea that you just, uh, g- glanced off of, which was amortized inference. Um, and maybe I should try to explain what I think it means, 'cause I think it's probably wrong, and y- this- this will help you correct, or...

Adam Marblestone 22:36 It's been a few years for me too, so. (laughs)

Dwarkesh Patel 22:37 Okay. Um, right now the way the models work is you have an input, it maps it to an output. And this is amortizing a process that... th- the real process, which we think is, like, what intelligence is, which is, like, you have some prior over how the world could be. Like, what are the causes that make the wor- world the way that it is? And then, the way- when you see some observation, you should be like, "Okay, here's all the ways the world could be. Um, this cause explains what's happening best." Now, the- like, doing this calculation over every possible cause is computationally intractable, so then you would just have to sample, like, "Oh, here's a po- potential cause. Does this explain this o- observation? Uh, no. Forget it, let's- let's keep sampling."

Adam Marblestone 23:25 Right.

Dwarkesh Patel 23:25 And then eventually you get the cause. The cause then- the cause explains the observation, and then this becomes your posterior.

Adam Marblestone 23:33 That's actually pretty good, I think, of sort of- yeah. So- yeah, this- Bayesian inference, like, in general is, like, of this very intractable thing.

Dwarkesh Patel 23:39 Right.

Adam Marblestone 23:40 It- the algorithms that we have for doing that tend to require a- taking a lot of samples, Monte Carlo methods, taking a lot of samples.

Dwarkesh Patel 23:46 Yeah.

Adam Marblestone 23:46 And taking the samples takes time. I mean, this is like the original, like, Boltzmann machines and stuff were using-

Dwarkesh Patel 23:52 Yeah.

Adam Marblestone 23:52 ... techniques like this. Um, and still it's used with probabilistic programming, other types of methods often. And so- uh, yeah. So the Bayesian inference problem, which is, like, basically the problem of, like, perception, like given some model of the world and given some data, like, how should I update my-

Dwarkesh Patel 24:07 Right.

Adam Marblestone 24:07 ... or how sh- uh, what- what- what are the, like, the- the variables, you know, missing variables in my- in my internal model?

Dwarkesh Patel 24:13 And I guess the-

Adam Marblestone 24:14 Yeah.

Dwarkesh Patel 24:14 ... the idea is that neural networks are hopefully...Um, obviously, the n- there's mechanistically, the neural network is not starting with like, "Here is my model of the world, and I'm gonna try to explain this data." But the hope is that instead of starting with, um, "Hey, d- does this cause explain this observation? No. Here, does this cause explain this explanation?"

Adam Marblestone 24:34 Right.

Dwarkesh Patel 24:34 Yes. What you do is just like observation-

Adam Marblestone 24:36 What's the most, what's the cause that we, the neural net thinks is, is the best one, yeah.

Dwarkesh Patel 24:40 Yeah. Observation to cause, so the feedforward like goes observation to cause.

Adam Marblestone 24:43 Right. Observation to cause.

Dwarkesh Patel 24:44 To then output that like...

Adam Marblestone 24:45 Um, yes. You don't have to, you don't have to evaluate all these energy values or whatever-

Dwarkesh Patel 24:48 Right, right. Yeah.

Adam Marblestone 24:48 ... and, and sample around to make them higher and lower. Um, you just say, um, approximately that process would result in this being the top one or something like that.

Dwarkesh Patel 24:58 Exactly, yeah. One way to think about it might be that test time compute, inference time compute-

Adam Marblestone 25:03 Right.

Dwarkesh Patel 25:03 ... is actually doing this sampling again because it... you literally read its chain of thought. It's like actually doing this toy example we were talking about where it's like, "Oh, can I solve this problem by doing X? Nah."

Adam Marblestone 25:13 Yeah.

Dwarkesh Patel 25:14 "I need a different approach." And this raises the question. I mean, over time, it is the case that the capabilities which were, uh, which required inference time compute to elicit get distilled into the model, so you're amortizing the thing which previously you needed to do these like rollouts-

Adam Marblestone 25:30 Right.

Dwarkesh Patel 25:30 ... these like Monte Carlo rollouts to, um, to figure out. And so, in general, there... maybe there's this principle of digital minds which can be copied have different trade-offs which are relevant than biological minds which cannot.

Adam Marblestone 25:42 Mm-hmm.

Dwarkesh Patel 25:43 And so, in general, it should make sense to amortize more things because you can literally copy the, copy the amortization, right?

Adam Marblestone 25:48 Yes.

Dwarkesh Patel 25:48 Or copy the things that you have, um, sort of like built in.

Adam Marblestone 25:51 Yeah.

Dwarkesh Patel 25:51 Um, and it... May-... This is a tangential question where it might be interesting to speculate about in the future as these things become more intelligent and the way we train them becomes more economically rational, what will make sense to amortize into these minds which evolution-

Adam Marblestone 26:10 Mm.

Dwarkesh Patel 26:10 ... did not think it was worth amortizing into biological minds. They do it... You have to retrain every time.

Adam Marblestone 26:13 Right. I mean, first of all, I think the probabilistic AI people would be like, "Of course you need test time compute because this inference problem is really hard."

Dwarkesh Patel 26:21 Mm.

Adam Marblestone 26:21 And the only ways we know how to do it involve lots of test time compute. Otherwise, it's just a crappy approximation that's never going to like...

Dwarkesh Patel 26:26 Mm.

Adam Marblestone 26:26 You have to do infinite data or something to like make this. So, I think the... some of the probabilistic people will be like, "No, it's like inherently probabilistic and like amortizing it in this way like just doesn't make sense." And so... And they might then also point to the brain and say, "Okay, well, the brain, the neurons are kind of stochastic, and they're sampling, and they're doing, doing things." And so maybe the brain actually is doing more like the non-amortized inference, the real inference. Um, but it's also kind of strange how perception can work in just like milliseconds or whatever. It doesn't seem like it uses that much sampling. So, it's also clearly also doing some kind of, um, baking things into, into like approximate forward passes or something like that to do this. And yeah, so in the future, you know, I don't know. I mean, I think... Is it already a trend to some degree that things that are... people were having to use test time compute for are getting like used to train back the, the base model, right?

Dwarkesh Patel 27:20 Yeah.

Adam Marblestone 27:20 Yeah. That... So now it can do it in one pass.

Dwarkesh Patel 27:22 Right.

Adam Marblestone 27:23 Yeah. So I mean, I think... Yeah, you know, maybe evolution did or didn't do that. Uh, I think evolution still has to pass everything through the genome, right?

Dwarkesh Patel 27:33 Right.

Adam Marblestone 27:33 To build the network. So... And the environment in which humans are living is very dynamic, right? And so maybe that's... If we believe this is true that, that there's a learning subsystem per Steve Burns and a steering subsystem that the l- the learning subsystem doesn't have a lot of like pre-initialization or pre-training. Um, it has a certain architecture, but then within lifetime it learns. Um, then evolution didn't, you know, actually like amortize that much into that network.

Dwarkesh Patel 27:59 Right.

Adam Marblestone 27:59 It amortized it instead into a set of innate behaviors and a set of these bootstrapping cost functions-

Dwarkesh Patel 28:04 Yeah.

Adam Marblestone 28:04 ... or ways of building up very particular reward signals.

Dwarkesh Patel 28:08 Yeah.

Adam Marblestone 28:08 Yeah.

Dwarkesh Patel 28:08 Uh, this framework helps explain this, um, mystery that people have pointed out and I've asked a few guests about, which is h- h- um, if you want to analogize evolution to pre-training, well, how do you explain the fact that so little information is conveyed through the genome? So, three gigabytes is the size of the total human genome. Obviously, a small fraction of that is actually relevant to coding at the brain.

Adam Marblestone 28:31 Yeah.

Dwarkesh Patel 28:31 Um, and if previously people made this analogy that actually pre-... uh, evolution has found the hyperparameters of the model, the, the numbers which tell you how many layers should there be. The architecture basically, right?

Adam Marblestone 28:43 Yeah.

Dwarkesh Patel 28:43 Like what, what... How, how should things be wired together? But if a big part of the story that increases sample efficiency, aids learning, generally makes systems more performant is the reward function, is the loss function-

Adam Marblestone 28:55 Yeah.

Dwarkesh Patel 28:55 ... and if evolution found those loss functions which aid learning, then it actually kind of makes sense how so... you can like build an intelligence with so little information because like the reward function... and you're like, right, in Python, right? The reward function is like-

Adam Marblestone 29:09 Right.

Dwarkesh Patel 29:09 ... literally a line. Uh-

Adam Marblestone 29:11 Yes.

Dwarkesh Patel 29:11 And so you just like have like a thousand lines like this, and that's... doesn't take up that much space.

Adam Marblestone 29:15 Yes. And it also gets to do this generalization thing with the, the thing I, the thing I was describing where we were talking with about the spider, right? Of where it learns that just the word spider, you know-

Dwarkesh Patel 29:23 Yeah.

Adam Marblestone 29:23 ... triggers the spider, you know, reflex or whatever. Um, it gets to exploit that too, right? So it gets to build a reward function that actually has a bunch of generalization in it just by specifying these innate spider stuff and the thought assessors, as Steve calls them, that do the learning. Um, so that's like potentially a really compact solution, um, to building up these more complex reward functions too that you need. So it doesn't have to anticipate everything about the future of the reward function, just has to anticipate what variables are relevant and what are heuristics for like finding what those variables are. Um, and then, yeah, so then it has to have like a very compact specification for like the learning algorithm and basic architecture of the learning subsystem. And then it has to specify all this Python code of like all the stuff about the spiders and all the stuff about friends and all the stuff about your mother and all the stuff about mating and, um, and, and, and social groups and joint eye contact. It has to specify all that stuff. Um, and so is this really true? And so I think that there is some evidence for it. So, so, so Fei Chen and Evan Macosko and various other researchers who have been doing like these single cell atlases. So one of the things that, uh...... um, neuroscience technology or sort of scaling up neuroscience technology, again, this is kind of like my, one of my obsessions, um, has done, uh, through, through, um, the BRAIN Initiative, a big, you know, neuroscience funding program, is they've basically gone through different areas, pre- especially of the mouse brain and mapped, like, where are the different cell types? Um, how many different types of cells are there in different areas of cortex? Are they the same across different areas? And then you, then you look at these subcortical regions, which are more like the, like, steering subsystem or reward function generating regions. How many different types of cells do they have and which neurons types do they have? We don't know how they're all connected and exactly what they do or what the circuits are or what they mean. But you can just, like, quantify, like, how many different kinds of cells are there, um, with sequencing the RNA. And there are a lot more weird and diverse and bespoke cell types in the steering subsystem, basically, than there are in the learning subsystem. Like, the cortical cell types, there's enough to buil-... It seems like there's enough to build a learning algorithm up there and specify some hyperparameters. And in the, in the s- steering subsystem, there's, like, a gazillion, you know, thousands of really weird cells, which might be, like, the one for the spider flinch reflex and the one for "I'm about to taste salt" and the one-

Dwarkesh Patel 31:45 So why would, um, each reward function need a different cell type?

Adam Marblestone 31:49 Well, so this is where you get innately wired circuits, right? So in the, in the learning algorithm part, in this, in the learning, learning subsystem, um, you set up, specify the initial architecture, you specify a learning algorithm. It's all, all the, all the, all the juices is happening through plasticity of the synapses, changes of the synapses within that big network. But it's kind of like a relatively repeating architecture, um, how it's initialized. It's just, like, um, the amount of Python code needed to make, you know, a eight-layer transformer is not that different from-

Dwarkesh Patel 32:19 Right.

Adam Marblestone 32:19 ... one that makes a three-layer transformer, right? You're just replicating.

Dwarkesh Patel 32:22 Yeah.

Adam Marblestone 32:23 Whereas all this Python code for the reward function, you know, if superior colliculus sees something that's skittering and, and, you know-

Dwarkesh Patel 32:29 Yeah.

Adam Marblestone 32:29 ... you're feeling goosebumps on your skin or whatever, then triggers spider reflex. That's just a bunch of, like, bespoke, species-specific, uh, situation-specific crap that no-... The cortex doesn't know about spiders. It just knows about layers and-

Dwarkesh Patel 32:45 Right.

Adam Marblestone 32:45 ... and, and learning.

Dwarkesh Patel 32:45 But, but you're saying that the c-... The, the only way to have this, like, write this reward function-

Adam Marblestone 32:49 Yeah.

Dwarkesh Patel 32:49 ... is to have a special cell type?

Adam Marblestone 32:51 Yeah.

Dwarkesh Patel 32:51 Okay.

Adam Marblestone 32:52 Yeah. Well, I think so. I think you either have to have special cell types or you have to somehow, somehow otherwise get special wiring rules that evolution can say, "This neuron needs to wire to this neuron without any learning." And the way that that is most likely to happen, I think, is that those cells express, like, different receptors and proteins that say, "Okay, when this one comes in contact with this one, let's form a synapse."

Dwarkesh Patel 33:15 This-

Adam Marblestone 33:15 Um, so it's genetic wiring. Um, yeah.

Dwarkesh Patel 33:18 Uh-huh.

Adam Marblestone 33:18 And those need cell types to do it, yeah.

Dwarkesh Patel 33:20 I'm sure this would make a lot more sense if I knew 101 neuroscience, but, like, it seems like there's still a lot of complexity or generality rather in the steering subsystem. So if the steering sus- system has its own visual, uh-

Adam Marblestone 33:36 Yeah.

Dwarkesh Patel 33:36 ... system that's separate from the visual cortex-

Adam Marblestone 33:38 Yeah.

Dwarkesh Patel 33:39 ... different features still need to plug into that vision system in the... So, like, the spider thing needs to plug into it.

Adam Marblestone 33:47 Yes.

Dwarkesh Patel 33:47 And also the, um, the, uh, l- love thing needs to plug into it, et cetera, et cetera.

Adam Marblestone 33:54 Yes.

Dwarkesh Patel 33:55 So it seems complicated. Like I feel like we should-

Adam Marblestone 33:58 No, it's, it's still complicated, and that's, that's all the more reason why a lot of the genomic, you know, real estates in the genome and in terms of these different cell types and so on would go into wiring up the steering subsystem.

Dwarkesh Patel 34:09 And can we tell-

Adam Marblestone 34:09 Uh, pre-wiring it, yeah.

Dwarkesh Patel 34:10 Can we tell how much of the genome is, like, clearly working? So I guess you could tell how many are relevant to the producing the RNA that r- that manifest or the epigenetics-

Adam Marblestone 34:20 Right.

Dwarkesh Patel 34:20 ... that manifest in different cell types in the brain, right?

Adam Marblestone 34:22 Yeah, this is what the cell types helps you get at it. I don't think, I don't think it's exactly like, "Oh, this percent of the genome is doing this." But you could say, okay, in these, all these steering subsystems, subtypes, you know, how many different genes are involved in sort of specifying which is which and how they wire? Um, and how much genomic real estate do those genes take up, um, versus the ones that specify, you know, visual cortex versus audior- auditory cortex? You kind of are just reusing the same genes to do the-

Dwarkesh Patel 34:46 That's right.

Adam Marblestone 34:46 ... same thing twice. Whereas the spider reflex hooking up... Yes, you're right. They have to s-... They have to build a vision system, and they have to build some auditory systems and touch systems and navigation-type systems so, you know, the even feeding into the hippocampus and stuff like that. There's head direction cells. Even the fly brain has innate circuits-

Dwarkesh Patel 35:04 Yeah.

Adam Marblestone 35:05 ... that, you know, figure out its orientation and help it navigate in the world. And it uses vision, figures optical flow of, of how it's flying and, you know, uh, how is it, how is its flight related to the wind direction? It has all these innate stuff that I think we, in a mammal brain, we would all put that in, lump that into the steering subsystem. So there's a lot of work. So all the genes basically that go into specifying all the things a fly has to do, we're going to have stuff like that too, just all in the steering subsystem (laughs) .

Dwarkesh Patel 35:30 But do we, do we have some estimate of, like, here's how many nucleotides, here are how many megabases it takes to-

Adam Marblestone 35:37 Uh, I don't know. I mean, but, but, but, um, I mean, I think pe-... You might be able to talk to biologists about this-

Dwarkesh Patel 35:41 Interesting.

Adam Marblestone 35:42 ... you know, to, to some degree, because you can say, "Well, we just have a ton in common..." I mean, we have a lot in common with yeast from a genes perspective. Yeast is still used as a model-

Dwarkesh Patel 35:53 Yeah.

Adam Marblestone 35:53 ... for, you know, dr- some amount of drug development (laughs) and stuff like that-

Dwarkesh Patel 35:56 (laughs) .

Adam Marblestone 35:56 ... in biology. And so so much of the genome is just going towards you have a cell at all. It can recycle waste. It can get energy.

Dwarkesh Patel 36:03 Yeah.

Adam Marblestone 36:03 It can replicate. Um, and then it, then you, you know, what we have in common with a mouse. And so we ad-... We do know at some level that, you know, the differences between us and a chimpanzee or something, and that includes the social instincts and the more advanced, you know, differences in cortex and so on. Um, it's, it's a s-... It's a tiny number of genes that go into these additional amount of making the eight-layer transformer instead of the six-layer transformer or tweaking that reward function or, yeah.

Dwarkesh Patel 36:28 So th- this would help explain why the-... hominid brain exploded in size so fast, which is presumably, like, uh, eh, tell me if this is correct, but under the story, we've, um, social learning or some other thing increased the ability to learn from the environment, it like increased our sample efficiency, right?

Adam Marblestone 36:48 Yeah.

Dwarkesh Patel 36:48 Instead of having to go and kill the boar yourself and figure out, like, how to do that. You can just be like, "Uh, the elder told me this is how you make a spear." And then now it increases the-

Adam Marblestone 36:58 Right.

Dwarkesh Patel 36:58 ... incentive to have a bigger cortex which can learn these things.

Adam Marblestone 37:01 That's right. Yes. And that can be done with a relatively few genes-

Dwarkesh Patel 37:04 Right.

Adam Marblestone 37:04 ... because it's really, it's really replicating what the mouse-

Dwarkesh Patel 37:06 Like, 10x.

Adam Marblestone 37:06 ... already has, it's making more of it. And it's maybe not exactly the same, and there may be tweaks, but it's like, from our perspective, you don't have to reinvent-

Dwarkesh Patel 37:14 Right.

Adam Marblestone 37:15 ... all this stuff, right?

Dwarkesh Patel 37:16 And so then, um, how far back in the history of, of the evolution of the brain does the cortex go back? And is the idea that, like, the cortex has always out, ha- have figured out this omnidirectional inference thing, that- that's been a solved problem for a long time.

Adam Marblestone 37:29 Yeah.

Dwarkesh Patel 37:29 And then with the big unlock with primates is this, we got the reward function which increased the returns to having omnidirectional inference? Or is, is, is-

Adam Marblestone 37:37 That's a good question. Yeah.

Dwarkesh Patel 37:38 ... the cortex, is the cor- the omnidirectional inference also something that took a while to unlock?

Adam Marblestone 37:41 I'm not sure that there's agreement about that. I think there might be specific questions about language, you know, are there tweaks to be able, you know, whether that's through auditory and memory, some combination of auditory memory regions. There may also be, like, um, macro wiring, right? Of like, you need to wire auditory regions into memory regions or something like that, and into some of these social instincts to get-

Dwarkesh Patel 38:03 I see.

Adam Marblestone 38:03 ... language, for example, to happen. So there might be s- but that might be s- also a small number of gene changes-

Dwarkesh Patel 38:08 Yep.

Adam Marblestone 38:08 ... to be able to say, "Oh, I just need from my temporal lobe over here, going over to the auditory cortex," something, right? And there is some evidence for the, you know, the Broca's area, Wernicke's area, they're connected with these hippocampus and so on. And so prefrontal cortex. So there's, like, some small number of genes maybe for, like, ama- enabling humans to d- really properly do language.

Dwarkesh Patel 38:26 Mm-hmm.

Adam Marblestone 38:26 That could be a big one. But, yeah, I mean, I think that... Is it that something changed about the cortex and it became possible to do these things? Or is there, well, that potential was already there, but there wasn't the incentive to expand that capability and then use it, wire it to these social instincts and, and use it more.

Dwarkesh Patel 38:48 Mm.

Adam Marblestone 38:48 Um, I mean, I would lean somewhat toward the latter. I mean, I think a mouse i- has a lot of similarity in terms of cortex as a human.

Dwarkesh Patel 38:57 Right.

Adam Marblestone 38:57 Um...

Dwarkesh Patel 38:57 Although there's that, uh, the, Suzanne Herculewicz's work of the-

Adam Marblestone 39:02 Yeah.

Dwarkesh Patel 39:02 ... the, um, the, the number of neurons scales better with weight with primate brains than it does with rodent brains, right? So-

Adam Marblestone 39:11 Yeah. Mm-hmm.

Dwarkesh Patel 39:12 ... does that suggest that there actually was some improvement in the scalability of the cortex?

Adam Marblestone 39:16 Maybe, maybe. I'm not, I'm not super deep on this. There may, there may have been, yeah, changes in architecture, changes in the folding, changes in d- d- neuron properties and stuff-

Dwarkesh Patel 39:27 I think...

Adam Marblestone 39:27 ... that, that somehow slightly tweak this, but there's still a scaling.

Dwarkesh Patel 39:29 That's right. That's right, okay.

Adam Marblestone 39:30 Right? Either way, right? Um, and so I, I'm not saying there aren't something special about humans in the architecture of the learning subsystem at all, um, but yeah, I mean, it's, I think it's pretty widely thought that this has expanded, but then the question is, okay, well, how does that, how does that fit in also with the steering subsystem changes and i- the instincts that make use of this and allow you-

Dwarkesh Patel 39:51 Right.

Adam Marblestone 39:51 ... to bootstrap using this effectively? Um, but, uh, oh, I mean, just to save a m- a few other things, I mean, so even the fly brain has some amount of, for example, pro- prob- even, even very far back, um, the, I mean, I think you've read this, this great book, The Brief History of Intelligence, right?

Dwarkesh Patel 40:06 Mm-hmm.

Adam Marblestone 40:06 I think this is a really good book. Lots of AI researchers think this is a really good book, it seems like. Um, yeah, you have some amount of learning going back, um, all the way, uh, to anything that has a brain basically. Um, you have something kind of like primitive reinforcement learning at least, um, going back at least to, like, vertebrates. Like, imagine like a zebra fish is like a-

Dwarkesh Patel 40:32 Yeah.

Adam Marblestone 40:33 Um, and then there's kind of these other branches f- birds maybe kind of reinvented something kind of cortex-like, but it doesn't have the six layers.

Dwarkesh Patel 40:41 Mm-hmm.

Adam Marblestone 40:41 Um, but they have something a little bit cortex-like, um, so that, that s- some of those things, uh, after reptiles in some sense, birds and mammals both kind of made a s- a somewhat cortex-like, but differently organized thing. But even a fly brain has, like, associative learning centers that, um, actually do things that maybe look a little bit like this, like thought assessor concept from, from Behrens where there's, like, a specific dopamine signal to train specific subgroups of neurons in the fly mushroom body to associate different sensory information with, "Am I gonna get food now or am I gonna get-"

Dwarkesh Patel 41:13 Like, the future, yeah.

Adam Marblestone 41:13 ... "get hurt now," or, yeah.

Dwarkesh Patel 41:15 Brief tangent. I remember reading in, uh, one blog post that Bernd Millige wrote that the parts of the cortex which are associated with audio and vision have scaled disproportionately between other primates and humans, whereas the parts associated, say, with odor, have not.

Adam Marblestone 41:36 Mm-hmm.

Dwarkesh Patel 41:37 And I th- remember him saying something like, this is explained by that kind of data having worse scaling law properties, but I think the, and maybe he meant this, but another interpretation of actually what's happening there is that these social reward functions that are built into the steering subsystem needed to make use w- more of being able to see your elders and see what the visual cues are and hear what they're saying.

Adam Marblestone 42:05 Right. Yeah.

Dwarkesh Patel 42:05 And in order to make a sense of these cues which guide learning, you need it to activate these, um...

Adam Marblestone 42:11 Yeah.

Dwarkesh Patel 42:11 ... activate the vision and audio more than odor.

Adam Marblestone 42:14 I mean, there's all this stuff I feel like has come up in, in your sh- your shows before actually, but, like, even, like, the design of the human eye where you have, like, the pupil and the white and everything.

Dwarkesh Patel 42:24 Yeah.

Adam Marblestone 42:24 Like, we are designed to be able to establish relationships based on joint eye contact and, and s- and maybe this came up in the sudden episode.

Dwarkesh Patel 42:30 Mm-hmm.

Adam Marblestone 42:30 I can't remember, but, um, yeah, we're, we, we have to bootstrap to the point where we can detect eye contact and where we can communicate by language, right? And that's like what the, the first couple years of life are, are trying to do.

Dwarkesh Patel 42:42 Right.

Adam Marblestone 42:42 Yeah.

Dwarkesh Patel 42:42 Okay. Uh, I want to ask you about RL. So, um...Currently, the way these LLMs are trained, you know, they are, um... If, if they solve the unit test or solve a math problem, that whole trajectory, every token in that trajectory is upweighted. And what's going on with humans? Is there... Are there different types of model-based versus model-free that are happening in different parts of the brain?

Adam Marblestone 43:02 Yeah. I mean, this is, this is another one of these things. I, I mean, again, all my answers to these questions, a- and any specific thing I say is all just kind of like... Directionally, this is... We can kind of explore around this. I find this interesting. Maybe the li- I feel like the literature points in these directions in some very broad way. What I actually want to do is, like, go and map the entire mouse brain and, like, figure this out comprehensively and, like, make neuroscience the ground truth science. So I don't know, basically. (laughs) Um, but, uh, but yeah, I mean, there... So first of all, I mean, I think with Ilya on the podcast, I mean, he was like, "It's weird that we don't use value functions," right?

Dwarkesh Patel 43:33 Right.

Adam Marblestone 43:33 We use like the most dumbest form of RL based-

Dwarkesh Patel 43:36 Yeah.

Adam Marblestone 43:36 And of course, there are... These people are incredibly smart, and they're optimizing for how to do it on GPUs, and it's really incredible what they're achieving. But, like, conceptually, it's a really dumb form of RL, even compared to, like, what was being done in, like, 10 years ago, right? Like, even, uh, you know, the Atari game-playing stuff, right, was using, like, Q-learning, which is basically, like... It's a kind of temporal difference learning, right?

Dwarkesh Patel 43:56 Yep.

Adam Marblestone 43:56 And the temporal difference learning basically means you have some kind of a value function of, like, g- what action I choose now doesn't just tell me literally what happens immediately after this. It tells me, like, what is the long-run consequence of that for my expected, you know, total reward or something like that. Um, and so you would have value functions like... The fact that we don't have, like, value functions at all is like, in the LLMs, is like... It's crazy. It, I mean, I think it, I think because Ilya said it, I, I can say it, give him... I know, you know, one, 1/100th of what he does about AI, but, like, it's kind of crazy that this is working. (laughs)

Dwarkesh Patel 44:32 Yeah. (laughs)

Adam Marblestone 44:33 Um, but, uh, yeah, I mean, in terms of the brain, um, well, so I think there are some parts of the brain that are thought to do something that's very much like model-free RL, that sort of parts of the basal ganglia, um, sort of striatum and basal ganglia. They have, like, a, a certain finite, like... It is thought that they have a certain, like, finite relatively small action space, and the types of actions they could take, first of all, might be like, tell the spinal cord, or tell the brainstem and spinal cord to do this motor action: yes/no. Um, or it might be more complicated cognitive-type actions, like tell the thalamus to allow this part of the cortex to talk to this other part, or release the memory of this in the hippocampus and start a new one or something, right? There, there's... But there's some s- finite set of actions that kind of come out of the basal ganglia, and that is just a very simple RL. So, there are probably parts of other brains and our brain that are just like doing very simple, naïve-type RL algorithms. Um, layer one thing on top of that is that some of the major work in neuroscience, like Peter Dayan's work and a bunch, bunch of work that is part of why I think DeepMind did the temporal difference learning stuff in the first place, um, is they were very interested in neuroscience. Um, and there's a lot of neuroscience evidence that the dopamine is giving this reward prediction error signal, um, rather than just reward yes/no-

Dwarkesh Patel 45:57 Right.

Adam Marblestone 45:57 ... you know, a gazillion time steps in the future. It's a prediction error. Um, and that's consistent with, like, learning these value functions. Um, so there's that, and then there's maybe, like, higher order stuff. So, we have these cortex making this world model. Well, one of the things the cortex world model can contain is a model of when you do and don't get rewards, right? Again, it's predicting what the steering subsystem will do. It could be predicting what the basal ganglia will do. And so you have a model in your cortex that has more generalization and more concepts and all this stuff that says, "Okay, these types of plans, these types of actions will lead, in these types of circumstances, to rewards." So, I have a model of my reward. Um, some people also think that you can go the other way. And so this is part of the inference picture. There's this idea of RL as inference. Um, you could say, "Well, conditional on my having a high reward, sample a plan that I would have had to get there." That's inference of-

Dwarkesh Patel 46:54 Right.

Adam Marblestone 46:54 ... the plan part from the reward part. I'm clamping the reward as high and inferring-

Dwarkesh Patel 46:58 Yeah.

Adam Marblestone 46:59 ... the plan sampling from plans that could lead to that. Um, and so if you have this very general cortical thing, it can just do... If you have this, like, general, very general model-based system and the model, among other things, includes plans and rewards, then you just get it for free, basically. Yeah.

Dwarkesh Patel 47:14 I see. So, like, uh, in neural net- network l- uh, parlance, there's a value head associated to the, the, the omni-directional inference that's happening in the brain.

Adam Marblestone 47:23 Yes, yeah.

Dwarkesh Patel 47:24 I can see.

Adam Marblestone 47:24 Or there's a value input. Um, yeah.

Dwarkesh Patel 47:26 Oh, okay. Interesting.

Adam Marblestone 47:27 Yeah. And it can, and it can predict... One of, one of the, one of the almost sensory variables it can predict is, is what rewards it's going to get.

Dwarkesh Patel 47:34 Yeah. By, by the way, speaking of this thing about amortizing things, um, yeah, obviously value is like amortized rollouts-

Adam Marblestone 47:44 Right. Yeah.

Dwarkesh Patel 47:44 ... of looking up reward.

Adam Marblestone 47:46 Yeah. Something like that. Yeah.

Dwarkesh Patel 47:48 Yeah.

Adam Marblestone 47:48 Yeah. It's like a statistical a- average or prediction of it, yeah.

Dwarkesh Patel 47:52 Right. Tangential thought. Uh, you know, Joe, Joe Henrich and others have this idea that the way human societies have learned to do things is just like, how do you figure out that, you know, this kind of bean, which actually just almost always poisons you, is edible if you do this ten-step incredibly complicated process? Any one of which, if you fail at, the bean will be poisonous.

Adam Marblestone 48:16 Uh-huh.

Dwarkesh Patel 48:17 How do you figure out how to hunt the seal in this particular way with this, like, particular weapon at this particular time of the year, et cetera? Um, there's no way but, uh, just like trying shit over generations.

Adam Marblestone 48:28 Mm-hmm.

Dwarkesh Patel 48:30 And it strikes me this is actually very much like model-free RL happening at like a civilizational level. Um, no, not exactly 'cause like-

Adam Marblestone 48:36 I mean, evolution is the simplest algorithm-

Dwarkesh Patel 48:38 Right.

Adam Marblestone 48:38 ... in some sense, right? And if we believe that all of this can come from evolution, like the outer loop can be like extremely not foresighted, and yeah.

Dwarkesh Patel 48:45 Right.

Adam Marblestone 48:45 Yeah.

Dwarkesh Patel 48:45 Um, that, that, that's interesting. Just like, uh, hierarchies of...... evolution model for a cult- uh, evolution model for a-

Adam Marblestone 48:52 And so what does that tell you? Maybe that simple algorithms can just get you anything if you do it enough or Right, right, right, yeah. Yeah, I don't know. So...

Dwarkesh Patel 48:58 But yeah, so you- you have like maybe this, yeah, evolution model-free, basal ganglia model-free, cortex model-based-

Adam Marblestone 49:06 Mm-hmm.

Dwarkesh Patel 49:06 ... culture, uh, m- model-free potentially? Um, I mean there's like, you pay attention to your elders or whatever, so there's like-

Adam Marblestone 49:13 Yeah, maybe there's like group selection or whatever-

Dwarkesh Patel 49:15 Right, right, yeah.

Adam Marblestone 49:15 ... of these things as like more model-free, yeah. But now I think culture, well, it stores some of the model.

Dwarkesh Patel 49:24 Yeah.

Adam Marblestone 49:24 Right?

Dwarkesh Patel 49:25 So, let's say you want to train an agent to help you with something like processing loan applications. Training an agent to do this requires more than just giving the model access to the right tools, things like browsers and PDF readers and risk models. There's this level of tacit knowledge that you can only get by actually working in an industry. For example, certain loan applications will pass every single automated check despite being super risky. Every single individual part of the application might look safe, but experienced underwriters know to compare across documents to find subtle patterns that signal risk. Label Box has experts like this in whatever domain you're focused on, and they will set up highly realistic training environments that include whatever subtle nuances and watch-outs you need to look out for. Beyond just building the environment itself, Label Box provides all the scaffolding you need to capture training data for your agent. They give you the tools to grade agent performance and capture the video of each session, and to reset the entire environment to a clean state between every episode. So whatever domain you're working in, Label Box can help you train reliable, real-world agents. Learn more at labelbox.com/thorcash.

Dwarkesh Patel 50:32 Stepping back, how, um... is it a disadvantage or an advantage for humans that we get to use biological hardware, in comparison to computers as they exist now? So let, uh, by, what I mean by this question is like, if there's the algorithm, would the algorithm just qualitatively perform much worse or much better if, um, inscribed in the hardware of today? And the reason to think it might... like, here's what I mean, like, you know, obviously the brain has had to make a t- bunch of trade-offs which are not relevant to computing hardware. It has to be much more energetically efficient. Maybe as a result, it has to learn, uh, run on slower speeds, so that there can be-

Adam Marblestone 51:09 Right.

Dwarkesh Patel 51:09 ... a smaller voltage gap, and so the brain just runs at 200 hertz.

Adam Marblestone 51:13 Right.

Dwarkesh Patel 51:13 Um, and has to like run on 20 watts. On the other hand, may- you know, with like robotics, we have clearly experienced that fingers are way more nimble than we can make motors so far.

Adam Marblestone 51:23 Yeah.

Dwarkesh Patel 51:23 And so maybe there's something in the brain that is the equivalent of like cognitive, uh, dexterity-

Adam Marblestone 51:27 Right.

Dwarkesh Patel 51:27 ... which is like, maybe due to the fact that we can do unstructured sparsity, we can co-locate the memory and the compute.

Adam Marblestone 51:32 Yes.

Dwarkesh Patel 51:33 Where does this all net out? Are you like, "Fuck, we would be so much-"

Adam Marblestone 51:36 (laughs)

Dwarkesh Patel 51:36 "... smarter if we didn't have to deal with these brains," or are you like, "Oh..."?

Adam Marblestone 51:38 I mean, I think in the end we will get the best of both worlds-

Dwarkesh Patel 51:41 Right.

Adam Marblestone 51:41 ... somehow, right? I think th- I think an obvious downside of the brain is it cannot be copied. (laughs)

Dwarkesh Patel 51:45 Yeah.

Adam Marblestone 51:45 You don't have, you know, external read-write access to every neuron and synapse.

Dwarkesh Patel 51:50 Yeah.

Adam Marblestone 51:50 Whereas you do, I can just edit something in the weight matrix-

Dwarkesh Patel 51:52 Right.

Adam Marblestone 51:53 ... you know, in Python or whatever, (laughs) uh, it, you know, and- and load that up and copy that, um, in principle, right? Um, so the fact that it can't be copied and kind of random accessed is like very annoying. But otherwise, maybe these are, it like has a lot of advantages. So, or, and it also tells you that you want to like somehow do the co-design of the algorithm and the... it maybe that that even doesn't change it that much from all of what we discussed, but you want to somehow do this co-design. So, um, yeah, how do you do it with really slow, low-voltage switches? That's going to be really important for the energy consumption, the co-locating memory and compute. So like I- I think that probably just like hardware companies will try to co-locate memory and compute. They will try to use lower voltages, allow some stochastic stuff. There are some people that think that this like, all this probabilistic stuff that we were talking about, oh, oh, it's actually energy-based models and so on, is doing lo- it is doing lots of sampling. It's not just amortizing everything. That the neurons are also very natural for that, because they're naturally stochastic, and so you don't have to do a random number generator-

Dwarkesh Patel 52:58 Right.

Adam Marblestone 52:58 ... and a bunch of Python code basically to generate a sample. The neuron just generates samples, and it can tune what the different probabilities are.

Dwarkesh Patel 53:06 Yeah.

Adam Marblestone 53:06 And so, and like learn- learn those tunings.

Dwarkesh Patel 53:10 Yeah.

Adam Marblestone 53:10 And so it could be that it's very co-designed with like some kind of inference method or something, yeah.

Dwarkesh Patel 53:16 I- it'd be hilarious, I mean the- the (...) these interviews like, you know all these people that folks make fun of on Twitter, you know, Yann LeCun, Yann LeCun and Beth Jezos and whatever-

Adam Marblestone 53:25 They kind of...

Dwarkesh Patel 53:25 ... they're like-

Adam Marblestone 53:25 Oh no.

Dwarkesh Patel 53:26 ... no, like what, yeah, maybe, I don't know. (laughs)

Adam Marblestone 53:27 That is actua- that is actually one read of- of me-

Dwarkesh Patel 53:30 Maybe they got it right.

Adam Marblestone 53:30 ... and granted, you know, I- I haven't really worked on AI at all since LLMs, you know, took off.

Dwarkesh Patel 53:35 Mm-hmm.

Adam Marblestone 53:35 So I'm- I'm just like out of the loop, but, um, I- I'm surprised and I- I- I think it's amazing how the scaling is- is working and everything. But yeah, I think Yann LeCun and Beth Jezos are kind of onto something about the-

Dwarkesh Patel 53:46 (laughs) Good for them.

Adam Marblestone 53:46 ... about the probabilistic models, or at least po- possibly. And in fact, that's what, you know, all the neuroscientists and all the AI people thought, like until 2021 or something. (laughs)

Dwarkesh Patel 53:54 Right.

Adam Marblestone 53:54 Right?

Dwarkesh Patel 53:54 So, there's a bunch of cellular stuff happening in the brain that is not just about neuron to neuron synaptic connections. How much of that is functionally doing more work than the synapses themselves are doing versus it's just a bunch of kludge that you have to do in order to make the synaptic thing work? So, the way you need to, you know, with a digital mind, you can nudge the synapse, sorry, the parameter extremely easily.

Adam Marblestone 54:25 Yeah.

Dwarkesh Patel 54:25 But with a cell to modulate a synapse according to the gradient signal, it just takes-

Adam Marblestone 54:31 Right.

Dwarkesh Patel 54:31 ... so- all of this crazy machinery. So like-

Adam Marblestone 54:33 Yeah, yeah.

Dwarkesh Patel 54:33 ... is it actually doing more than it takes extremely little code to do and...

Adam Marblestone 54:36 So I don't know, but I'm- I'm not a believer in the like radical like, oh, actually memory is not synapses mostly, or like, learning is mostly genetic changes or something like that. I think it would just make a lot of sense, I think you put it really well, for it to be more like the second thing you said. Like let's say you want to do weight normalization across all the n- weights c-... coming out of your neuron, right, or into your neuron. Well, you probably have to go, like, somehow tell the nucleus about this, of the cell-

Dwarkesh Patel 55:04 Yeah.

Adam Marblestone 55:04 ... and then have that kind of send everything back out to the synapses or something, right? And so there's gonna be a lot of cellular changes, right? Or let's say that, you know, you just had a lot of plasticity, and, like, you're part of this memory, and now that's got consolidated into the cortex or whatever. And now we wanna reuse you as, like, a new one that can learn again. There's gonna be a ton of cellular changes.

Dwarkesh Patel 55:25 Yeah.

Adam Marblestone 55:25 So there's gonna be tons of stuff happening in the cell. But algorithmically, it's not really adding something beyond these algorithms, right? It's just implementing something that in a digital computer is very easy for us to go and just find the weights and change them.

Dwarkesh Patel 55:37 Hmm.

Adam Marblestone 55:37 Um, and it is a cell. It just literally has to do all this with molecular machines itself. (laughs)

Dwarkesh Patel 55:42 Yeah.

Adam Marblestone 55:42 Um, without any central controller, right? It's kind of incredible. Um, there are some things that cells do I think that, that seem, like, more convincing. So, in the cerebellum, so one of the things the cerebellum has to do is, like, predict over time, like predict what is the time delay? You know, let, let's say that, um, you know, I see a flash and then a s- uh, uh, you know, some number of milliseconds later, I'm gonna get, like, a puff of air in my eyelid or something, right? Uh, the cerebellum can be very good at predicting what's the timing between the flash and the air puff, so that now your eye will just, like, close automatically. Like, the cerebellum is, like, involved in that type of-

Dwarkesh Patel 56:16 Mm-hmm.

Adam Marblestone 56:17 ... reflex, like learned reflex. Um, and there are some cells in the cerebellum where it seems like the cell body is playing a role in storing that time constant, changing that time constant of delay, versus that all being somehow done with, like, I'm gonna make a longer ring of synapses to make that delay-

Dwarkesh Patel 56:37 Hmm.

Adam Marblestone 56:37 ... longer. It's like, no, the cell body will just, like, store that time delay for you. Um, so there are some examples, but I'm not a believer, like, out of the box in, like, essentially this theory that, like, what's happening is con- changes and connections between neurons.

Dwarkesh Patel 56:53 Yeah.

Adam Marblestone 56:54 And that's, like, the main algorithmic thing that's going on. Like, I think that's a very good reason to, to still believe that it's that rather than some, like, crazy cellular stuff.

Dwarkesh Patel 57:03 Yeah.

Adam Marblestone 57:03 Yeah.

Dwarkesh Patel 57:04 Going back to this whole perspective of, like, our, our intelligence is not just this omnidirectional inference thing that builds a world model, but really this system that teaches us what to pay attention to, what are the important salient factors to learn from, et cetera, I, I wanna see if there's some intuition we can drive from this about what different kinds of intelligence might be like.

Adam Marblestone 57:29 Hmm.

Dwarkesh Patel 57:29 So, it seems like AGI or superhuman intelligence should still have this, um, uh, uh, like, ability to learn a world model that's quite general.

Adam Marblestone 57:40 Yeah.

Dwarkesh Patel 57:40 But then it might, um, be incentivized to pay attention to different things-

Adam Marblestone 57:47 Yeah.

Dwarkesh Patel 57:47 ... that are relevant for what, you know, the m- the modern post-singularity environment. How, how different should we-

Adam Marblestone 57:55 Well-

Dwarkesh Patel 57:55 ... expect different intelligences to be basically?

Adam Marblestone 57:56 Yeah, I mean, I think one way of this question is, like, is it actually possible to, like, make the paperclip maximizer or whatever, right? If you ma- if you try to make the paperclip maximizer, does it end up, like, just not being smart or something like that?

Dwarkesh Patel 58:07 Hmm.

Adam Marblestone 58:07 Because it's, it was just a... The only reward function it had was, like, make paperclips.

Dwarkesh Patel 58:11 Interesting. Yeah, yeah.

Adam Marblestone 58:12 If I channel Steve Burns more, I mean, I think he's very concerned that the, the sort of minimum viable things in the steering subsystem that you need to get something smart is way less than the minimum viable set of things you need for it to have human-like social instincts and ethics and stuff like that. So, a lot of why you want to know about the steering subsystem is actually the specifics of how you do alignment essentially or what human behavior and social instincts and, is, versus just what you need for capabilities. And we talked about it in a slightly different way 'cause we were sort of saying, "Well, in order for humans to, like, learn socially, they need to make eye contact and learn from others." But we already know from LLMs, right, that depending on your starting point, you can learn language-

Dwarkesh Patel 58:52 Yeah.

Adam Marblestone 58:52 ... without that stuff, right? And so, yeah. And, and so I think that it probably is possible to make, like, super powerful, you know, model-based RL, you know, optimizing systems and stuff like that, that don't have most of what we have in the human brain reward functions, and as a consequence, might want to maximize paperclips, and that's a concern. Yeah.

Dwarkesh Patel 59:11 Right. But, but you were pointing out that in order to make a competent paperclip maximizer-

Adam Marblestone 59:17 Yeah.

Dwarkesh Patel 59:17 ... the kind of thing that can build the spaceships and learn the physics and whatever, um, it needs to have some drives w- which elicit learning, including-

Adam Marblestone 59:27 Yeah.

Dwarkesh Patel 59:27 ... say, curiosity and exploration.

Adam Marblestone 59:29 Yeah, curiosity and s- and interest in others, uh, of s-

Dwarkesh Patel 59:33 Yeah.

Adam Marblestone 59:33 ... interest in social interactions, curiosity, um, yeah. But, but that-

Dwarkesh Patel 59:37 (laughs)

Adam Marblestone 59:37 ... that's pretty, that's pretty minimal I think, and it, and it, that's true for humans.

Dwarkesh Patel 59:41 Right.

Adam Marblestone 59:41 But it might be less true for, like, something that's already pretrained as an LLM or something, right?

Dwarkesh Patel 59:44 I see.

Adam Marblestone 59:44 And so, uh, so most of why we want to know the steering subsystem I think, uh, if I'm channeling Steve, is alignment reasons. Yeah.

Dwarkesh Patel 59:51 Right. How, how confident are we that we even have the right algorithmic conceptual vocabulary to think about what the brain is doing? Uh, uh, and what I mean by this is, you know, there, there was one big contribution to AI from neuroscience which was this idea of the neuron-

Adam Marblestone 1:00:10 Yeah.

Dwarkesh Patel 1:00:10 ... which, like, and V-, you know, 1950s just, like, this original contribution. But then it seems like a lot of what we've learned afterwards about what the high-level algorithm the brain is implementing from the backprop to o- o- if there's something analogous-

Adam Marblestone 1:00:24 Yeah.

Dwarkesh Patel 1:00:24 ... to backprop happening in the, happening in the brain-

Adam Marblestone 1:00:26 Yeah.

Dwarkesh Patel 1:00:26 ... to always V1 doing something like CNNs-

Adam Marblestone 1:00:29 Right.

Dwarkesh Patel 1:00:29 ... to TD learning and Bellman equations, um, actor-critic, whatever-

Adam Marblestone 1:00:34 Yeah.

Dwarkesh Patel 1:00:34 ... seems inspired by what is, like, we come up with some idea, like, maybe we can make AI neural networks work this way-

Adam Marblestone 1:00:40 Yeah.

Dwarkesh Patel 1:00:40 ... and then we notice that s- something in the brain also works that way.

Adam Marblestone 1:00:44 Yes.

Dwarkesh Patel 1:00:44 So, why not think there's more things like this where in the future-

Adam Marblestone 1:00:46 Uh, there may be, yeah. I think the reason that I'm n- I think that we might be onto something is that, like, the AIs we're ma- making based on these ideas are working surprisingly well. Th- there's also a bunch of, like, just empirical stuff, like, like, convolutional neural nets and variants of convolutional neural nets, um...... I'm not for sure what the absolute latest-latest, but compared to other, like, models in computational neuroscience of, like, what the visual system is doing, are just, like, more predictive, right?

Dwarkesh Patel 1:01:13 Mm-hmm.

Adam Marblestone 1:01:13 So you can just, like, score, um, even, like, pretrained on, like, cat pictures and stuff, CNNs, what is the representational similarity that they have on some arbitrary other image versus, you know, compared to the brain activations, um, measured in different ways. Um, Jim DiCarlo's lab has a, like, brain score. And like, the AI model is actually like... The- the- it- there seems to be some relevance there, in terms of like even... Like, neurosciences don't necessarily have something better than that. So yes, I mean, that's just kind of recapitulating what you're saying, is that like the best computational neuroscience theories we have, seem to have been, like, invented-

Dwarkesh Patel 1:01:50 Right.

Adam Marblestone 1:01:50 ... part- largely as a result of AI models, um, and like find things that work. And so find backprop works and then say, "Can we approximate backprop with cortical circuits or something?"

Dwarkesh Patel 1:01:58 Yeah.

Adam Marblestone 1:01:58 And there's- there's kind of been things like that. Now, some people totally disagree with this, right? Um, so like Yuri Buzoki is a neuroscientist who has a book called The Brain From Inside Out, where he basically says like all our psychology concepts, like AI concepts, all this stuff is just like made up stuff. We actually have to do is like figure out what is the actual set of primitives that like the brain actually uses, and our vocabulary is not gonna be adequate to that. We have to start with the brain and make new vocabulary, rather than, say, backprop and then try to apply that to the brain or something like that. And, you know, he studies a lot of, like, oscillations and stuff in the brain, as opposed to individual neurons and what they do.

Dwarkesh Patel 1:02:34 Yeah.

Adam Marblestone 1:02:34 And, you know, I don't know. I- I think that there's a case to be made for that, and from a kind of research program design perspective, I think there's... Like, one thing we should be trying to do is just, like, simulate a tiny worm or a tiny zebra fish, um, like from almost, like, as biophysical or like as- as bottom-up as possible, like get connectome molecules activity, and like just study it as a physical, dynamical system and like look what it does. Um, but I don't know. I mean, just when I like... It just feels like the AI is really good fodder for computational neuroscience. Like, those might actually be pretty good models. We should look at that. Um, so I- I'm not a person who thinks that... I- I- I think I- I both think that there should be a part of the research portfolio that is like totally bottom-up and not trying to a- apply our vocabulary that we learn from AI onto these systems, and that there should be a b- another big part of this that's kind of trying to reverse engineer it using that vocabulary or variants of that vocabulary, um, and that we should just be pursuing both. And- and my guess is that the reverse engineering one is actually going to, like, kind of work-ish or something.

Dwarkesh Patel 1:03:45 Mm-hmm.

Adam Marblestone 1:03:45 Like, we do see things like TD learning, which, you know, Sutton also invented-

Dwarkesh Patel 1:03:49 Right.

Adam Marblestone 1:03:49 ... separately, right?

Dwarkesh Patel 1:03:50 That- that must be a crazy feeling to just like, yeah-

Adam Marblestone 1:03:53 Yes, it's- it's crazy, yeah.

Dwarkesh Patel 1:03:53 ... this- this, like, equation I wrote down is like in the brain.

Adam Marblestone 1:03:56 Yeah, and it seems like the dopamine is like doing-

Dwarkesh Patel 1:03:58 Yeah.

Adam Marblestone 1:03:58 ... some of that. Yeah.

Dwarkesh Patel 1:03:59 So I- l- let me ask you about this. Uh, you know, you- you guys are f- finding different groups that are trying to-

Adam Marblestone 1:04:05 Yeah.

Dwarkesh Patel 1:04:06 ... figure out what's up in the brain. If we had a perfect representation, however you define it, of the brain, why think it would actually f- let us figure out the answer to these questions? We have neural networks which are way more interpretable, not just because we understand what's in the weight matrices, but because there are weight matrices. There are these boxes with numbers in them.

Adam Marblestone 1:04:26 Right.

Dwarkesh Patel 1:04:27 And even then, we can tell very basic things. We can kind of see circuits for-

Adam Marblestone 1:04:33 Yeah.

Dwarkesh Patel 1:04:33 ... uh, very basic pattern matching or following one token with another.

Adam Marblestone 1:04:37 Right.

Dwarkesh Patel 1:04:38 I- I- I- I feel like we ha- we don't really have an explanation of why LLMs are intelligent, just because they're-

Adam Marblestone 1:04:42 Yeah. Well, I would- I would somewhat-

Dwarkesh Patel 1:04:43 ... y- yeah, they're interpretable.

Adam Marblestone 1:04:43 ... I would somewhat dispute it. I think we have some architectural... We have some description of what the LLM is, like, fundamentally doing. And what that's doing is that I have an architecture and I have a learning rule and I have hyperparameters and I have initialization and I have training data.

Dwarkesh Patel 1:04:57 But tha- th- those are things we learn from-

Adam Marblestone 1:04:59 Yeah.

Dwarkesh Patel 1:04:59 ... because we built them, not because we interpreted them-

Adam Marblestone 1:05:01 Yes, yes, yes, yes.

Dwarkesh Patel 1:05:01 ... from seeing the weights.

Adam Marblestone 1:05:02 We built them.

Dwarkesh Patel 1:05:03 Which is the ano- the analogous thing to connectome, is like seeing the weights.

Adam Marblestone 1:05:05 What I think we should do is we should describe the brain more in that language, of things like architectures, learning rules, initializations, rather than trying to find the Golden Gate Bridge circuit and saying exactly how does this neuron-

Dwarkesh Patel 1:05:15 Hmm.

Adam Marblestone 1:05:15 ... actually, you know... That's gonna be some incredibly complicated learned pattern. Um, yeah, Konrad Kording and Tim Lillicrap have this paper from f- while ago, maybe five years ago, called, "What Does It Mean To Understand A Neural Network?" Or, "What Would It Mean To Understand A Neural Network?" Um, and what they say is, yeah, basically that. Like, you could imagine you train a neural network to like, compute the digits of pi or something. Or like some crazy... You know, it's like, it's like this crazy pattern. And then you also train that thing to, like, predict the most complicated thing you find, predict stock prices, equi- basically predict the really complex systems, right? Computational, you know, computationally complete systems. I could predict... I could train a neural network to d- do cellular automata or whatever crazy thing. And it's like we're never gonna be able to fully capture that with interpretability, I think. It's just gonna just be doing really complicated computations internally. But we can still say that the way it got that way is that it had an architecture and we gave it this training data (laughs) and it had this loss function. And so I want to describe the brain in the same way. And I think that this framework that I've been kind of laying out is like, we under- need to understand the cortex and how it embodies a learning algorithm. I don't need to understand how it computes Golden Gate Bridge, right?

Dwarkesh Patel 1:06:19 But if you, if you can see all the neurons, if you have the connectome-

Adam Marblestone 1:06:21 Yeah.

Dwarkesh Patel 1:06:21 ... why does that teach you what the learning algorithm is?

Adam Marblestone 1:06:25 Well, I guess there are a couple different views of it. So it depends on these different parts of this portfolio. So on the totally bottom-up, we have to simulate everything portfolio, it kinda just doesn't. You have to just, like, see what are the... You have to make a simulation of the zebrafish brain or something.

Dwarkesh Patel 1:06:38 Yeah.

Adam Marblestone 1:06:38 And then you, like, see what are the, like, emergent dynamics in this, and you come up with new names and new concepts and all that. That's like, that's like the most extreme bottom-up neuroscience view. Um, but even there, the connectome is, like, really important for doing that botta- biophysical or bottom-up simulation. Um, but then on the other hand, you can say, "Well, what if we can actually apply some ideas from AI?" We basically need to figure out, is it an energy-based model or is it, you know, an amortized, you know, VAE-type model? You know, is it doing backprop or is it doing something else? Um, are the learning rules local or global? I mean-If we have some repertoire of possible ideas about this, can we... Just think of the connectome as a huge number of additional constraints that will help to refine to ultimately have a consistent picture of that. I think about this for the, the steering subsystem stuff too, just very basic things about it. How many different types of dopamine signal, or of steering subsystem signal, or thought assessor, or so on? How many different types of what broad categories are there? Like even this very basic information that there's more cell types in the hypothalamus than there are in the cortex, like that's new information, right?

Dwarkesh Patel 1:07:45 Mm-hmm.

Adam Marblestone 1:07:45 About how much structure is built there versus somewhere else. Yeah, how many different dopamine neurons are there? Is the wiring between prefrontal and auditory the same as the wiring between prefrontal and visual? You know, it's like, uh, the most basic things we don't know. And the problem is learning even the most basic things by a series of bespoke experiments takes an incredibly long time. Whereas just learning all that at once by getting a connectome is just, like, way more efficient.

Dwarkesh Patel 1:08:10 What is the timeline on this? Because presumably the idea of this is to, um... Well, first, inform the development of AI. You want to be able to figure out, uh, how we do the, um... How we get AIs to want to care about what other people-

Adam Marblestone 1:08:24 Yeah.

Dwarkesh Patel 1:08:25 ... think of its internal thought pattern. But interp researchers are making progress on this question just by inspecting, you know, normal neural networks. There must be some feature-

Adam Marblestone 1:08:35 They're doing, they're ma- They, uh, you can do interp on LLMs that exist.

Dwarkesh Patel 1:08:38 Yeah, but-

Adam Marblestone 1:08:38 You can't do interp on a hypothetical model-based reinforcement algorithm like the brain-

Dwarkesh Patel 1:08:43 Sure.

Adam Marblestone 1:08:43 ... that we will eventually converge to when we do AGI.

Dwarkesh Patel 1:08:46 Fair, fair.

Adam Marblestone 1:08:46 Yeah. Yeah.

Dwarkesh Patel 1:08:46 But, um, yeah. You know, what, w- what timelines on AI do you need for this-

Adam Marblestone 1:08:51 Right. I see.

Dwarkesh Patel 1:08:52 ... research to be practical-

Adam Marblestone 1:08:53 Yeah.

Dwarkesh Patel 1:08:53 ... and relevant to AI?

Adam Marblestone 1:08:54 I think it's fair to say it's not super practical and relevant if you're in, like, an AI 2027 scenario.

Dwarkesh Patel 1:09:01 Yeah.

Adam Marblestone 1:09:01 You know, and so, like, what science I'm doing now is not going to affect the science of, like, f- ten years from now, because what's going to affect the science of ten years from now is the outcome of this, like, AI 2027 scenario.

Dwarkesh Patel 1:09:10 Yeah.

Adam Marblestone 1:09:10 Right? So it kind of doesn't matter that much probably if I have the connectome. Maybe it, it slightly tweaks certain things. But, um, but I think there, there's a lot of reason to think maybe that we will get a lot out of this paradigm. But then the real thing, the thing that is like the, the tran- the tr- the, the, like, single event that is, like, transformative for the entire future or something type event is still, like, you know, more than five years away or something. (laughs)

Dwarkesh Patel 1:09:38 Sa- sorry. Is that because the... Like, we haven't captured omnidirectional inference. We haven't figured out the right ways to get a mind to pay attention to things in a way that makes it

Narrator 1:09:50 You didn't.

Adam Marblestone 1:09:50 I mean, I would take the entirety of your, like, collective podcast with everyone as-

Dwarkesh Patel 1:09:54 (laughs)

Adam Marblestone 1:09:54 ... like, showing, like, the distribution of these things, right?

Dwarkesh Patel 1:09:58 Okay.

Adam Marblestone 1:09:58 I don't know, right?

Dwarkesh Patel 1:09:59 (laughs)

Adam Marblestone 1:09:59 Um, but I mean, what was Karpathy's timeline, right? You know-

Dwarkesh Patel 1:10:02 Ten years.

Adam Marblestone 1:10:02 ... with Demis's timeline, right? So these, thes- the n- not everybody has a three-year timeline. And, and so I think if you-

Dwarkesh Patel 1:10:08 But there's different reasons. And I'm curious which ones there is.

Adam Marblestone 1:10:10 There are different reasons. What are mine? I don't know. I'm just watching your podcast. I'm trying to- I'm trying to understand the distribution. I don't have a super strong claim that LLMs can't do it. Um...

Dwarkesh Patel 1:10:19 But I mean is it correct-

Adam Marblestone 1:10:19 Yeah.

Dwarkesh Patel 1:10:19 ... like, data efficiency, or is it the o-

Adam Marblestone 1:10:21 I think part of it is just it is weirdly different than all this brain stuff.

Dwarkesh Patel 1:10:25 Yeah, yeah, yeah.

Adam Marblestone 1:10:26 And so intuitively, it's just weirdly different than all this brain stuff, and I'm kind of waiting for, like, the thing that starts to look more like brai- Like, I th- thinking of AlphaZero, and model-based RL, and all of these other things that were being worked on ten years ago had been giving us the GPT-5 type capabilities, then I would be like, "Oh, wow, we're both in the right paradigm and seeing the results-

Dwarkesh Patel 1:10:45 Right.

Adam Marblestone 1:10:45 ... a priori." So my model, uh, my prior and my data are agreeing. (laughs)

Dwarkesh Patel 1:10:49 Yeah, yeah.

Adam Marblestone 1:10:49 Right? And now, it's like I don't know what exactly-

Dwarkesh Patel 1:10:52 That's true. Yeah.

Adam Marblestone 1:10:52 ... my data is. Looks pretty good, but my prior is sort of weird. So, yeah. So it's, so I, I don't have a super strong opinion on it. So I think there's a possibility that essentially all other scientific research that is being done is, like, not... I- i- it's somehow obviated, but I- I don't put a huge amount of probability on that. I think my timelines might be more in the, like, yeah, ten-year-ish range. And if that's the case, I mean, I think there, there, yeah, there is probably a difference between a world where we have connectomes on hard drives and we have understanding of steering subsystem architecture. We've compared the, the, you know, even the most basic properties of what are the reward functions, cost function architecture, et cetera of, you know, mouse versus a shrew versus a small primate, et cetera.

Dwarkesh Patel 1:11:32 Th- this is practical in ten years?

Adam Marblestone 1:11:33 Yeah. Uh, I think it has to be a really big push. Um...

Dwarkesh Patel 1:11:36 Like, how, how, how much funding? How does it compare to where we are now?

Adam Marblestone 1:11:39 It's, like, billion, low billions dollars scale funding w- in a very concerted way, I would say, to make these

Narrator 1:11:44 Yeah.

Dwarkesh Patel 1:11:44 And how much is, is on it now?

Adam Marblestone 1:11:46 Um, well, so, so if I just talk about some of the specific things we have going.

Dwarkesh Patel 1:11:50 Yeah.

Adam Marblestone 1:11:50 So with connectomics, so, uh, so E11Bio is kind of, like, the, the, the, our main thing on connectomics. Um, th- they are basically trying to make the technology of connectomic brain mapping, um, several orders