Andrej Karpathy on Dwarkesh Patel: Why Agents Take a Decade

Andrej Karpathy 0:00 Reinforcement learning is terrible. (laughs)

Dwarkesh Patel 0:02 (laughs)

Andrej Karpathy 0:02 It just so happens that everything that we had before it is much worse. (laughs)

Dwarkesh Patel 0:06 (laughs)

Andrej Karpathy 0:06 I'm actually optimistic. I think this will work. I think it's tractable. I'm only sounding pessimistic because when I go on my Twitter timeline-

Dwarkesh Patel 0:12 (laughs)

Andrej Karpathy 0:12 ... I see all this stuff that makes no sense to me. A lot of it is, I think, honestly just, uh, fundraising. We're not actually building animals. We're building ghosts. These are like sort of ethereal spirit entities because they're fully digital and they're kind of like mimicking humans, and it's a different kind of intelligence. It's business as usual because we're in an intelligence explosion already and have been for decades. Everything is gradually being automated, has been for hundreds of years. Don't write blog posts, don't do slides, don't do any of that.

Dwarkesh Patel 0:37 (laughs)

Andrej Karpathy 0:37 Like, build the code, arrange it, get it to work. It's the only way to go, otherwise you're missing knowledge. If you have a perfect AI tutor, maybe you can get extremely far. The geniuses of today are barely scratching the surface of what a human mind can do, I think.

Dwarkesh Patel 0:48 Today, I'm speaking with Andrej Karpathy. Andrej, why do you say that this will be the decade of agents and not the year of agents?

Andrej Karpathy 0:54 Mm-hmm. Uh, well, first of all, uh, thank you for, uh, having me here. I'm, uh, excited to be here. So the quote that you just mentioned, "It's the decade of agents," that's actually a reaction to an existing, preexisting quote, I should say, where I think a lot of th- some of the labs... I'm not actually sure who said this, but they were alluding to this being the year of agents-

Dwarkesh Patel 1:11 Hmm.

Andrej Karpathy 1:11 ... uh, with respect to LLMs and, uh, how they were gonna evolve. And I think, um, I was triggered by that-

Dwarkesh Patel 1:17 (laughs)

Andrej Karpathy 1:17 ... because I feel like there's some over-predictions going on in the industry.

Dwarkesh Patel 1:20 Yeah.

Andrej Karpathy 1:20 And, uh, in my mind, this is really a lot more accurately described as the decade of agents.

Dwarkesh Patel 1:25 Yeah.

Andrej Karpathy 1:25 And we have some very early agents that are actually like extremely impressive and that I use daily. Uh, you know, Claude and Codex and so on. But I still feel like there's, uh, so much work to be done. And so I think my, like my reaction is like, we'll be working with these things for a decade. They're gonna get better, uh, and, uh, it's gonna be wonderful. But I think I was just reacting to the timelines, I suppose, of the, of the, uh, implication.

Dwarkesh Patel 1:46 And w- what do you think it will take a decade to accomplish?

Andrej Karpathy 1:48 Yeah.

Dwarkesh Patel 1:48 What are the bottlenecks?

Andrej Karpathy 1:50 Well, um, actually make it work.

Dwarkesh Patel 1:52 Mm-hmm.

Andrej Karpathy 1:52 So in my mind, I mean, when you're talking about an agent, I guess, or what the labs have in mind and what maybe I have in mind as well, is it's, uh, you should think of it almost like an employee or like an intern that you would-

Dwarkesh Patel 2:00 Yeah.

Andrej Karpathy 2:00 ... hire to work with you. Uh, so for example, you work with some employees here.

Dwarkesh Patel 2:03 Yeah.

Andrej Karpathy 2:04 Um, when would you prefer to have an agent like Claude or Codex, uh, do that work?

Dwarkesh Patel 2:08 Yeah.

Andrej Karpathy 2:08 Like currently, of course, they can't. Uh, what would it take for them to be able to do that?

Dwarkesh Patel 2:11 Right.

Andrej Karpathy 2:11 W- why don't you do it today?

Dwarkesh Patel 2:12 Yeah.

Andrej Karpathy 2:13 And the reason you don't do it today is because they just don't work. So-

Dwarkesh Patel 2:15 Right.

Andrej Karpathy 2:15 ... uh, like they don't have enough intelligence, they're not multimodal enough, they can't do computer use and all this kind of stuff, and they don't do a lot of the things that you've alluded to earlier. You know, they don't have continual learning. You can't just tell them something and they'll remember it.

Dwarkesh Patel 2:26 Yeah.

Andrej Karpathy 2:26 And they're just cognitively lacking and it's just not working, and I just think that it will take about a decade to work through all those issues.

Dwarkesh Patel 2:32 Interesting. So, um, a- as a professional podcaster and a-

Andrej Karpathy 2:37 (laughs)

Dwarkesh Patel 2:37 ... uh, (laughs) a- a viewer of AI From Afar, i- it's s- easy to identify for me, like, oh, here's what's lacking. Continual learning is lacking, or multimodality is lacking. But I don't really have a good, um, way of trying to put a timeline on it.

Andrej Karpathy 2:52 Mm-hmm.

Dwarkesh Patel 2:52 Like if somebody's like, "How long will continual learning take?" I... there's no, like, prior I have about, like, this is a project that should take five years, 10 years, 50 years.

Andrej Karpathy 3:00 Mm-hmm.

Dwarkesh Patel 3:01 W- why a decade? Why not one year?

Andrej Karpathy 3:02 Mm-hmm.

Dwarkesh Patel 3:02 Why not 50 years?

Andrej Karpathy 3:04 I see. Um, yeah, I guess this is where you get into like a bit of, uh, I guess, my own intuition a little bit.

Dwarkesh Patel 3:08 Yeah.

Andrej Karpathy 3:08 And also just kind of doing s- a bit of an extrapolation of, with respect to my own experience in the field.

Dwarkesh Patel 3:13 Yeah. Mm-hmm.

Andrej Karpathy 3:13 Right? So, I guess I've been in AI for almost two decades. I mean, it's got to be maybe 15 years or so. Not that long. Um, you had Richard Sutton here-

Dwarkesh Patel 3:20 Yeah.

Andrej Karpathy 3:20 ... who was around, of course, for much longer. But I do have about 15 years of experience of people making predictions, of seeing how they actually, uh, turned out. And also I was in the industry for a while and I was-

Dwarkesh Patel 3:29 Hmm.

Andrej Karpathy 3:29 ... in research, and I worked in the industry for a while. So, I guess I kind of have, uh, just a general intuition that I have left from that. Uh, and, uh, I feel like the problems are, uh, tractable, they're surmountable.

Dwarkesh Patel 3:41 Yeah.

Andrej Karpathy 3:41 But, uh, they're still difficult. And if I just average it out, it just kind of feels like a decade, I guess, to me.

Dwarkesh Patel 3:47 Thi- this is actually quite interesting. I want to, like, hear not only the history, but what people in the room-

Andrej Karpathy 3:53 Mm-hmm.

Dwarkesh Patel 3:53 ... felt was about to happen at various different breakthrough moments.

Andrej Karpathy 3:57 Right.

Dwarkesh Patel 3:57 Like, what were the ways in which their feelings were either overly pessimistic or overly optimistic?

Andrej Karpathy 4:03 Yeah.

Dwarkesh Patel 4:03 Yeah. Sh- should we just go through each of them one by one?

Andrej Karpathy 4:04 Uh, yeah. I mean, that's a giant question because, of course, you're talking about 15 years of stuff that's happened.

Dwarkesh Patel 4:08 That's right.

Andrej Karpathy 4:08 I mean, AI is actually like so wonderful because there have been a number of, I would say, seismic shifts-

Dwarkesh Patel 4:12 Yeah.

Andrej Karpathy 4:13 ... that were like, the entire field has sort of, like, suddenly looked a different way.

Dwarkesh Patel 4:16 Yeah.

Andrej Karpathy 4:16 Right? And I guess I've maybe lived through two or three of those.

Dwarkesh Patel 4:19 Yeah.

Andrej Karpathy 4:19 And I still think there will continue to be some, because they come with some kind of like almost surprising regularity. Well, my, when my career began, of course, like, when I started to work on deep learning, when I became interested in deep learning, this was just kind of like by chance of being right next to Geoff Hinton at University of Toronto. And Geoff Hinton, of course, is kind of like the godfather figure of AI. And he was training all these neural networks, and I thought it was incredible and interesting. But this was not like the main thing that everyone in AI was doing by far.

Dwarkesh Patel 4:43 Yeah.

Andrej Karpathy 4:43 This was a niche little subject on the side. That's kind of maybe like the first, like, dramatic sort of seismic shift-

Dwarkesh Patel 4:49 Yeah.

Andrej Karpathy 4:49 ... that came with the AlexNet and so on. I would say, like, AlexNet sort of reoriented everyone, and everyone started to train neural networks. Uh, but it was still like very, like, per task, per specific task.

Dwarkesh Patel 4:59 Yeah.

Andrej Karpathy 4:59 So maybe I have an image classifier or I have a n- neural machine translator or something like that.

Dwarkesh Patel 5:04 Yeah.

Andrej Karpathy 5:04 And people became very slowly actually interested in basically kind of agents, I would say. Uh, (laughs) um, and people started to think, okay, well, maybe we have a check mark next to the visual cortex or something like that, but what about the other parts of the brain, and how can we get an actual, like, full agent or a full entity that can actually interact in the world? And I would say the Atari uh, sort of, uh, deep reinforcement learning shift in 2013 or so-

Dwarkesh Patel 5:24 Yeah.

Andrej Karpathy 5:24 ... uh, was part of that early effort of agents, in my mind, because it was an attempt to try to get agents that not just perceive the world, but also take actions and interact-

Dwarkesh Patel 5:32 Yeah.

Andrej Karpathy 5:32 ... and get rewards from environments. And at the time, this was Atari games.

Dwarkesh Patel 5:35 Right.

Andrej Karpathy 5:36 And I kind of feel like that was a misstep, actually. Uh, and it was a misstep that actually even the early OpenAI, that I was a part of, of course, uh, kind of adopted, because at that time, the zeitgeist was reinforcement learning environments, games, game playing, beat games, got lots of different types of games, and OpenAI was doing a lot of that. So, that was maybe like another, like, prominent part of, I would say, AI where maybe for two or three or four years, everyone was doing reinforcement learning on games.

Dwarkesh Patel 6:02 Mm-hmm.

Andrej Karpathy 6:03 And, uh, basically it was a little bit of a misstep.... and what I was trying to do at OpenAI, actually, is, like, I was always a little bit suspicious of games as being, like, this thing that would actually lead to AGI. Because, in my mind, you want something like an accountant or, uh, like something that's actually interacting with the real world, and I just didn't see how games kind of, like, add up to it. And so, my project at OpenAI, for example, was, um, within the scope of the Universe Project on an, on an agent that was using keyboard and mouse to operate, uh, web pages.

Dwarkesh Patel 6:29 Mm.

Andrej Karpathy 6:30 And I really wanted to have something that, like, interacts with, you know, the actual digital world-

Dwarkesh Patel 6:33 Mm-hmm.

Andrej Karpathy 6:33 ... that can do knowledge work. And it just so turns out that, um, this was extremely early, way too early.

Dwarkesh Patel 6:38 Mm-hmm.

Andrej Karpathy 6:39 So early that they... we shouldn't have been working on that, you know?

Dwarkesh Patel 6:41 (laughs)

Andrej Karpathy 6:41 Uh, because, um, if you're just fumbling your way around and keyboard mashing and mouse clicking and trying to get rewards in these environments, um, your reward is too sparse and you just won't learn, and you're gonna burn a forest, uh, computing-

Dwarkesh Patel 6:54 Yeah.

Andrej Karpathy 6:54 ... and you're never actually gonna get something off the ground. And so what you're missing is this, uh, power of representation in the neural network.

Dwarkesh Patel 7:00 Yeah.

Andrej Karpathy 7:00 And so for example, today, people are training those computer-using agents, but they're doing it on top of a large language model. And so you actually have to get the language model first. You have to get the representations first, and you have to do that by all the pre-training and all the LLM stuff. So, I kind of feel like maybe, loosely speaking, it was like people keep maybe trying to get the full thing too early a few times, where people, like, really try to go after agents too early, I would say.

Dwarkesh Patel 7:22 Mm-hmm.

Andrej Karpathy 7:22 And that was Atari and Universe, uh, and even my own experience. And you actually have to do some things first before we sort of get to those agents. Um, and maybe now the agents are a lot more competent, but maybe we're still missing, uh, sort of some parts, uh, of that stack. But I would say maybe those are, like, the three, like, major buckets of what people were doing. Uh, training neural nets, uh, per tasks, trying to... the first round of agents, uh, and then maybe the LLMs and actually seeking the representation power of the neural networks before you, uh, tack on everything else on top.

Dwarkesh Patel 7:51 Interesting. Yeah. I guess if I were to steel man this sort of a Sutton perspective, it would be that humans actually can just take on everything at once, right? Even animals can take on everything at once, right?

Andrej Karpathy 8:00 Mm-hmm.

Dwarkesh Patel 8:00 A- a- animals are maybe a better example because they don't even have the scaffold of language.

Andrej Karpathy 8:03 Mm-hmm.

Dwarkesh Patel 8:03 They just get thrown out into the world-

Andrej Karpathy 8:05 Yeah.

Dwarkesh Patel 8:05 ... and they just have to make sense of everything without any labels. Um-

Andrej Karpathy 8:09 Right.

Dwarkesh Patel 8:09 ... and the vision for AGI then should just be something which, like, just looks at sensory data-

Andrej Karpathy 8:14 Yeah.

Dwarkesh Patel 8:14 ... looks at the computer screen, and it just, like, figures out what's going on from scratch. I mean, if a human was put in a similar situation, had to be trained from scratch, but I mean, this is just like a human growing up. We're an animal growing up.

Andrej Karpathy 8:23 Yeah.

Dwarkesh Patel 8:24 So, why shouldn't that be the vision for AI-

Andrej Karpathy 8:25 Yeah.

Dwarkesh Patel 8:25 ... rather than, like, this thing where we're doing millions of years of training?

Andrej Karpathy 8:29 I think that's a really good question, and I think, um... I mean, so- so Sutton was on your podcast-

Dwarkesh Patel 8:34 Right.

Andrej Karpathy 8:34 ... and I saw the podcast, and I had a writeup about that podcast almost-

Dwarkesh Patel 8:37 Yeah.

Andrej Karpathy 8:37 ... that gets into a little bit of how I see things. And I- I kind of feel like I'm very careful to make analogies to animals.

Dwarkesh Patel 8:44 Hmm.

Andrej Karpathy 8:44 Because they came about by a very different optimization process.

Dwarkesh Patel 8:48 That's right.

Andrej Karpathy 8:48 Animals are evolved, and they actually come with a huge amount of hardware that's built in. Um, and when... for example, my example in the post was the zebra. A zebra gets born, and a few minutes later it's running around and following its mother. That's an extremely complicated thing to do. (laughs)

Dwarkesh Patel 9:01 Yeah.

Andrej Karpathy 9:02 Um, that's not reinforcement learning, that's something that's baked in. And evolution obviously has some way of encoding the weights of our neural nets in ATCGs, and I have no idea how that works, but it apparently works. So, I kind of feel like, uh, brains just were... came from a very different process, and I- I'm very hesitant to take inspiration from it, because we're not actually running that process. So in my post, I kind of said, "We're not actually building animals. Uh, we're building ghosts-"

Dwarkesh Patel 9:26 Yeah.

Andrej Karpathy 9:27 "... or spirits or whatever people want to call it." Uh, because, um, we're not, uh, we're not doing training by evolution. Uh, we're doing training by basically imitation of humans-

Dwarkesh Patel 9:37 Yeah.

Andrej Karpathy 9:37 ... and the data that they've put on the internet. And so you end up with these, like, sort of ethereal spirit entities because they're fully digital and they're kind of like mimicking humans, and it's a different kind of intelligence. Like, if you imagine a space of intelligences, we- we're starting off at a different point almost. We're not, we're not really building animals, but I think it's also possible to make them a bit more animal-like over time.

Dwarkesh Patel 9:55 Mm.

Andrej Karpathy 9:55 And I think we should be doing that. And so I kind of feel like... Sorry, just, I guess, one more point is-

Dwarkesh Patel 9:59 Please, yeah.

Andrej Karpathy 9:59 ... I do feel like Sutton basically has a very... like his framework is, uh, like, we're- we want to build animals.

Dwarkesh Patel 10:04 Yeah.

Andrej Karpathy 10:04 And I actually think that would be wonderful. If we can get that to work, that would be amazing. If there was a single, like, algorithm that you can just, uh, you know, run on the internet and it learns everything-

Dwarkesh Patel 10:13 Yeah.

Andrej Karpathy 10:13 ... that would be incredible. I almost suspect that... I'm not actually sure that it exists, and that's certainly actually not what animals do, because animals have this outer loop of evolution.

Dwarkesh Patel 10:23 Right.

Andrej Karpathy 10:24 Um, and a lot of what looks like learning is actually a lot more maturation of the brain, and I think there's act- actually very little reinforcement learning for animals. And I think a lot of the reinforcement learning is actually, like, more like motor tasks. It's not intelligence tasks. So I actually kind of think humans don't actually, like, really use RL, roughly speaking, is what I would say.

Dwarkesh Patel 10:41 Can you repeat the last sentence? A lot of that intelligence is not motor tasks, it's what, sorry?

Andrej Karpathy 10:44 A- a lot of the reinforcement learning, in my perspective, would be things that are a lot more like motor-like, like, uh, simple, uh, kind of like tasks-

Dwarkesh Patel 10:50 Yeah.

Andrej Karpathy 10:50 ... throwing a hoop-

Dwarkesh Patel 10:51 That's right.

Andrej Karpathy 10:51 ... something like that. Um, but I don't think that humans use reinforcement learning for a lot of intelligence tasks, like problem solving and so on.

Dwarkesh Patel 10:58 Interesting.

Andrej Karpathy 10:59 That doesn't mean we don't have... we- we shouldn't do that for research, but I just feel like that's what animals do or don't.

Dwarkesh Patel 11:06 Yeah. I'm gonna take a s- a second to digest that because there's a lot of-

Andrej Karpathy 11:08 (laughs)

Dwarkesh Patel 11:08 ... different ideas. Maybe o- s- one clarifying question I can ask to, um, understand the perspective. So, I think you s- suggest that, look, evolution is doing the kind of thing that pre-training does in the sense of building something which can then understand the world. The difference, I guess, is that evolution has to be titrated in the case of humans through three gigabytes of DNA, and so that's very unlike the weights of a model. I mean, literally the weights of the model are a brain, which obviously is not encoded in the- the sperm and the eg- or is- does not exist in the sperm and the egg, so it has to be grown. And also, the information for every single synapse in the brain simply cannot exist in the three gigabytes that exist in the DNA. Evolution seems closer to finding the algorithm-

Andrej Karpathy 11:55 Mm-hmm.

Dwarkesh Patel 11:55 ... which then does the lifetime learning.

Andrej Karpathy 11:58 Mm-hmm.

Dwarkesh Patel 11:58 Now, maybe the lifetime learning is not analogous to RL, to your point. Is that compatible with the thing you were saying, or would you disagree with that?

Andrej Karpathy 12:05 I think so. I would agree with you that there's some miraculous compression going on-

Dwarkesh Patel 12:08 Yeah.

Andrej Karpathy 12:08 ... because obviously the weights of the neural net are not stored in the ATCGs.

Dwarkesh Patel 12:11 Yeah.

Andrej Karpathy 12:11 There's some kind of a dramatic compression, and there's some kind of, like, learning algorithms encoded that- that take over and do-

Dwarkesh Patel 12:17 Yeah.

Andrej Karpathy 12:17 ... some of the learning online. So, I definitely agree with you on that. Uh, basically, I would say I'm a lot more kind of, like, practically minded. I don't come at it from the perspective of like, "Let's build animals."

Dwarkesh Patel 12:26 Right.

Andrej Karpathy 12:26 I come from the per- perspective of like, "Let's build useful things." So I have a hard hat on, and I'm just observing that, look, we're not gonna do evolution.... because I don't know how to do that (laughs) . Uh, but it does turn out we can build these ghost, uh, spirit-like entities-

Dwarkesh Patel 12:37 Yeah.

Andrej Karpathy 12:37 ... by imitating internet documents. This works, and it's actually kind of like, it's a way to bring you up to something that has a lot of sort of built-in knowledge and intelligence in some way, uh, similar to maybe what evolution has done. So, that's why I kind of call pre-training this kind of, like, crappy evolution. It's like the practically possible version with our technology and what we have available to us to get to a starting point where we can actually do things like reinforcement learning and so on.

Dwarkesh Patel 13:02 Hmm. Just to steel man the other perspective, because after doing this interview and thinking about it a bit, here's an important point here. Evolution does not give us the knowledge really, right? It gives us the algorithm to find the knowledge.

Andrej Karpathy 13:14 Mm-hmm.

Dwarkesh Patel 13:14 And that seems different from pre-training.

Andrej Karpathy 13:15 Yeah.

Dwarkesh Patel 13:15 So, if, perhaps the perspective is that pre-training helps build the kind of entity which can learn better.

Andrej Karpathy 13:21 Hmm.

Dwarkesh Patel 13:21 It teaches meta learning, and therefore, it is some similar to, like, finding an algorithm.

Andrej Karpathy 13:25 Mm-hmm.

Dwarkesh Patel 13:26 Um, but if, if it's, like, evolution gives us knowledge and, uh, pre-training gives us knowledge, they're, they're not...

Andrej Karpathy 13:29 Mm-hmm.

Dwarkesh Patel 13:29 That analogy seems to break down.

Andrej Karpathy 13:31 Yeah, yeah. So, it's subtle, and I think you're, you're right to push back on it. But basically, the thing that pre-training is doing, so you're basically getting the next token predictor-

Dwarkesh Patel 13:37 Yeah.

Andrej Karpathy 13:37 ... on, over the internet, and you're training that into a neural net. It's doing two things, actually, that are kind of, like, unrelated. Number one, it's picking up all this knowledge, as I call it. Number two, it's actually becoming intelligent.

Dwarkesh Patel 13:48 Hmm.

Andrej Karpathy 13:48 Um, by observing the algorithmic patterns in the internet, it actually kind of, like, boots up all these, like, little circuits and algorithms inside-

Dwarkesh Patel 13:54 Oh.

Andrej Karpathy 13:54 ... the neural net to do things like in-context learning-

Dwarkesh Patel 13:56 Yeah.

Andrej Karpathy 13:56 ... and all this kind of stuff. And actually, uh, you don't actually need or want the knowledge. I actually think that's probably actually holding back the neural networks overall-

Dwarkesh Patel 14:03 Interesting.

Andrej Karpathy 14:03 ... because it's actually, like, getting them to rely on the knowledge a little too much sometimes. For example, I, I kind of feel like agents, one thing they're not very good at is going off the data manifold-

Dwarkesh Patel 14:11 Hmm.

Andrej Karpathy 14:11 ... of what exists on the internet. If they had less knowledge, or less memory-

Dwarkesh Patel 14:15 Yeah.

Andrej Karpathy 14:16 ... actually, maybe they would be better.

Dwarkesh Patel 14:17 Yeah.

Andrej Karpathy 14:17 And so what I think we have to do kind of going forward, and this would be part of the research paradigms, is actually, I think, we need to start, um, we need to figure out ways to remove some of the knowledge, and to keep what I call this cog, uh, as this cognitive core.

Dwarkesh Patel 14:28 Mm-hmm.

Andrej Karpathy 14:28 As this, like, intelligent entity that is kind of stripped from knowledge but contains the algorithms and contains the magic, you know, of intelligence and problem-solving and the strategies of it and all this kind of stuff.

Dwarkesh Patel 14:39 Th- there's so much interesting stuff there. Okay. So, let's start with in-context learning. This is an obvious point, but I think it's worth just, like, saying it explicitly-

Andrej Karpathy 14:47 Mm-hmm.

Dwarkesh Patel 14:47 ... and meditating on it. The situation in which these models seem the most intelligent, in which they are like, I talk to them, and I'm like, "Wow, there's really something on the other end that's, like, responding to me thinking about things." If it, like, makes a mistake, it's like, "Oh, wait, that's actually the wrong way to think about it. I'm backing up." All that is happening in-context. That's where I feel like the real intelligence you can, like, visibly see.

Andrej Karpathy 15:06 Yep.

Dwarkesh Patel 15:06 And that in-context learning process is developed by gradient descent on pre-training, right? Like, it meta, it spontaneously meta learns in-context learning.

Andrej Karpathy 15:15 Mm-hmm.

Dwarkesh Patel 15:16 But the in-context learning itself is not gradient descent, in the same way that our lifetime intelligence as humans to be able to do things is conditioned by evolution. But our actual learning during our lifetime is, like-

Andrej Karpathy 15:29 Yeah.

Dwarkesh Patel 15:29 ... happening through some other process.

Andrej Karpathy 15:30 I actually don't fully agree with that, but you should continue your thought (laughs) .

Dwarkesh Patel 15:33 Oh, oh, okay. I actually then, I, I'm very curious to understand how that analogy breaks down.

Andrej Karpathy 15:36 I think I'm hesitant to say that in-context learning is not doing gradient descent, uh, because, I mean, it's not doing explicit gradient descent.

Dwarkesh Patel 15:43 Mm-hmm.

Andrej Karpathy 15:43 But I, I still think that... So, in-context learning, basically it's, it's pattern completion within-

Dwarkesh Patel 15:47 Hmm.

Andrej Karpathy 15:47 ... uh, a token window, right? And it just turns out that there's a huge amount of patterns on the internet. And so you're right. The model kind of, like, learns to complete the pattern.

Dwarkesh Patel 15:54 Yeah.

Andrej Karpathy 15:54 And that's inside the weights. The weights of the neural network are trying to discover patterns and complete the pattern.

Dwarkesh Patel 15:59 Mm-hmm.

Andrej Karpathy 16:00 And there's some kind of an adaptation that happens inside the neural network, right? Uh, which is kind of magical and just falls out from-

Dwarkesh Patel 16:05 Yeah.

Andrej Karpathy 16:05 ... internet, just because there's a lot of patterns. I will say that there have been some papers that I thought were interesting that actually look at the mechanisms behind in-context learning.

Dwarkesh Patel 16:13 Mm-hmm.

Andrej Karpathy 16:13 And I do think it's possible that in-context learning actually runs a small gradient descent loop, uh, internally in the layers of the neural network. And so, uh, I recall one paper in particular where they were doing, um, uh, linear regression actually, using in-context learning. So, basically, your inputs into the neural network are X, Y pairs.

Dwarkesh Patel 16:29 Mm-hmm.

Andrej Karpathy 16:30 X, Y, X, Y, X, Y that happen to be on a line.

Dwarkesh Patel 16:33 Yeah.

Andrej Karpathy 16:33 And then you do X and you expect a Y.

Dwarkesh Patel 16:35 Yeah.

Andrej Karpathy 16:35 And the neural network, when you train it in this way, actually does do, um, uh, does do linear regression. And, um, normally, when you would run linear regression, you have a small gradient descent optimizer that basically, uh, looks at X,Y looks at an error-

Dwarkesh Patel 16:47 Mm-hmm.

Andrej Karpathy 16:47 ... calculates the gradient of the weights, and does the update a few times.

Dwarkesh Patel 16:50 Yeah.

Andrej Karpathy 16:50 It just turns out that when they looked at the weights of that in-context learning algorithm, uh, they actually found some analogies to, uh, to gradient descent, uh, mechanics. In fact, I think even the paper went str- was stronger because they actually hardcoded the weights of the neural network to do gradient descent through, uh, attention and all the, all the internals o- of, of the neural network. So, I guess that's just my only pushback, is that who knows how in-context learning works? But I actually think that it's probably doing a little bit of some kind of funky gradient descent internally, and that I think that that's, that's possible. So, I guess I, I was only pushing back on, you're saying it's not doing in-context learning. Who knows what it's doing? But it's probably maybe doing something similar to it, but we don't know (laughs) .

Dwarkesh Patel 17:28 So, then it's worth thinking about, okay, if both of them are implementing gradient des- sorry, if in-context learning and pre-training are both implementing something like gradient descent-

Andrej Karpathy 17:37 Right.

Dwarkesh Patel 17:37 ... why does it feel like in-context learning, actually, we're getting to this, like, continual learning, real intelligence-like thing? Whereas you don't get the analogous feeling just from pre-training. At least-

Andrej Karpathy 17:48 Hmm.

Dwarkesh Patel 17:48 ... you could argue that. And so if it's the same algorithm, what could be different? Well, one way you could think about it is how much information does the model store per information it receives from training? And if you look at pre-training, if, uh, I think l- if you look at LLaMA 3, for example-

Andrej Karpathy 18:03 Mm-hmm.

Dwarkesh Patel 18:03 ... I think it's trained on 15 trillion tokens.

Andrej Karpathy 18:06 Mm-hmm.

Dwarkesh Patel 18:06 And, uh, if you look at a 70B model, that would be the equivalent of .07 bits-

Andrej Karpathy 18:13 (laughs)

Dwarkesh Patel 18:13 ... per token in data it sees in pre-training, in terms of, like, the information in the weights of the model-

Andrej Karpathy 18:17 Yeah.

Dwarkesh Patel 18:17 ... compared to the tokens it reads. Whereas if you look at the KV cache-

Andrej Karpathy 18:20 Mm-hmm.

Dwarkesh Patel 18:21 ... and how it grows per additional token in in-context learning, it's like 320 kilobytes.

Andrej Karpathy 18:26 Yeah.

Dwarkesh Patel 18:26 So, that's a 35 million fold difference in how much information per token is assimilated by the model.

Andrej Karpathy 18:33 Yeah. Yeah.

Dwarkesh Patel 18:33 I wonder if that's relevant at all.

Andrej Karpathy 18:34 I think I kind of agree. I mean, the way I usually put this is that anything that happens during the training of the neural network, the knowledge is only kind of like a hazy recollection.... of what happened in train- in the training time. And that's because the compression is dramatic. You've, you're taking 15 trillion tokens and you're compressing it to just your final network of a few billion parameters. So obviously, it's a massive amount of compression going on. Uh, so I kind of, uh, refer to it as like a hazy recollection of the internet documents.

Dwarkesh Patel 18:58 Mm-hmm.

Andrej Karpathy 18:58 Whereas, anything that happens in the context window of the neural network, where you're plugging all the tokens and it's building up all this KV cache representation, it's very directly accessible to the neural net. So I compare the KV cache and the, the stuff that happens at test time to, like, more like a working memory. Uh, like all the stuff that's in the, in the, in the context window is very directly accessible-

Dwarkesh Patel 19:15 Yeah.

Andrej Karpathy 19:15 ... to the neural net. So, there's always like these, um, almost surprising analogies between LLMs and humans, and I find them kind of surprising because we're not trying to build a human brain, of course-

Dwarkesh Patel 19:24 Yeah.

Andrej Karpathy 19:24 ... uh, just directly. We're just finding that this works and we're doing it. But I do think that anything that's in the weights, it's kind of like a hazy recollection of what you read a year ago. Anything that you give it as a context, uh, at test time, is directly in the working memory, um, and I think that's a very powerful analogy to think through things. So, when you, for example, go to an LLM and you ask it about, uh, some book and what happened in it, like, uh, Nick Lane's book or something like that-

Dwarkesh Patel 19:46 Yeah.

Andrej Karpathy 19:47 ... the LLM will often give you some stuff which is roughly correct. But if you give it the full chapter and ask it questions, you're gonna get much better results because it's now loaded in the working memory of the model. So, I basically agree with your very long way of saying that I kind of agree, and that's why.

Dwarkesh Patel 19:59 Mm-hmm. Stepping back, what is it that part about human intelligence that we, like, have, uh, most failed to replicate with these models?

Andrej Karpathy 20:08 Um, I almost feel like, um, just, uh, just a lot of it, still. (laughs)

Dwarkesh Patel 20:14 (laughs)

Andrej Karpathy 20:14 So maybe one way to think about it, I don't know if this is the, the best way, but I almost kind of feel like, again, making these analogies, imperfect as they are, um, we've stumbled by with the transformer neural network, which are extremely powerful, very general. You can train transformers on audio or video or text or whatever you want, and it just learns patterns, and they're very powerful and it works really well. That, to me, almost indicates that this is kind of like some piece of cortical tissue. Uh, it's something like that. Because the cortex is famously very, um, plastic as well.

Dwarkesh Patel 20:42 Mm-hmm.

Andrej Karpathy 20:42 You can rewire, um, you know, parts of brains, and, uh, there was the slightly gruesome experiments with rewiring like visual cortex to the auditory cortex and, uh, this animal, like, learned fine, et cetera. Um, so I think that this is kind of like a cortical tissue. I think when we're doing reasoning and planning inside the neural networks, so basically doing a reasoning traces, um, for thinking models, that's kind of like the prefrontal cortex. Um, and then, um, I think we, uh, maybe those are like little check marks, (laughs) but I still think there's many, uh, brain parts and nuclei that are not explored. So maybe, for example, there's a basal ganglia doing a bit of reinforcement learning when we fine-tune the models on reinforcement learning. But, you know, you know, whereas like the hippocampus, not obvious what that would be. Some parts are probably not important. Maybe the cerebellum is like not important to cognition its thoughts, so, so maybe we can skip some of it. Uh, but I still think there's, for example, the amygdala, all the emotions and instincts, um, and there's probably like a bunch of other n- nuclei in the brain that are very ancient that I don't think we've like really replicated. I don't actually know that we should be pursuing, you know, the building of an analog of human brain. Uh, I'm again, an engineer mostly at heart, but, um, I still feel like maybe another way to answer the question is, you're not gonna hire this thing as an intern, and it's missing a lot of... it's-

Dwarkesh Patel 21:53 Mm-hmm.

Andrej Karpathy 21:53 ... because it comes with a lot of these cognitive deficits-

Dwarkesh Patel 21:55 Yeah.

Andrej Karpathy 21:55 ... that we all intuitively feel when we talk to the models. Um, and so it's just like not fully there yet, and you can look at it as like not all the brain parts are checked off yet. (laughs)

Dwarkesh Patel 22:03 Mm-hmm. This is maybe relevant to the t- uh, cr- uh, question of thinking about how fast these issues will be solved. So, sometimes people will say about continual learning, "Look, actually y- you could already, y- you could easily replicate this capability." Just as in-context learning emerged spontaneously as a result of pre-training, m- continual learning over longer horizons will emerge spontaneously if the model is incentivized to recollect information over longer horizons or horizons longer than one session. So if there's, um, some like outer loop RL which it has many sessions within that outer loop, then like this continual learning where it uses like, it fine-tunes itself or it writes to an external memory or something will just sort of like emerge spontaneously. Do you think, do you think things are (inaudible) plausible? I just-

Andrej Karpathy 22:52 Yeah.

Dwarkesh Patel 22:52 I don't have really a prior over like how-

Andrej Karpathy 22:53 Yeah, yeah.

Dwarkesh Patel 22:53 ... plausible is that, how likely is that to happen?

Andrej Karpathy 22:55 I don't know that I fully resonate with that-

Dwarkesh Patel 22:56 Okay.

Andrej Karpathy 22:56 ... because I feel like these models, when you boot them up and they have zero tokens in the window, they're always like restarting from scratch where they were.

Dwarkesh Patel 23:03 Mm-hmm.

Andrej Karpathy 23:03 So, I don't actually know in that worldview what it looks like, uh, because, um, again, making, maybe making some analogies to humans just because I think it's roughly concrete and kind of interesting to think through. I feel like when I'm awake, I'm building up a context window of stuff-

Dwarkesh Patel 23:16 Yeah.

Andrej Karpathy 23:16 ... that's happening during the day. But I feel like when I go to sleep, something magical happens where, uh, I don't actually think that that context window stays around. Um-

Dwarkesh Patel 23:23 Mm-hmm.

Andrej Karpathy 23:23 I think there's some process of distillation into weights of my brain.

Dwarkesh Patel 23:27 Yeah.

Andrej Karpathy 23:27 Um, and this happens during sleep and all this kind of stuff. We don't have an equivalent for all that in l- uh, large language models, and that's to me more adjacent to when you talk about continual learning and so on, uh, as absent.

Dwarkesh Patel 23:38 Yeah.

Andrej Karpathy 23:38 These models don't really have, uh, this distillation phase, um, of taking what happened, analyzing it, obsessively thinking through it, (laughs) um, s- uh, basically doing some kind of a synthetic data generation process-

Dwarkesh Patel 23:49 Yeah.

Andrej Karpathy 23:49 ... and distilling it bac- back into the weights and maybe having, uh, uh, you know, specific neural net per person. Uh, maybe it's a LoRA-

Dwarkesh Patel 23:56 Mm-hmm.

Andrej Karpathy 23:56 ... it's not a full, uh, yeah, it's not a full weight, uh, neural network that's... it's just small, so- some of the small sparse-

Dwarkesh Patel 24:03 Yeah.

Andrej Karpathy 24:03 ... subset of the weights are changed. But basically, we do want to create ways of creating these individuals that have very long contexts as not only remaining in the context window because the context windows grow very, very long. Like, maybe we have some very elaborate sparse attention over it.

Dwarkesh Patel 24:17 Mm-hmm.

Andrej Karpathy 24:17 But I still think that humans obviously have some process for distilling some of that knowledge into the weights. We're missing it. And I do also think that humans, um, have some kind of a very elaborate sparse attention scheme-

Dwarkesh Patel 24:28 Mm-hmm.

Andrej Karpathy 24:29 ... um, which I think, uh, we're starting to see some early hints of. Uh, so DeepSeek, uh, V3.2, uh, just came out, and I saw that they have like a sparse attention as an example. And this is one way to have very, very long context windows. So, I almost feel like we are redoing a lot of the-... cognitive tricks that evolution came up with-

Dwarkesh Patel 24:45 Yeah.

Andrej Karpathy 24:46 ... through a very different process. But we're, I think, gonna converge on a similar architecture cognitively.

Dwarkesh Patel 24:50 Interesting. I- in 10 years, do you think it'll still be something like a transformer, but with a much more modified attention and more sparse, uh, MLPs and so forth?

Andrej Karpathy 24:58 Well, the way I like to think about it is, okay, let's, uh, translation invariants and time, right? So 10 years ago, where were we? 2015, uh, we had, uh, convolutional neural networks-

Dwarkesh Patel 25:07 Yeah.

Andrej Karpathy 25:07 ... primarily. Residual networks just came out. Um, so remarkably similar, I guess, but quite a bit different still. I mean, transformer was not around. Um, you know, all the, um, all these sort of like more modern, uh, tweaks on the transformer were not around. So maybe some of the things that we can bet on, I think, in 10 years, uh, by translational (laughs) sort of equivariance is, um, we're still training giant neural networks with, uh, forward, backward pass and update through gradient descent. Um, but maybe it looks a little bit different-

Dwarkesh Patel 25:36 Mm-hmm.

Andrej Karpathy 25:36 ... and it's just everything is much bigger. Actually, recently, I, uh, also went back all the way to 1989, which was kind of a fun, uh, exercise for me a few years ago, uh, because I was reproducing, uh, Yann LeCun's 1989 convolutional network-

Dwarkesh Patel 25:49 Huh.

Andrej Karpathy 25:49 ... which was the first neural network I'm aware of trained via gradient descent, like modern neural network-trained gradient descent on, uh, digit recognition.

Dwarkesh Patel 25:56 Mm-hmm.

Andrej Karpathy 25:57 And I was just interested in, okay, how can I modernize this? How much of this is algorithms? How much of this is data? How much of this progress is, uh, compute and systems? And I was able to very quickly, like, halve the learning rate just knowing my tra- time travel by 33 years. So if I time travel by algorithms to 33 years, I could adjust what Yann LeCun did in 1989, and I could basically halve the learning, halve the error. But, uh, to get further gains, I had to add a lot more data. I had to, like, 10x the training set, and then I had to actually add more, uh, computational optimizations, uh, h- how to basically train for much longer-

Dwarkesh Patel 26:28 Mm-hmm.

Andrej Karpathy 26:28 ... with dropout and other regularization techniques. And so it's almost like all these things have to improve simultaneously. So, um, you know, we're probably gonna have a lot more data. We're probably gonna have a lot better hardware. Probably gonna have a lot better kernels and software. We're probably gonna have better algorithms. And all of those, it's almost like no one of them is winning too much. All of them are surprisingly equal. (laughs)

Dwarkesh Patel 26:48 Hmm.

Andrej Karpathy 26:48 And this has kind of been the trend for a while. So I guess to answer maybe your question, I expect differences, uh, algorithmically to what's happening today. Uh, but I do also expect that some of the things that have stu- stuck around for very long time will probably still be there. It's probably still a giant neural network trained with gradient descent.

Dwarkesh Patel 27:04 Mm-hmm.

Andrej Karpathy 27:04 That would be my guess.

Dwarkesh Patel 27:05 It's surprising that all of those things together only halved-

Andrej Karpathy 27:09 Mm-hmm.

Dwarkesh Patel 27:09 ... um, uh, half the error.

Andrej Karpathy 27:12 Yeah.

Dwarkesh Patel 27:12 Which is, so it's, like, 30 years of progress is, uh, maybe, maybe half is a lot 'cause, like, if you halve the error, that actually means that-

Andrej Karpathy 27:18 Half is a lot, yeah.

Dwarkesh Patel 27:18 Yeah, yeah. Okay. Um-

Andrej Karpathy 27:20 But it's, uh, I guess what was shocking to me is everything needs to improve across the board.

Dwarkesh Patel 27:23 Yeah.

Andrej Karpathy 27:23 Uh, architecture, optimizer, loss function, and also has improved across the board forever. So I kind of expect all those changes to be well, alive and well.

Dwarkesh Patel 27:31 Well, yeah, actually, I, I was about to ask you a very similar question about NanoChat.

Andrej Karpathy 27:34 Mm-hmm.

Dwarkesh Patel 27:34 'Cause since you just c- coded up recently, e- every single sort of step in the, you know, process of building a chat bot is, like, fresh in your RAM.

Andrej Karpathy 27:42 Yeah. (laughs)

Dwarkesh Patel 27:42 And I'm curious if you had similar thoughts about, like, oh, there was no one thing that was relevant to going from GPT-2 to NanoChat. Uh, like, what, what are sort of, like, surprising takeaways from the experience?

Andrej Karpathy 27:55 Oh, building NanoChat?

Dwarkesh Patel 27:56 Yeah.

Andrej Karpathy 27:56 So NanoChat is a kind of a repository I released, was it yesterday-

Dwarkesh Patel 27:59 Mm-hmm.

Andrej Karpathy 28:00 ... or day before? I can't remember. (laughs)

Dwarkesh Patel 28:02 (laughs) We can see the sleep deprivation that went into the... (laughs)

Andrej Karpathy 28:06 Yeah, yeah. Um, well, it's just trying to be a... it's trying to be the simplest complete repository that covers the whole pipeline end to end-

Dwarkesh Patel 28:13 Yeah.

Andrej Karpathy 28:13 ... of building a ChatGPT clone. And so, you know, you have all of the steps, not just any individual step, which is s- much... I worked on all the individual steps sort of in the past and released small pieces of code that could have, um, show you how that's done in algorithmic sense, um, uh, in, like, simple code. But this kind of handles all the entire pipeline. I, I think in terms of learning, it's not, uh, it's not so much, um... I don't know that I actually found something that I learned from-

Dwarkesh Patel 28:37 Mm-hmm.

Andrej Karpathy 28:37 ... from it necessarily. I kind of already had in my mind as, like, how you build it, and this is just a process of mechanically, uh, uh, building it and making it clean enough and, uh, so that people can actually learn from it and, um, that, uh, they find it useful.

Dwarkesh Patel 28:51 Yeah. What is the best way for somebody to learn from it? Uh, is it just, like, delete all the code and try to reimplement it from scratch, try to add modifications to it?

Andrej Karpathy 28:58 Uh, yeah, I think that's a, uh, that's a great question. I would probably say, so basically it's about e- 8,000 lines of code-

Dwarkesh Patel 29:03 Yeah.

Andrej Karpathy 29:03 ... that takes you through the entire pipeline. I would probably put it on the right monitor. Like, if you have two monitors, you put it on the-

Dwarkesh Patel 29:08 Mm-hmm.

Andrej Karpathy 29:08 ... on the right. Um, and you wanna build it from scratch, you build it from start. You're not allowed to copy-paste.

Dwarkesh Patel 29:14 Mm-hmm.

Andrej Karpathy 29:14 You're allowed to reference. You're not allowed to copy-paste. Maybe that's how I would do it. Um, but I also think the repository by itself, i- it is, like, a pretty large beast. I mean, it's, you know, it's r- it's... when you write this code that you don't go from top to bottom. You go from chunks and you grow the chunks, and, uh, that information is absent. Like, you wouldn't know where to start. And so I think-

Dwarkesh Patel 29:32 Mm-hmm.

Andrej Karpathy 29:32 ... it's not just a final repository that's needed. It's like the building of the repository, which is a complicated chunk-growing process.

Dwarkesh Patel 29:38 Right.

Andrej Karpathy 29:38 Uh, so that part is not there yet. I, I would love to actually, like, add that probably later this week or something in some way. Like, either it's a, uh, it's probably a video or something like that. But, um, but maybe roughly speaking, th- that's what I would try to do is build the stuff yourself, uh, but, uh, don't allow yourself copy-paste.

Dwarkesh Patel 29:54 Yeah.

Andrej Karpathy 29:54 I do think that there's two types of knowledge, almost. Like, there's the high-level surface knowledge, but the thing is that when you actually build something from scratch, you're forced to come to terms with what you don't actually understand, and you don't know that you don't understand it.

Dwarkesh Patel 30:05 Interesting.

Andrej Karpathy 30:06 And it always leads to a deeper understanding. Uh, and, um, it's like, just the only way to, to build is like, uh, if I can't build it, I don't understand it. Is that a Feynman quote, I believe, or something along those lines?

Dwarkesh Patel 30:16 Yeah.

Andrej Karpathy 30:17 100%, I've always believed this very strongly, uh, because there's all these, like, micro things that are just not properly arranged and you don't really have the knowledge. You just think you have the knowledge. So don't write blog posts, don't do slides, don't do any of that.

Dwarkesh Patel 30:28 (laughs)

Andrej Karpathy 30:28 Like, build the code, arrange it, get it to work. It's the only way to go. Otherwise, you're missing knowledge.

Dwarkesh Patel 30:32 Um,

Dwarkesh Patel 30:33 you tweeted out that coding models were actually of very little help to you in assembling this repository, and I'm curious why that was.

Andrej Karpathy 30:41 Yeah. Uh, so the repository, I guess I built it over a period of a bit more than a month, and I would say there's, like, three major classes of how people interact with code right now. Some people completely reject all of LLMs, and they are just, uh, writing by scratch...... I think this is probably not the, the right thing to do anymore. Um, the intermediate part, which is where I am, is you still write a lot of things from scratch, but you use, uh, the autocomplete, uh, that's basically, uh, available now from these models. So, when you start writing out a little b- piece of it, it will, it will autocomplete for you and you can just tab through-

Dwarkesh Patel 31:11 Yeah.

Andrej Karpathy 31:11 ... and most of the time, it's correct. Sometimes it's not and you edit it. But you're still very much the, um, sort of architect of what you're writing. And then there's the, you know, vibe coding. Uh, you know, "Hi, please implement this or that. Uh, you know, enter." And then let the model do it.

Dwarkesh Patel 31:25 Yeah.

Andrej Karpathy 31:25 And that's the agents. Um, I do feel like the agents work in very specific settings, and I would use them in specific settings. But again, these are all tools available to you, and you have to, like, learn what they, what they're good at-

Dwarkesh Patel 31:37 Right.

Andrej Karpathy 31:37 ... and what they're not good at, and when to use them. So, the agents are actually pretty good, for example, if you're doing boilerplate stuff.

Dwarkesh Patel 31:41 Yeah.

Andrej Karpathy 31:42 Boilerplate code that's like just cop- you know, just copy-paste stuff.

Dwarkesh Patel 31:45 Yeah.

Andrej Karpathy 31:45 They're very good at that. They're very good at stuff that occurs very often in the internet, um, because there's lots of examples of it in the training sets of these models. Um, so, so there's, like, features of things that, where the models will do very well. I would say NanoChat is not an example of this, because, uh, it's a fairly unique repository. There's not that much code, I think, in the way that I've structured it. And, um, and it's not boilerplate code. It's, like, actually, like, intellectually intense code almost.

Dwarkesh Patel 32:11 Mm-hmm.

Andrej Karpathy 32:11 And everything has to be very precisely arranged. And the models were always trying to... They kept trying to... I mean, they have so many cognitive deficits, right?

Dwarkesh Patel 32:17 Mm-hmm.

Andrej Karpathy 32:17 So, one example, they keep trying to... They keep misunderstanding the code, um, because they, they have too much memory from all the typical ways of doing things-

Dwarkesh Patel 32:26 Mm-hmm.

Andrej Karpathy 32:26 ... on the internet that I just wasn't adopting. Uh, so the models, for example... I mean, I don't know if I wanna get into the full details, but they keep, they keep, um, they keep thinking I'm writing normal code and I'm not (laughs) .

Dwarkesh Patel 32:37 May- maybe one example. I think it's-

Andrej Karpathy 32:37 Maybe one example is-

Dwarkesh Patel 32:38 ... quite interesting.

Andrej Karpathy 32:38 Uh, so the way to synchronize... So, you have eight GPUs-

Dwarkesh Patel 32:41 Yeah.

Andrej Karpathy 32:41 ... that are all doing forward-backwards. The way to synchronize gradients between them is to use a distributed data parallel container of PyTorch, which automatically does all the... As you're doing the backward, it will start communicating-

Dwarkesh Patel 32:51 Yeah.

Andrej Karpathy 32:51 ... and synchronizing the gradients. I didn't use DDP because I didn't want to use it, because it's not necessary. So, I threw it out, and I basically wrote my own synchronization routine that's inside the step of the optimizer. And so the models were trying to get me to use the DDP container-

Dwarkesh Patel 33:05 (laughs) Yeah.

Andrej Karpathy 33:06 ... and they were very concerned about... Okay, this gets way too technical, but I wasn't using that container because I don't need it, and I have a custom implementation of-

Dwarkesh Patel 33:13 Yeah.

Andrej Karpathy 33:13 ... something like it.

Dwarkesh Patel 33:13 And they just couldn't internalize that you had your own (overlapping)

Andrej Karpathy 33:15 Yeah, they couldn't-

Dwarkesh Patel 33:16 ... implementation?

Andrej Karpathy 33:16 They, they couldn't get past that. And then, um, they kept trying to, like, mess up the style. Like, they're way too over-defensive. They h- make all these try-catch statements.

Dwarkesh Patel 33:24 Yeah.

Andrej Karpathy 33:25 They keep trying to make a production code base, and I have a bunch of assumptions in my code and it's okay. And, uh, and it's just like, I don't need all this extra stuff in there. And so I just kind of feel like they're bloating the code base, they're bloating the complexity, they keep misunderstanding, they're using deprecated APIs a bunch of times. So, it's total mess, um, and, uh, it's just, it's just not that useful. I can go in and I can clean it up, but it's not that useful. I also feel like it's kind of annoying to have to, like, type out what I want in English, because it's just too much typing. Like, if I just navigate to the part of the code that I want, and I go where I n- or I know the code has to appear, and I start typing out the first three letters, autocomplete gets it-

Dwarkesh Patel 34:01 Right.

Andrej Karpathy 34:01 ... and just gives you the code. And so I think it's... This is a very high information bandwidth to specify what you want, is h- if you point to the code where you want it, and you type out the first few pieces-

Dwarkesh Patel 34:09 Yeah.

Andrej Karpathy 34:10 ... and the model will complete it. So, I guess what I mean is, um, I think these models are good in certain parts of the stack. Actually use the models a little bit in, um... There are two examples where I actually use the models that I think are illustrative. Uh, one was when I generate a report. Uh, that's actually more boilerplate-y. So, actually, I've coded part- partially some of that stuff. That was fine, um, because it's not like mission critical stuff-

Dwarkesh Patel 34:33 Mm-hmm.

Andrej Karpathy 34:33 ... and it works fine. And then the other part is when I was rewriting the tokenizer, uh, in Rust. Uh, I'm actually not as good at Rust because I'm fairly new to Rust. So, I was doing... There's a bit of vibe coding going on, uh, in, when I was writing some of the Rust code. But I had Python implementation that I fully understand, and I'm just making sure I'm making more efficient version of it-

Dwarkesh Patel 34:50 Mm-hmm.

Andrej Karpathy 34:50 ... and I have tests, so I feel safer doing that stuff. Um, and so basically, they lower or, like, they increase accessibility to, uh, languages or paradigms that you might not as be- not be as familiar with. Uh, so I think they're very helpful there as well.

Dwarkesh Patel 35:04 Yeah.

Andrej Karpathy 35:04 Uh, because there's a ton of Rust code out there. The models are actually pretty good at it. I happen to not know that much about it, so the models are very useful there.

Dwarkesh Patel 35:10 Right. Um, the reason I think this question is so interesting is because the main story people have about AI exploding and getting to super intelligence pretty rapidly is AI automating, AI engineering, and AI research.

Andrej Karpathy 35:24 Mm-hmm.

Dwarkesh Patel 35:25 And so they'll look at the fact that you can have cloud code and make entire applicat- CRUD applications from scratch and be like, "If you had this incapability inside of OpenAI and DeepMind and everything, well-"

Andrej Karpathy 35:33 Mm-hmm.

Dwarkesh Patel 35:33 "... just imagine the level of, like, just, you know, uh, 1,000 of you or a million of you in parallel finding little-"

Andrej Karpathy 35:39 Yeah.

Dwarkesh Patel 35:39 "... architectural tweaks." And so it's quite interesting to hear you say that this is the thing they're sort of asymmetrically worse at.

Andrej Karpathy 35:46 Yeah.

Dwarkesh Patel 35:46 And it's, like, quite relevant to forecasting-

Andrej Karpathy 35:48 Uh-huh.

Dwarkesh Patel 35:48 ... whether the AI 2027 type explosion is likely to happen anytime soon.

Andrej Karpathy 35:53 I think that's a good way of putting it, and I think you're getting at some of my, like, why my timelines are a bit longer. Uh, you're right. Um, I think, um, yeah, they're not very good at code that hasn't ever been written before-

Dwarkesh Patel 36:03 Yeah.

Andrej Karpathy 36:03 ... maybe is, like, one way to put it, which is like what we're trying to achieve when we're building these models (laughs) .

Dwarkesh Patel 36:07 V- very naive question, but, um, the architectural tweaks that you're adding to, uh, NanoChat, they're in a paper somewhere, right? They might even be in a repo somewhere, so-

Andrej Karpathy 36:17 Yeah.

Dwarkesh Patel 36:17 ... it's... Is it, um... Is it surprising that they aren't able to integrate that into th- whenever you're like, "Add rope embeddings or something," they, they do that in the wrong way?

Andrej Karpathy 36:29 It's- it's tough. I think they kind of know... They kind of know, but they don't fully know, and they don't know how to fully integrate it-

Dwarkesh Patel 36:34 Mm-hmm.

Andrej Karpathy 36:34 ... into the repo and your style and your code and your place, and some of the custom things that you're doing-

Dwarkesh Patel 36:38 Yeah.

Andrej Karpathy 36:38 ... and, and, uh, how it fits with all the assumptions of the repository and all this kind of stuff. So, I think they do have some knowledge, but, um, they haven't gotten to the place where they can actually integrate it, make sense of it, uh, and so on. I do think that a lot of this stuff, by the way, continues to improve. So, um, I think currently, probably state-of-the-art model that I go to is the GPT-5 Pro.

Dwarkesh Patel 36:57 Mm-hmm.

Andrej Karpathy 36:57 Um, and, uh, that's a very, very powerful model. So, if I actually have 20 minutes, I will copy-paste my entire repo and I go to GPT-5 Pro, the Oracle-... for, like, some questions, and-

Dwarkesh Patel 37:05 Yeah.

Andrej Karpathy 37:06 ... often it's not too bad, and surprisingly good compared to what existed a year ago.

Dwarkesh Patel 37:09 Yeah.

Andrej Karpathy 37:10 Um, but I do think that, uh, overall the models are, are, um, they're not there. And I kind of feel like the industry, it's, it's, um, it's over... It's, it's making too big of a jump.

Dwarkesh Patel 37:21 Mm-hmm.

Andrej Karpathy 37:21 And it's trying to pretend like this is amazing, and it's not. It's slop. And I think they're not coming to terms with it, and maybe they're trying to fundraise or something like that. I'm not sure what's going on. But it's, we're at this intermediate stage. The models are amazing. They still need a lot of work. For now, autocomplete is my sweet spot. (laughs)

Dwarkesh Patel 37:38 Mm-hmm.

Andrej Karpathy 37:38 But sometimes for t- some types of code, I will go to a knowledge agent.

Dwarkesh Patel 37:41 Yeah.

Andrej Karpathy 37:41 Yeah.

Dwarkesh Patel 37:42 A- and see, this, this is also, here's another reason why this is very interesting. Um, through the history of programming, there's been many productivity improvements, compilers, linting, better programming languages, et cetera, which have increased pr- programmer productivity-

Andrej Karpathy 37:57 Yeah.

Dwarkesh Patel 37:57 ... but have not led to an explosion.

Andrej Karpathy 37:59 Mm-hmm.

Dwarkesh Patel 37:59 So that's like one... That sounds very much like a- a- autocomplete tab.

Andrej Karpathy 38:03 Mm-hmm. Yeah.

Dwarkesh Patel 38:03 And this other category is just like automation of the programmer.

Andrej Karpathy 38:07 Yeah.

Dwarkesh Patel 38:07 And it's just interesting you're seeing more in the category of the historical analogies of like-

Andrej Karpathy 38:11 Yeah.

Dwarkesh Patel 38:11 ... uh, you know, better compilers or something.

Andrej Karpathy 38:13 Yeah. I mean, because this gets at one other kind of thought of that is like, I do feel like I have a hard time differentiating where AI begins and stops.

Dwarkesh Patel 38:20 Mm-hmm.

Andrej Karpathy 38:20 Because I do see AI as fundamentally an extension of computing in some, in some pretty fundamental way. And I, I feel like I see a continuum of this kind of like recursive self-improvement, uh, or like of speeding up, uh, programmers all the way from the beginning. Like, even like, I would say like, uh, code editors.

Dwarkesh Patel 38:35 Yeah.

Andrej Karpathy 38:35 Um, uh, syntax highlighting.

Dwarkesh Patel 38:38 Yeah.

Andrej Karpathy 38:38 Uh, syntax, uh, or like checking even of the, of the types-

Dwarkesh Patel 38:42 Yeah.

Andrej Karpathy 38:42 ... like data type checking. Um, all these kinds of tools that we've built for each, uh, for each other, even search engines. Like, why aren't search engines part of AI?

Dwarkesh Patel 38:49 Mm-hmm.

Andrej Karpathy 38:50 Like, I don't know, like, uh, ranking is kind of AI, right? At some point Google was like, even early on, they were thinking of themselves as an AI company doing-

Dwarkesh Patel 38:56 Right.

Andrej Karpathy 38:56 ... Google Search Engine, which I think is totally fair. And so I kind of see it as a lot more of a continuum than I think other people do, and I don't... It's hard for me to draw the line, and I kind of feel like, okay, we're now getting a much better autocomplete, and now we're also getting some agents which are kind of like these loopy things, but they kind of go off rails sometimes. Um, and what's going on is that the human is progressively doing a bit less and less of the low level stuff. For example, we're not writing the assembly code, because we have compilers.

Dwarkesh Patel 39:20 Yeah.

Andrej Karpathy 39:20 Right? Like, compilers will take my high level language in C and write the assembly code.

Dwarkesh Patel 39:23 Yeah.

Andrej Karpathy 39:23 So we're abstracting ourselves very, very slowly. And there's this what I call autonomy slider of, like, more and more stuff is automated of the stuff that can be automated at any point in time. And we're doing a bit less and less and, uh, raising ourselves in the later abstraction over the automation.

Dwarkesh Patel 39:37 One of the big problems with RL is that it's incredibly information sparse. LabelBox can help you with this by increasing the amount of information that your agent gets to learn from with every single episode. For example, one of their customers wanted to train a coding agent, so LabelBox augmented an IDE with a bunch of extra data collection tools and staffed a team of expert software engineers from their aligner network to generate trajectories that were optimized for training. Now, obviously these engineers evaluated these interactions on a pass/fail basis, but they also rated every single response on a bunch of different dimensions, like readability and performance. And they wrote down their thought processes for every single rating that they gave. So, you're basically showing every single step an engineer takes and every single thought that they have while they're doing their job. And this is just something you could never get from usage data alone. And so LabelBox packaged up all these evaluations and included all the agent trajectories and the corrective human edits for the customer to train on. This is just one example. So go check out how LabelBox can get you high quality frontier data across domains, modalities, and training paradigms. Reach out at labelbox.com/thwarkash.

Dwarkesh Patel 40:53 Let's talk about RL a bit.

Andrej Karpathy 40:55 Mm-hmm.

Dwarkesh Patel 40:55 Uh, you too did some very interesting things about this. Um, conceptually, how should we think about the way that humans are able to build a rich world model just from interacting with our environment and in ways that seems almost irrespective of the final reward at the end of the episode?

Andrej Karpathy 41:12 Mm-hmm.

Dwarkesh Patel 41:13 If somebody ha- you know, somebody's starting to start a business, and at the end of 10 years she finds out whether the business succeeded or failed, we say that she's earned a bunch of wisdom and experience.

Andrej Karpathy 41:21 Mm-hmm. Yeah.

Dwarkesh Patel 41:22 But it's not because, like, the log probs of every single thing that happened over the last 10 years-

Andrej Karpathy 41:26 Yeah.

Dwarkesh Patel 41:26 ... are upweighted or downweighted. Something much more deliberate and, uh, rich is happening.

Andrej Karpathy 41:30 Yeah.

Dwarkesh Patel 41:30 How... What is the ML analogy a- and how does that compare to what we're doing with LLMs right now?

Andrej Karpathy 41:34 Yeah, maybe the way I would put it is, uh, humans don't use reinforcement learning is maybe what I've- (laughs)

Dwarkesh Patel 41:38 Hmm.

Andrej Karpathy 41:38 ... as I've said it all. I, I think they do something different, which is, yeah, you experience... So reinforcement learning is a lot worse than I think the average person thinks. (laughs)

Dwarkesh Patel 41:45 (laughs) Reinforcement learning is terrible. (laughs)

Andrej Karpathy 41:49 (laughs)

Dwarkesh Patel 41:49 (laughs)

Andrej Karpathy 41:50 It just so happens that, uh, everything that we had before it is much worse. (laughs)

Dwarkesh Patel 41:54 (laughs)

Andrej Karpathy 41:55 Uh, because previously we were just imitating people, so it has all these issues. Um, so in reinforcement learning, say you're working with, uh, you're solving a math problem, because it's very simple. You're given a math problem and you're trying to find the solution. Um, now in reinforcement learning, you will try, uh, lots of things in parallel first. So, uh, you're given a problem. You try hundreds of different attempts. And these attempts can be complex, right? They can be like, "Oh, let me try this. Let me try that. This didn't work. That didn't work," et cetera. And then maybe you get an answer. And now you check the back of the book and you see, okay, the correct answer is this. And then you can see that, okay, this one, this one, and that one got the correct answer, but these other 97 of them didn't. So literally what reinforcement learning does is it goes to the ones that worked really well, and every single thing you did along the way-

Dwarkesh Patel 42:39 Yeah.

Andrej Karpathy 42:39 ... every single token gets upweighted of, like, do more of this. The problem with that is, uh, I mean, people will say that, uh, your estimator has high variance, but what... I mean, it's just noisy. It's noisy. (laughs) Uh, so basically, it kind of almost assumes that every single little piece of the solution that you made that arrived at the right answer was correct thing to do, which is not true. Like, you may have gone down the wrong alleys, uh, until you arrived at the right solution. Every single one of those incorrect things you did, as long as you got to the correct solution, will be upweighted as do more of this. It's terrible.

Dwarkesh Patel 43:06 ... yeah.

Andrej Karpathy 43:06 It's noise. You've done all this work only to find a single... at the end you get a single number of like, oh, you did correct. And - and based on that, you weigh that entire trajectory as like upweight or downweight. And so you're... the way I like to put it is, you're sucking supervision through a straw. Uh, because you've done all this work, that could be minutes of rollout, and you're - you're like sucking the bits of supervision of the final reward signal-

Dwarkesh Patel 43:28 (laughs)

Andrej Karpathy 43:28 ... through a straw, and you're like putting it... you're like... (laughs) you're basically like, um... yeah, you're broadcasting that across the entire trajectory and using that to upweight or downweight that trajectory. It's just stupid and crazy.

Dwarkesh Patel 43:38 Uh-

Andrej Karpathy 43:38 A human would never do this. Number one, a human would never do hundreds of rollouts, right?

Dwarkesh Patel 43:42 Right.

Andrej Karpathy 43:42 Uh, number two, when a person sort of finds a solution, they will have a pretty complicated process of review, of like, "Okay, I think these parts that I did well. These parts I did not do that well. I should probably do this or that." And they think through things. There's nothing in current LLMs that does this. There's no equivalent of it. Um, but I do see papers popping out that are trying to do this, because it's obvious to everyone in the field.

Dwarkesh Patel 44:03 Yeah.

Andrej Karpathy 44:04 So I kind of see as like the first imitation learning actually, by the way, was extremely surprising and miraculous and amazing that we can, uh, fine-tune by imitation in humans. Um, and that was incredible. Because in the beginning all we had was base models. Base models are autocomplete. Uh, and it wasn't obvious to me at the time, uh, and I had to learn this in the paper that like blew my mind was InstructGPT, because it pointed out that, hey, you can train the pretrained model, which is autocomplete, and if you just fine-tune it on text that looks like conversations, the model will very rapidly adapt to become, uh, very conversational, and it keeps all the knowledge from pretraining. And this blew my mind, because I didn't understand that this just like stylistically can adjust so quickly and become an assistant to a user, through- through just a few loops of fine-tuning on that kind of data. It was very miraculous to me that - that that worked. So incredible, and that was like two years, three years of work. And now came RL. And RL allows you to do a bit better than just imitation learning, right? Because you - you can have these re- um, reward functions and you can hill climb on the reward functions. And so some problems have just correct answers, you can hill climb on that without getting expert trajectories to imitate. So that's amazing. And the model can also discover solutions that the human might never come up with.

Dwarkesh Patel 45:11 Mm-hmm.

Andrej Karpathy 45:12 Uh, so this is incredible, and yet it's still stupid. (laughs)

Dwarkesh Patel 45:15 (laughs)

Andrej Karpathy 45:15 Um, so I think we need... we need more. And so I saw a paper from Google yesterday that tried to have this reflect and review pa- um, uh, idea, uh, in mind. Uh, what was the... memory bank paper or something? I don't know. Uh, I've actually seen a few papers along these lines. So I expect there to be some kind of a major update to how we do algorithms for LLMs, uh, coming in that realm. And then I think we need three or four or five more. (laughs)

Dwarkesh Patel 45:40 (laughs)

Andrej Karpathy 45:40 Um, something like that.

Dwarkesh Patel 45:42 But you- you're- you're so good at coming up with evocative, evocative phrases. "Sucking supervision through a straw" is like so good.

Andrej Karpathy 45:49 (laughs)

Dwarkesh Patel 45:49 Um, why hasn't... so you're saying like your problem with outcome based reward is that you have this huge trajectory, and then at the end you're - you're trying to learn every single possible thing about what you should do and what you should learn about the world from that one final bit.

Andrej Karpathy 46:05 Mm-hmm.

Dwarkesh Patel 46:06 Um, why hasn't... uh, given the fact that this is obvious, why hasn't process-based supervision, as an alternative, been a successful way to make models-

Andrej Karpathy 46:13 Yeah.

Dwarkesh Patel 46:13 ... more capable? What- what has been preventing-

Andrej Karpathy 46:15 Yeah.

Dwarkesh Patel 46:15 ... us from using this alternative paradigm?

Andrej Karpathy 46:16 So process-based supervision just refers to the fact that we're not going to have a reward function only at the very end of...

Dwarkesh Patel 46:20 Yeah.

Andrej Karpathy 46:20 ... after you've made 10 minutes of work-

Dwarkesh Patel 46:22 Yeah.

Andrej Karpathy 46:22 ... I'm not going to tell you if you did well or not well.

Dwarkesh Patel 46:24 Yeah.

Andrej Karpathy 46:24 I'm going to tell you at every single step of the way how well you're doing. Um, and this is basically... the reason we don't have that is it's not trick- it's tricky how you do that properly.

Dwarkesh Patel 46:31 Hmm.

Andrej Karpathy 46:31 Um, because you have partial solutions and you don't know how to assign credit. So when you get the right answer, it's just, uh, an equality match to the answer. Very simple to implement.

Dwarkesh Patel 46:39 Yeah.

Andrej Karpathy 46:40 If you're doing basically process supervision, how do you assign, in an automatable way, partial credit assignment? It's not obvious how you do it. Lots of labs, I think, are trying to do it with these LLM judges.

Dwarkesh Patel 46:50 Yeah.

Andrej Karpathy 46:50 So basically you get LLMs to try to do it. So you prompt an LLM, "Hey, look at a partial solution of a student. How well do you think they're doing if the answer is this?"

Dwarkesh Patel 46:57 Yeah.

Andrej Karpathy 46:57 And they try to tune the prompt. Um, the reason that I think this is kind of tricky is quite subtle, and it's the fact that any time you use an LLM to assign a reward, those LLMs are giant things with billions of parameters, and they're gameable. And if you're reinforcement learning with respect to them, you will find adversarial examples-

Dwarkesh Patel 47:13 Interesting.

Andrej Karpathy 47:13 ... for your LM judges, almost guaranteed. You can't do this for too long. You do maybe 10 steps or 20 steps, maybe it will work. But you can't do 100 or 1,000, because it's not obvious. Uh, because, um... I know... I understand it's not obvious, but basically the model will find little cracks, you know... you know, it will find all these like spurious things in the nooks and crannies of the giant model and find a way to cheat it. So one example, uh, that's prominently in my mind is... I- I think this, I think this was probably public. Uh, but basically, if you're using an LM judge for a reward, so you just give it a solution from a student and ask it if the solu- student did well or not. We were training with reinforcement learning against that reward function, and it worked really well, and then, um, suddenly the reward became extremely large.

Dwarkesh Patel 47:55 Hmm.

Andrej Karpathy 47:55 Like it was a massive jump and it did perfect. And you're looking at it like, wow, this -this means the student is perfect in all these problems. It's fully solved math. But actually what's happening is that when you look at the completions that you're getting from the model, they are complete nonsense. They start out okay, and then they change to duh-duh-duh-duh-duh-duh-duh.

Dwarkesh Patel 48:11 Hmm.

Andrej Karpathy 48:11 So it's just like, oh, okay, let's take two plus three and we do this and this and then duh-duh-duh-duh-duh-duh-duh.

Dwarkesh Patel 48:15 Hmm.

Andrej Karpathy 48:15 And you're looking at it, and it's like, this is crazy. How is it getting a reward of one or 100%? Um, and you look at the LM judge, and it turns out that duh-duh-duh-duh-duh is an adversarial examples for the model, and it assigns 100% probability to it. And it's just because this is an out of sample example to the LLM.

Dwarkesh Patel 48:30 Yeah.

Andrej Karpathy 48:30 It's never seen it during training, and you're in pure generalization land.

Dwarkesh Patel 48:33 Right.

Andrej Karpathy 48:34 It's never seen it during training. And in the pure generalization land, you can find these examples that, that, uh, break it.

Dwarkesh Patel 48:40 You're- you're basically training the LLM to be a prompt injection model. Um-

Andrej Karpathy 48:44 Not even that. Prompt injection is way too fancy. You're, you're finding adversarial examples, as they're called.

Dwarkesh Patel 48:48 Right.

Andrej Karpathy 48:49 These are nonsensical, uh, solutions, um, that are obviously wrong, but the model thinks are amazing. (laughs)

Dwarkesh Patel 48:55 So to the extent you think this is the bottleneck to making RL more functional, then that will require making LLMs better judges, if you want to do this-

Andrej Karpathy 49:03 Yeah.

Dwarkesh Patel 49:03 ... in an automated way. And then so, uh, is it just going to be like some sort of GAN-like approach, where you got to train models to be more robust-

Andrej Karpathy 49:09 Yeah.

Dwarkesh Patel 49:09 ... to...

Andrej Karpathy 49:09 I think the labs are probably doing all that. Like, okay, so the obvious thing is like duh-duh-duh should not get 100% reward.

Dwarkesh Patel 49:14 Yeah.

Andrej Karpathy 49:14 Okay, well, take duh-duh-duh-duh, put it in the training set of the LM judge, and say, "This is not 100%. This is 0%."

Dwarkesh Patel 49:19 Yeah.

Andrej Karpathy 49:19 You can do this. But every time you do this, you get a new LLM, and it still has adversarial examples.

Dwarkesh Patel 49:24 Yeah.

Andrej Karpathy 49:24 There's infinity adversarial examples.

Dwarkesh Patel 49:26 Yeah.

Andrej Karpathy 49:26 And I think probably if you iterate this a few times, it'll probably be harder and harder to find adversarial examples.

Dwarkesh Patel 49:31 Hmm.

Andrej Karpathy 49:31 But I'm not 100% sure, because this thing has a trillion parameters or whatnot. Um, so I bet you the, the labs are trying. Uh, I don't actually... I s- I still think...... I still think we need other ideas. (laughs)

Dwarkesh Patel 49:44 Interesting. Do, do you have some shape of what the other ideas could be?

Andrej Karpathy 49:49 Um, so, like, this, uh, this idea of, like, a re- review, um-

Dwarkesh Patel 49:53 Yeah.

Andrej Karpathy 49:53 ... review a solution and come up with synthetic examples, such that-

Dwarkesh Patel 49:56 Yeah.

Andrej Karpathy 49:56 ... when you train on them, you get, uh, you get better. And, like, meta-learn it in some way.

Dwarkesh Patel 50:00 Yeah.

Andrej Karpathy 50:00 And I think there's some papers that I'm starting to see pop out. I only am at a stage of, like, reading the abstracts, because a lot of these papers, you know, they're just ideas. Someone has to actually, like, make it work on a frontier LLM lab scale, uh, in full generality. Because when you see these papers, they pop up and it's just, like, a little bit of noisy, you know? It's cool ideas, but I haven't actually seen anyone convincingly, uh, show that this is possible. That said, the LLM labs are fairly closed. Uh, so who knows what they're doing now.

Dwarkesh Patel 50:24 Mm-hmm.

Andrej Karpathy 50:24 But yeah.

Dwarkesh Patel 50:26 So I guess, like, I, I, I see a very, um, not easy, but, like, I, I can conceptualize how you would ha- be able to train on synthetic examples.

Andrej Karpathy 50:34 Mm-hmm.

Dwarkesh Patel 50:34 Or synthetic problems that you have made for yourself.

Andrej Karpathy 50:36 Mm-hmm.

Dwarkesh Patel 50:36 But there seems to be another thing humans do, maybe sleep is this, maybe daydreaming is this.

Andrej Karpathy 50:40 Mm-hmm.

Dwarkesh Patel 50:41 Which is not necessarily come up with fake problems, but just, like, reflect.

Andrej Karpathy 50:46 Yeah.

Dwarkesh Patel 50:46 And I'm not sure what the ML analogy for, you know, daydreaming or sleeping.

Andrej Karpathy 50:50 Mm-hmm. Yeah.

Dwarkesh Patel 50:50 But just, like, just reflecting, I haven't come up with any problem.

Andrej Karpathy 50:52 Yeah, yeah.

Dwarkesh Patel 50:52 I mean, obviously, the very basic analogy would just be, like, fine-tuning on reflection bits. But I feel like in practice, that probably wouldn't work that well.

Andrej Karpathy 50:59 Yeah.

Dwarkesh Patel 50:59 So I don't know if you have some take on what, what the analogy of, like, th- this thing is.

Andrej Karpathy 51:05 Yeah, I do think that, that we're missing some aspects there. So as an example, uh, when you're reading a book-

Dwarkesh Patel 51:09 Yeah.

Andrej Karpathy 51:09 ... um, I almost feel like currently, when LLMs are reading a book, uh, what that means is we stretch out the sequence of text and the model is predicting the next token.

Dwarkesh Patel 51:17 Yeah.

Andrej Karpathy 51:17 And it's getting some knowledge from that. Uh, that's not really what humans do, right?

Dwarkesh Patel 51:20 Yeah.

Andrej Karpathy 51:20 So when you're reading a book, I almost don't even feel like the book is like exposition I'm supposed to be attending to and training on.

Dwarkesh Patel 51:25 Mm-hmm.

Andrej Karpathy 51:25 The book is a, is a set of prompts for me to do-

Dwarkesh Patel 51:28 Mm-hmm.

Andrej Karpathy 51:28 ... synthetic data generation, or for you to get into a book club and talk about it with your friends.

Dwarkesh Patel 51:32 Yeah.

Andrej Karpathy 51:33 And it's by manipulating that information that you actually gain that knowledge.

Dwarkesh Patel 51:36 Yeah.

Andrej Karpathy 51:37 And I, I think we have no equivalent of that, again, with LLMs. They don't really do that, but I'd love to see, during pre-training, some kind of a stage that, uh, thinks through the material and tries to reconcile it with what it already knows, and thinks through for, like, some amount of time and, um, gets that to work. And so there's no equivalence of any of this. This is all research. There's some subtle, very subtle, that I think are very hard to understand reasons why it's not trivial. So if I can just describe one-

Dwarkesh Patel 52:01 Degrease.

Andrej Karpathy 52:01 ... why can't we just synthetically generate and train on it?

Dwarkesh Patel 52:03 Mm-hmm.

Andrej Karpathy 52:04 Well, because every synthetic example, like, if I just give synthetic generation of the model thinking about a book, you look at it and you're like, "This looks great. Why can't I train on it?" Well, you could try, but the model will actually get much worse if you continue trying. And that's because all of the samples you get from m- models are silently collapsed. They're silently, this is not obvious, if you look at any individual example of it, they occupy a very tiny manifold of the possible space of, um, sort of thoughts about content. So the LLMs, when they come off, uh, they, they're what we call collapsed. They have a collapsed data distribution. If you sample... Uh, one easy way to say it is go to ChatGPT and ask it, "Tell me a joke." It only has, like, three jokes.

Dwarkesh Patel 52:39 (laughs)

Andrej Karpathy 52:40 It's not giving you the whole breadth of possible jokes.

Dwarkesh Patel 52:42 Yeah.

Andrej Karpathy 52:42 It's giving you, like... It knows, like, three jokes.

Dwarkesh Patel 52:44 Yeah.

Andrej Karpathy 52:44 Uh, they're soundly collapsed. So basically, you're not getting the richness and the diversity and the entropy, uh, from these models as you would get from humans. So humans are a lot more sort of noisier, but at least they're not biased. They're not, um, in, in a statistical sense, they're not silently collapsed. They maintain a huge amount of entropy. So how do you get synthetic data generation to work despite the collapse and while maintaining the entropy, is a research problem.

Dwarkesh Patel 53:07 Um, just to make sure I understood, the reason that the collapse is relevant to synthetic data generation is because you want to be able to come up with synthetic problems or reflections which are not already in your data distribution?

Andrej Karpathy 53:20 I guess what I'm saying is, um, say we have a chapter of a book and I ask an LLM to think about it.

Dwarkesh Patel 53:25 Yeah.

Andrej Karpathy 53:25 Um, it will give you something that looks very reasonable.

Dwarkesh Patel 53:28 Yeah.

Andrej Karpathy 53:28 But if I ask it 10 times, you'll notice that all of-

Dwarkesh Patel 53:31 Ah.

Andrej Karpathy 53:31 ... them are the same.

Dwarkesh Patel 53:32 You can't just see if scaling-

Andrej Karpathy 53:34 Yeah.

Dwarkesh Patel 53:34 ... sca- scaling, quote unquote, "reflection"-

Andrej Karpathy 53:36 Yeah.

Dwarkesh Patel 53:36 ... on the same amount of-

Andrej Karpathy 53:38 Yeah.

Dwarkesh Patel 53:38 ... uh, you know, prompt information and then-

Andrej Karpathy 53:40 Yeah.

Dwarkesh Patel 53:40 ... get returns from that, okay.

Andrej Karpathy 53:41 Yeah, yeah, yeah. So any individual sample will look okay, but the distribution of it is, is quite terrible.

Dwarkesh Patel 53:46 Interesting.

Andrej Karpathy 53:46 And it's quite terrible in such a way that if you continue training on too much of your own stuff, you actually collapse. I actually think that, um, there's no, like, fundamental solutions to this possibly.

Dwarkesh Patel 53:54 Hmm.

Andrej Karpathy 53:54 And I also think humans collapse over time.

Dwarkesh Patel 53:55 Mm-hmm.

Andrej Karpathy 53:56 Uh, I think this is, uh, again, these analogies are surprisingly good, but humans collapse during the course of their lives. This is why children, uh, have, uh, completely, um, you know, they haven't overfit yet.

Dwarkesh Patel 54:05 Yeah.

Andrej Karpathy 54:05 And they will say stuff that will shock you because it's kind of you can see where they're coming from-

Dwarkesh Patel 54:09 Yeah.

Andrej Karpathy 54:09 ... but it's just not the thing people say-

Dwarkesh Patel 54:11 Yeah.

Andrej Karpathy 54:12 ... and because they are not yet collapsed. But we're collapsed, we end up revisiting the same thoughts. We end up r- s- you know, saying more and more of the same stuff, and the learning rates go down and, uh, the collapse continues to get worse, and then, um, yeah, everything deteriorates. (laughs)

Dwarkesh Patel 54:27 Have, have you seen this super interesting paper that dreaming is a way of preventing this kind of overfitting and collapse?

Andrej Karpathy 54:33 Huh.

Dwarkesh Patel 54:33 That the reason dreaming is, uh, evolutionary-

Andrej Karpathy 54:37 Yeah.

Dwarkesh Patel 54:37 ... adaptive is to put you in weird situations-

Andrej Karpathy 54:40 Mm-hmm.

Dwarkesh Patel 54:40 ... that are, like, uh, very unlike your day-to-day reality-

Andrej Karpathy 54:43 Yeah.

Dwarkesh Patel 54:43 ... so that, to prevent this kind of overfitting?

Andrej Karpathy 54:44 It's an interesting idea. I mean, I do think that when you're generating things in your head and then you're attending to it, you're kind of like training on your own samples.

Dwarkesh Patel 54:51 Yeah.

Andrej Karpathy 54:51 You're training on your synthetic data and if you do it for too long, you go off rails.

Dwarkesh Patel 54:54 Yeah.

Andrej Karpathy 54:54 Um, and you collapse way too much. So you always have to, like, seek, um, entropy in your life. (laughs)

Dwarkesh Patel 55:00 Yeah.

Andrej Karpathy 55:00 Uh, so talking to other people is a great source of entropy and, uh, things like that. So maybe the brain has also built some internal mechanisms, uh, for increasing the amount of entropy, um, uh, in, in that process. But yeah, maybe that's an interesting idea.

Dwarkesh Patel 55:14 This is a very ill-formed thought, so I'll, I'll just put it out and l- let you react to it. The best learners that we are aware of, which are children, are extremely bad at recollecting information. In fact, at the very earliest stages of childhood, you will forget everything. You're just an amnesiac about everything that happens before a certain, uh, year date. But you're, like, extremely good at picking up new languages and learning-

Andrej Karpathy 55:35 Mm-hmm.

Dwarkesh Patel 55:36 ... from the world.

Andrej Karpathy 55:36 Mm-hmm.

Dwarkesh Patel 55:36 And maybe there's some element of, like, being able to see the forest for the trees.

Andrej Karpathy 55:38 Mm-hmm.

Dwarkesh Patel 55:39 Whereas if you compare it to the ex- opposite end of the spectrum, you have-... LLM pre-training, which these models will literally regurgitate, word for word, what is the next thing-

Andrej Karpathy 55:48 Yeah.

Dwarkesh Patel 55:48 ... in a Wikipedia page.

Andrej Karpathy 55:50 Yeah.

Dwarkesh Patel 55:50 But their ability to learn abstract concepts really quickly, the way a child can, is much more limited. And then adults are somewhere in between, where they don't have the flexibility of childhood learning-

Andrej Karpathy 56:00 Mm-hmm.

Dwarkesh Patel 56:00 ... but they can, you know, adults can memorize facts and information in a way that is harder for kids.

Andrej Karpathy 56:05 Yeah.

Dwarkesh Patel 56:05 And I don't know if there's something interesting about that-

Andrej Karpathy 56:08 I think there's something very interesting about that.

Dwarkesh Patel 56:09 ... spectrum, yeah.

Andrej Karpathy 56:09 Yeah, 100%. I do think that humans actually, um, they do kind of like have a lot more of an element, compared to LLMs-

Dwarkesh Patel 56:16 Yeah.

Andrej Karpathy 56:16 ... of like seeing the forest for the trees.

Dwarkesh Patel 56:17 Yeah.

Andrej Karpathy 56:17 And, and we're not actually that good at memorization, which is actually a feature.

Dwarkesh Patel 56:21 Mm-hmm.

Andrej Karpathy 56:22 Um, because we're not that g- good at memorization, uh, we actually are kind of like forced to, uh, find the patterns, uh, um, like in a more, in a more general sense. I think LLMs for, in comparison, are extremely good at memorization. They will recite passages from all these, uh, training sources. Uh, you can give them completely nonsensical data. Like you can take, um, you can hash some amount of text or something like that. You get a completely random sequence. If you train on it, even just, I think, a single iteration or two, it can suddenly regurgitate the entire thing.

Dwarkesh Patel 56:49 Yeah.

Andrej Karpathy 56:49 It will memorize it. There's no way a person can read a single sequence of random numbers and recite it to you.

Dwarkesh Patel 56:54 No.

Andrej Karpathy 56:54 Um, and that's a feature, not a bug, almost, uh, because it forces you to like only learn their generalizable components, whereas LLMs are distracted by all the memory that they have of the pre-train documents, and it's probably very distracting to them, uh, in a certain sense. So that's why when I talk about the cognitive core, I actually want to remove the memory, which is what we talked about. I'd love to have it, them have less memory so that they have to look things up, uh, and that they only maintain the algorithms for, like thought, uh, and the idea of an experiment and all this cognitive glue of, um, of acting.

Dwarkesh Patel 57:26 And this is also relevant to preventing model collapse.

Andrej Karpathy 57:29 Um, let me think. Um... I'm not sure. I think it's almost like a separate axis.

Dwarkesh Patel 57:37 Mm-hmm.

Andrej Karpathy 57:37 It's almost like the mo- the models are way too good at, uh, memorization, and somehow we should, we should remove that. And I think people, people are much worse, but it's a good thing. (laughs)

Dwarkesh Patel 57:45 Hmm. What is a solution to model collapse? I mean, you could... Okay, so there's very naive things you could attempt as just like, um, the distribution over logis should be-

Andrej Karpathy 57:54 Yeah.

Dwarkesh Patel 57:54 ... wider or something.

Andrej Karpathy 57:55 Yeah.

Dwarkesh Patel 57:55 Like there's many naive things you could try. What ends up being the problem with the naive approaches?

Andrej Karpathy 58:00 Um, yeah, I think that's a great question. I mean, you can imagine having a regularization for entropy and-

Dwarkesh Patel 58:04 Yeah.

Andrej Karpathy 58:04 ... things like that. I guess they just, they don't work as well empirically because, uh, right now, like the models are collapsed. But I will say, um, most of the tasks that we want of them don't actually demand the diversity.

Dwarkesh Patel 58:16 Mm-hmm.

Andrej Karpathy 58:16 It's probably the, the answer of what's going on. And so it's just that the mod- the frontier labs are trying to make the models useful, and I kind of just feel like the diversity of the outputs is not so much... Number one, it's much harder to work with and evaluate and all this kind of stuff, but maybe it's not what's actually capturing most of the value.

Dwarkesh Patel 58:31 Yeah.

Andrej Karpathy 58:31 Um...

Dwarkesh Patel 58:32 In fact, it's actively penalized, right?

Andrej Karpathy 58:33 Yeah.

Dwarkesh Patel 58:33 If you, if you're like super creative in RL-

Andrej Karpathy 58:36 Yeah.

Dwarkesh Patel 58:36 ... it's like not good.

Andrej Karpathy 58:37 Yeah. Or like maybe if you're doing a lot of writing help from LLMs-

Dwarkesh Patel 58:40 Yeah.

Andrej Karpathy 58:40 ... and stuff like that, I think it's probably bad because the models will give you these like silently, all the same stuff, you know. Uh, so they're not, um, they won't explore lots of different ways of answering a question, right?

Dwarkesh Patel 58:50 Yeah.

Andrej Karpathy 58:50 But I kind of feel like maybe this diversity is just not as big of a... Yeah, maybe like, yeah, not as many applications need it, so the models don't have it, but then it's actually a problem with synthetic generation time, et cetera. So we're actually shooting ourselves in the foot by not allowing this entropy to maintain in the model. And I think, uh, possibly, uh, the labs should try harder. (laughs)

Dwarkesh Patel 59:06 And then I think you hinted that it's a, it's a very fundamental problem.

Andrej Karpathy 59:10 Mm-hmm.

Dwarkesh Patel 59:10 It won't be easy to solve. And yeah, what's your intuition for that?

Andrej Karpathy 59:14 I don't actually know if it's, um, super fundamental. Uh, I don't actually know if I intended to s- to say that. I do think that, um... I haven't done these experiments, but I do think that you could probably regularize the entropy to be, uh, to be higher.

Dwarkesh Patel 59:26 Mm-hmm.

Andrej Karpathy 59:26 So you're encouraging the model to give you more and more solutions. Um, but you don't want it to start deviating too much from the training data. It's gonna start making up its own language. It's gonna start using words that are extremely rare. Uh, you know, so it's gonna drift too much from the distribution. Uh, so I think controlling the distribution is just like a tricky... It's just like someone just has to... It's probably not trivial in that sense.

Dwarkesh Patel 59:46 Mm-hmm. How many bits should the optimal core of-

Andrej Karpathy 59:52 Mm-hmm.

Dwarkesh Patel 59:52 ... intelligence end up being, if you just had to make a guess?

Andrej Karpathy 59:54 Yeah. That's a good question.

Dwarkesh Patel 59:54 The thing we put on the, uh, von Neumann probes.

Andrej Karpathy 59:57 Yeah, yeah, yeah.

Dwarkesh Patel 59:58 How big does it have to be?

Andrej Karpathy 59:59 (laughs) Uh, so it's really interesting in the history of the field because at one point everything was very, um, scaling pilled-

Dwarkesh Patel 1:00:05 Yeah.

Andrej Karpathy 1:00:05 ... in terms of like, oh, we're gonna make much bigger models, trillions of parameter models.

Dwarkesh Patel 1:00:08 Yeah.

Andrej Karpathy 1:00:08 And actually what the models have done in size is they've gone up and now as they've actually kind of like actually even come down.

Dwarkesh Patel 1:00:15 Yeah.

Andrej Karpathy 1:00:15 So their models are smaller. And even then, I actually think they memorized way too much. Um, so I think I had a prediction a while back that I almost feel like we can get cognitive cores that are very good at even like a billion, billion parameters. It, it should be all very like... Like if you talk to a billion-parameter model, I think in 20 years, you can actually have a very productive conversation. It thinks, um, and it's a lot more like a human. Uh, but if you ask it some factual question, might have to look it up, but it knows that it doesn't know and it might have to look it up and they will just do all the reasonable things.

Dwarkesh Patel 1:00:43 That, that's actually surprising that you think it will take a billion par- 'cause already we have a billion-parameter models or a couple of billion-parameter models-

Andrej Karpathy 1:00:49 Yeah.

Dwarkesh Patel 1:00:49 ... that are like very intelligent and it's-

Andrej Karpathy 1:00:51 Well, certainly our models are like a trillion parameters, right?

Dwarkesh Patel 1:00:53 That's right.

Andrej Karpathy 1:00:53 But they remember so much stuff, like it's just...

Dwarkesh Patel 1:00:55 Yeah. But I- I'm surprised that in 10 years, given the pace... Okay. We have GPT, OSS 20B, that's way better than GPT-4 original, which was a trillion plus-

Andrej Karpathy 1:01:08 Yeah.

Dwarkesh Patel 1:01:08 ... uh, parameters. So given that trend, I'm actually surprised you think in 10 years the cognitive core is still a billion parameters. I would... Uh, yeah, I'm surprised you're not like-

Andrej Karpathy 1:01:16 Do you think it's any more?

Dwarkesh Patel 1:01:16 ... it's gonna be like, uh, tens of millions or millions.

Andrej Karpathy 1:01:20 No, because I basically think that the training data is... So here's the issue. The training data is the internet-

Dwarkesh Patel 1:01:23 Yeah.

Andrej Karpathy 1:01:23 ... which is really terrible. (laughs)

Dwarkesh Patel 1:01:25 (laughs)

Andrej Karpathy 1:01:25 So there's a huge amount of gains to be made because the internet is terrible.

Dwarkesh Patel 1:01:28 Yeah.

Andrej Karpathy 1:01:28 Like if you actually... And even the internet, when you and I think of the internet, you're thinking of like, oh, Wall Street Journal or-

Dwarkesh Patel 1:01:33 Yeah.

Andrej Karpathy 1:01:33 ... that's not what this is.

Dwarkesh Patel 1:01:34 (laughs)

Andrej Karpathy 1:01:34 When you're actually looking at a pre-training dataset in the frontier lab and you look at a random internet document, it's total garbage. Like I don't even know how this works at all.

Dwarkesh Patel 1:01:42 (laughs)

Andrej Karpathy 1:01:42 It's some like sto- stock ticker symbols.

Dwarkesh Patel 1:01:45 Yeah.

Andrej Karpathy 1:01:45 Uh, it's a huge amount of slop and garbage from like all the corners of the internet. It's not like your Wall Street Journal article that's extremely rare.Um, so I almost feel like because the internet is so terrible, we actually have to sort of b- almost, like, build really big models to compress all that. Uh, most of that compression is memory work instead of, like, cognitive work.

Dwarkesh Patel 1:02:03 Interesting.

Andrej Karpathy 1:02:04 But what we really want is the cognitive part, actually delete the memory.

Dwarkesh Patel 1:02:06 Right.

Andrej Karpathy 1:02:06 And then, so s- I g- I guess what I'm saying is, like, we need intelligent models to help us refine even the pre-training set to just narrow it down to the cognitive components. And then I think you could get away with a much smaller model, because it's a m- much better data set, and you could train it on it. But probably, it's not trained directly on it. It's probably distilled from a much better model still, but-

Dwarkesh Patel 1:02:24 Right. But w- why is a distilled version still a billion, i- is I guess the thing I'm curious about?

Andrej Karpathy 1:02:28 I just feel like distillation work extremely well. So, um, almost every small model, if you have a small model, it's almost certainly distilled. Why would you train on-

Dwarkesh Patel 1:02:35 Right. No, no, no. But s- wh- why is a distillation not, in 10 years, not getting below one billion?

Andrej Karpathy 1:02:39 Oh, you think it should be smaller than