Richard Sutton on Dwarkesh Patel: Why LLMs Lack a Goal

Richard Sutton 0:00 Why are you trying to distinguish humans? Humans are animals. What we have in common is more interesting. What distinguishes us, we should be paying less attention to.

Dwarkesh Patel 0:08 I mean, we're trying to replicate intelligence, right? No animal can go to the moon or make semiconductors, so we wanna understand what makes humans special.

Richard Sutton 0:14 So, I like the way you consider that obvious, 'cause I consider the opposite obvious. If we understood a squirrel, we'd be almost all the way there. I am personally just kind of content being out of sync with my field for a long period of time, perhaps decades, because occasionally I have been proved right in the past. I don't think learning is really about training. It's about an active process. The child tries things and sees what happens. I think we should be proud that we are giving rise to this great transition in the universe.

Dwarkesh Patel 0:47 Today, I'm chatting with Richard Sutton, who is one of the founding fathers of reinforcement learning, and inventor of many of the main techniques used there, like TD learning and policy gradient methods. And for that, he received this year's Turing Award, which, if you don't know, is basically the Nobel Prize for computer science. Richard, congratulations.

Richard Sutton 1:05 Thank you, Dworkis.

Dwarkesh Patel 1:06 And, uh, thanks for coming on the podcast.

Richard Sutton 1:08 It's my pleasure.

Dwarkesh Patel 1:09 Okay, so first question. My audience and I are familiar with the LLM way of thinking about AI. Conceptually, what are we missing in terms of thinking about AI from the RL perspective?

Richard Sutton 1:21 Well, yes, I think it's really quite a different point of view, and it's, it can easily get separated and lose the ability to talk to each other.

Dwarkesh Patel 1:30 Mm-hmm.

Richard Sutton 1:31 And, um, yeah, large language models have become such a big thing, generative AI in general a big thing, um, and our field is subject to bandwagons and fashions, so we lose, we lose track of the, uh, basic, basic things. 'Cause I consider reinforcement learning to be basic AI, and what is intelligence? Uh, the problem is, is to understand your world.

Dwarkesh Patel 1:54 Right.

Richard Sutton 1:54 And, um, reinforcement learning is about understanding wh- your world, whereas large language models are about mimicking people, doing what people say you should do. They're not about figuring out what to do.

Dwarkesh Patel 2:06 Huh. I guess y- y- you would think that t- uh, to emulate the trillions of tokens in the corpus of internet text, you would have to build a world model. In fact, these models do seem to have very robust world models, and they, they're the best, um, world models we've made to date in AI, right? So, what, what, what do you think that, that's missing?

Richard Sutton 2:25 Uh, I would disagree with most of the things you just said.

Dwarkesh Patel 2:27 (laughs) Great.

Richard Sutton 2:28 (laughs)

Dwarkesh Patel 2:30 (laughs)

Richard Sutton 2:30 Just to mimic the, the, what people say is not really to build a model of the world at all, I don't think. You know, you're mimicking things that have, uh, a model of the world, the people.

Dwarkesh Patel 2:39 Right.

Richard Sutton 2:40 But I don't wanna a- a- approach the question in an adversarial way. Uh, but, but I would, I would question the idea that they, um, they have a world model. So, a world model would enable you to predict what would happen.

Dwarkesh Patel 2:53 Right.

Richard Sutton 2:53 Uh, they, they have, they have the ability to predict what a person would say. They don't have the ability to predict what will happen. What we want, I think, to quote Alan Turing, what we want is a machine that can learn from experience.

Dwarkesh Patel 3:06 Right.

Richard Sutton 3:06 Where experience is the things that actually happen in your life. You do things, you see what happens, um, and, uh, that's what you learn from.

Dwarkesh Patel 3:16 Yeah.

Richard Sutton 3:16 The large language models learn from something else. They learn from, here's a situation, and here's what a person did. And implicitly, the suggestion is you should do what the person did.

Dwarkesh Patel 3:26 Right. I guess maybe the, the crux, and I'm curious if you disagree with this, is some people will say, "Okay, so this imitation learning ha- has given us a good prior, or given these models a good prior, but reasonable ways to approach problems. And as we move towards the era of experience, uh, as you call it, this prior is gonna be the basis on which we teach these models from experience, because this gives them the opportunity to get, uh, answers right some of the time. And then on this, you can build, uh, you can train them on experience." D- do you agree with that perspective?

Richard Sutton 3:59 Uh, no. I, I agree that it's the r- it's the large language model perspective.

Dwarkesh Patel 4:04 Right.

Richard Sutton 4:04 I don't think it's a good perspective.

Dwarkesh Patel 4:06 Yeah.

Richard Sutton 4:06 Yeah.

Dwarkesh Patel 4:06 Curious why.

Richard Sutton 4:07 Um, so to be a prior for something, there has to be a real thing. I mean, the, j- uh, a prior bit of knowledge should be basis for actual knowledge.

Dwarkesh Patel 4:17 Right.

Richard Sutton 4:17 What is actual knowledge? There's no definition of actual knowledge in that, in that large language framework. What makes an action a good action to take? Uh, you recognize the value, the need for continual learning.

Dwarkesh Patel 4:32 Right.

Richard Sutton 4:32 So, if you need to learn continually, w- continually means learning during normal interaction with the world-

Dwarkesh Patel 4:38 Yeah.

Richard Sutton 4:38 ... and so then there must be some way during the normal interaction to tell what's right.

Dwarkesh Patel 4:43 Yup.

Richard Sutton 4:44 Okay. So, is there any way for it to tell, in the large language model setup, to tell what's the right thing to say? You will say something, and you will not get feedback about what the right thing to say is, because there's no definition of what-

Dwarkesh Patel 5:00 That's right.

Richard Sutton 5:01 ... the right thing to say is.

Dwarkesh Patel 5:01 Yeah.

Richard Sutton 5:02 There's no goal.

Dwarkesh Patel 5:03 Right.

Richard Sutton 5:04 And if there's no goal, then there's, there's one thing to say, another thing to say. There's no right thing to say.

Dwarkesh Patel 5:08 Right.

Richard Sutton 5:09 So, there's no ground truth. You can't have prior knowledge if you don't have ground truth, 'cause the prior knowledge is supposed to be a, a hint or an initial belief about what the truth is.

Dwarkesh Patel 5:20 Yeah.

Richard Sutton 5:21 But there isn't any truth. There's no right thing to say, right? Now, in reinforcement learning, there is a right thing to say-

Dwarkesh Patel 5:26 Yeah.

Richard Sutton 5:27 ... or right thing to do-

Dwarkesh Patel 5:28 Right.

Richard Sutton 5:28 ... because the, the right thing to do is the thing that gets you reward.

Dwarkesh Patel 5:31 Right.

Richard Sutton 5:32 So, we have a definition of, of what the right thing to do is, and so we can have, uh, prior knowledge or knowledge provided by pe- people about what the right thing to do is.

Dwarkesh Patel 5:40 Right.

Richard Sutton 5:40 And then we can check it to, to see 'cause, 'cause we have a definition of what the actual right thing to do is.

Dwarkesh Patel 5:46 Yeah.

Richard Sutton 5:47 Now, an even simpler case is when you have, when you're trying to make a model of the world, when you predict what will happen, you predict, and then you see what happens.

Dwarkesh Patel 5:54 Mm-hmm.

Richard Sutton 5:55 Okay? So, there's ground truth. There's no ground truth in, in, uh, large language models, 'cause you don't have a, a prediction about what will happen next.If you say something in your, in your, um, conversation, there's n- the large language models have no prediction about what the person will say in response to that, or, or what the, what the response will be.

Dwarkesh Patel 6:16 I mean, I think they do. Like, they... You can literally ask them, "What, what, what would you anticipate a user might say in response?" And they have a prediction.

Richard Sutton 6:23 Oh, no. They, they res- they will respond to that question right.

Dwarkesh Patel 6:26 Yeah.

Richard Sutton 6:26 But they have no prediction in the substantive sense that they won't be surprised by what happens. And if something happens that isn't what you might say they predicted, they will not change, because an unexpected thing has happened and they're... To learn that, they'd have to make an adjustment.

Dwarkesh Patel 6:44 I, I... So I think a c- capability like this does exist in context. So it's interesting to watch a model do chain of thought, and then suppose it's trying to solve a math problem. It'll say, "Okay, I'm gonna approach this problem using this approach at first," and it'll write this out and be like, "Oh, wait, I just realized this is the wrong conceptual way to approach the problem. I'm gonna restart by this another approach." And that flexibility does exist in context, right? Do you have something else in mind or do you just think that you need to extend this capability across longer horizons?

Richard Sutton 7:15 I'm just saying they don't have a, have a, a, a... In any meaningful sense, they don't have a prediction of what will happen next. They will not be surprised by what happened next, they will not make any changes if, if something happens-

Dwarkesh Patel 7:28 But isn't that, isn't that, isn't that-

Richard Sutton 7:29 ... based on what happens.

Dwarkesh Patel 7:30 Isn't that literally what next-token prediction is? Prediction about what's next-

Richard Sutton 7:33 No, it-

Dwarkesh Patel 7:33 ... and then updating on a surprise?

Richard Sutton 7:35 Next token is what they should say, what the action should be. It's not what the world will give them in response to what they do. Let's, let's go back to the, uh, their lack of goal.

Dwarkesh Patel 7:46 Mm-hmm.

Richard Sutton 7:47 For me, having a goal is the essence of intelligence.

Dwarkesh Patel 7:50 Right.

Richard Sutton 7:50 Something is intelligent if it can achieve goals. It's... I like John McCarthy's definition that intelligence is the computational power of the ability to achieve goals.

Dwarkesh Patel 7:59 Yeah.

Richard Sutton 7:59 So you have to have goals or you're, you're, you're not... You're just, you're just a, you're just a behaving system. You're, you're not, you're not anything special and you're not intelligent.

Dwarkesh Patel 8:09 Right.

Richard Sutton 8:10 And you agree that large language models don't have goals?

Dwarkesh Patel 8:13 I think they... No, they have a goal.

Richard Sutton 8:14 What's the goal?

Dwarkesh Patel 8:15 Next-token prediction.

Richard Sutton 8:17 That's not a goal. It doesn't... It, it doesn't change the world. Look, you know-

Dwarkesh Patel 8:22 I think the, the dis-

Richard Sutton 8:22 Tokens come at you, and if you predict them, you don't influence them.

Dwarkesh Patel 8:27 Oh, yeah. I, I... It's not a goal about the external world.

Richard Sutton 8:31 Yeah. It's not a goal. It's not a substantive goal. It's not... You can't look at a system and say, "Oh, it, uh, has a goal," if it's just sitting there predicting and being happy with itself that it's predicting accurately.

Dwarkesh Patel 8:42 Yeah. I, I guess maybe the bigger question I want your, uh, I wanna understand is why you don't think doing RL on top of LLMs is a productive direction? 'Cause be-... We, we seem to be able to give these models a goal of solving difficult math problems, and they're, in many ways, um, at the very peaks of human level in, in their capacity to solve math olympiad type problems, right? They got gold at IMO. So it seems like the model which got gold at the International Math Olympia does have the goal of getting math problems right. Um, so why can't we extend this to different domains?

Richard Sutton 9:15 Well, the math problems are different. Um, the... Under... Making a model of the physical world and, uh, carrying out the consequences of mathematical, um, assumptions or operations-

Dwarkesh Patel 9:27 Right.

Richard Sutton 9:27 ... those are very different things. Like, the, the empirical world has to be learned. You have to learn the consequences. Um, whereas the, uh, the math is, is more just computational. It's more like standard planning. So, so there, y- you can, you can, um... They can have a goal to, to, um, uh, to find the proof, and they are, in, in some way, given that goal to find the proof.

Dwarkesh Patel 9:57 Right. So, I mean, it's interesting because you wrote this essay in 2019 titled The Bitter Lesson, and this is the most influential essay perhaps in the history of AI. But people have used that as a justification for scaling up LLMs, because in their view, this is the one scalable way we have found to pour ungodly amounts of compute into learning about the world. And so it's interesting that your perspective is that the LLMs are actually not bitter lesson-pilled.

Richard Sutton 10:29 It's an interesting question whether, uh, large language models are, are, uh, a case of the bitter lesson.

Dwarkesh Patel 10:37 Yeah.

Richard Sutton 10:37 'Cause they are clearly, um, a, a way of using massive computation, th- things that will scale with computation up to, uh, up to the limits of the internet.

Dwarkesh Patel 10:48 Yeah.

Richard Sutton 10:50 Uh, but they're also a, a way of putting in, uh, lots of, um, human knowledge. And, uh, so, so this is an interesting question. Um, it's a, uh, sociological or industry question. Uh, will they reach the limits of, of, of the data and, uh, and be superseded by things that, that are, that are... can get more data just from experience rather than from, uh, from people? Uh, i- in some ways, it's a classic case of the, of the, of the bitter lesson. With the more, the more human knowledge we put into the large language models, the better they can do, and so it feels good. Um, and yet, uh, one... Well, I, in, in particular expect there to be systems that can learn from experience, which could well perform much, much better and be much more scalable. In which, uh, case, it will be another instance of the bitter lesson, that the things that, that used human knowledge were eventually superseded by things that just, um, trained from, uh, experience and computation.

Dwarkesh Patel 12:05 I, I guess that doesn't seem like the crux to me, because I think those people would also agree that the overwhelming amount of compute in the future-... will come from, uh, learning from experience. They just think that the scaffold or the basis of that, the, the thing you'll start with in order to pour in the compute to do this future experiential learning or on-the-job learning will be LLMs. And so, I, I guess I, I, I'm, I still don't understand why this is the wrong starting point altogether, why we need a whole new architecture to begin doing experiential continual learning, uh, and why we can't start with LLMs to do that.

Richard Sutton 12:45 Well, in every case, it's a bitter lesson. You know, you could start with, uh, human knowledge-

Dwarkesh Patel 12:51 Right.

Richard Sutton 12:51 ... and then just, and then do the scalable things.

Dwarkesh Patel 12:54 Yeah.

Richard Sutton 12:54 That's always the case. And there's no, uh, never any reason why that has to be bad.

Dwarkesh Patel 12:59 Right.

Richard Sutton 12:59 But, in fact, and in practice, it has always turned out to be bad because people get locked into the human knowledge approach and they psychologically or, you know, now I'm, now I'm speculating why it is, but this is what has always happened.

Dwarkesh Patel 13:15 Yeah.

Richard Sutton 13:15 That, uh, yeah, y- they get, they get, their lunch gets eaten (laughs) by the methods that are truly scalable.

Dwarkesh Patel 13:22 Gi- yeah, give me a sense of what the scalable method is.

Richard Sutton 13:24 The scalable method is you learn from experience. Um, you, uh, you, you try things and you see what, you see what works. No one, no one has to tell you, "Eh, eh..." First of all, you have a goal. So, (laughs) without a goal, uh, there's no sense of right or wrong or better or worse. So, large language models are trying to get by without having, uh, a goal or a sense of better or worse. That's just, you know, it's exactly starting in the wrong place.

Dwarkesh Patel 13:51 May- maybe it's, uh, um, interesting to compare this to humans. So in both the case of learning from imitation versus experience and on the question of goals, I think there's some interesting analogies. So, you know, kids will initially learn from imitation. Uh, you don't think so?

Richard Sutton 14:11 No, of course not.

Dwarkesh Patel 14:13 Uh, really?

Richard Sutton 14:14 Yeah.

Dwarkesh Patel 14:15 I think kids just, like, watch people. They, like, kind of try, try to, like, say the same words-

Richard Sutton 14:19 How old are those, these kids?

Dwarkesh Patel 14:22 I, I think the level of-

Richard Sutton 14:23 What about the first six months?

Dwarkesh Patel 14:25 I think they're cr- kind of imitating things. They're trying to, like, make their mouths sound the way they see their mother's mouth sound and then they'll say the same words without understanding what they mean. And as they get older, the complexity of the imitation they do increases, so that's, you're, you're, you're, you know, you're imitating maybe the skills that your, uh, people in your band are, uh, using to hunt down the deer or something. And then you go into the learning-from-experience RRL regime. But I think there's a lot of imitation learning happening with, uh, humans.

Richard Sutton 14:51 Yeah, surprising. Uh, yeah, you can have such a different point of view.

Dwarkesh Patel 14:54 Yeah.

Richard Sutton 14:55 Um, when I see kids, I see kids, uh, just trying things, and, like, waving their hands around and moving their eyes around and no one... No, no one, no one tells them... There i- there is no i- there is no, um, imitation for, uh, how they move their eyes around or even the sounds they make. They may, they may want to e- create the same sounds, but the, um, the actions, you know, the thing that the, uh, infant actually does, there, there's no targets for that. There are no examples for that.

Dwarkesh Patel 15:25 I agree that doesn't explain everything infants do, but I think it guides the learning process. I mean, even an LLM when it's trying to predict the next token, early in training it will, like, make a guess. It'll be different from what, like, it actually sees. And in some sense, it's like very short horizon RL where it's like making this guess of, like, "Well, I think this to- token will be this." It's actually this other thing, similar to how a kid will try to say a word, it comes out wrong.

Richard Sutton 15:46 The, the large language models is learning from training data. It's not learning from experience. It's r- it's learning from something that will never be a- available during its normal life. There's never any, uh, training data that says you should do this action in normal life.

Dwarkesh Patel 16:02 I, I think this is maybe m- more of a semantic distinction. Like, what do you call school? Is that not training data? You're not, like, going to school because it's like...

Richard Sutton 16:10 School is much later. Okay, I shouldn't have said never. But, but (laughs) I don't know. I think I would even say it about school. But formal schooling is, is the exception. You shouldn't, shouldn't base your theories on that.

Dwarkesh Patel 16:23 But, but to me, you have phases of learning where y- I think you're just sort of programming your biology that, like, early on, you're not that useful. And then, like, kind of why you exist is to understand the world and, like, learn how to interact with it. Um, and it seems kind of like a training phase. I agree that then there's, like, a sort of more gradual... There's not a sharp cutoff to, like, training to deployment. But there seems to be this, like, initial tr- training phase, right?

Richard Sutton 16:47 There's nothing whe- where you have training of what you should do. There's nothing. You g- you, you, you see things that happen. You're not, you're not told what to do. Uh, don't, don't, don't be difficult.

Dwarkesh Patel 17:00 (laughs)

Richard Sutton 17:00 I mean this is obvious.

Dwarkesh Patel 17:02 I mean, you're, like, literally taught what to do. This is, like, w- where the word training comes from is from humans, right?

Richard Sutton 17:07 So I don't think, uh, learning is really about training.

Dwarkesh Patel 17:11 Hmm.

Richard Sutton 17:11 I think learning is about, about learning. It's about an active process. The child d- tries things and sees what happens.

Dwarkesh Patel 17:18 Right.

Richard Sutton 17:19 Yeah. It does not... Eh, i- uh, we don't think, uh, we don't think about training when we think of the, uh, an infant growing up. These, these things are actually rather well understood. If you go to look about how psychologists think about learning, there's nothing like, uh, imitation. Maybe there are some ex- extreme cases where humans might do that or appear to do that, but there's no basic animal learning process called imitation. There are basic lear- animal learning processes for prediction and for trial and error control. I mean, it's really interesting how sometimes the most hardest things to see are the obvious ones. It's obvious, um, if you just look at animals and how they learn and you look at psychology and how our theories of them, um, it's obvious that, that supervised learning is not part of, uh, the way animals learn. We don't have, we don't have examples of desired behavior. What we have is examples of things that happen, things... One thing has, uh, followed another. And we have examples of, "We did something and, and"And, and there were consequences. But there are no examples of supervised learning. I mean, there are no... Supervised learning is not something that happens in nature. And, you know, school, e- even if that was the case, you know, we should forget about it because it's, it's just a... That's some special thing that happens in people. It doesn't happen broadly in nature. And, you know, squirrels don't go to school. Squirrels can learn all about the world. It's absolutely obvious, I would say, that, um, supervised learning doesn't happen in animals.

Dwarkesh Patel 18:59 So I, I, I interviewed this psychologist and anthropologist, Joseph Henrich, who has done work about cultural evolution and basically how did... What, you know, what distinguishes humans and how do humans pick up knowledge?

Richard Sutton 19:14 Why are you trying to distinguish humans? Humans are animals. What we have in common is more interesting. What we have, what distinguishes us, we shou- we should be paying less attention to.

Dwarkesh Patel 19:25 I mean, we're trying to replicate intelligence, right?

Richard Sutton 19:27 Yes.

Dwarkesh Patel 19:27 So if you wanna understand what is it that enables humans to go to the moon or to build semiconductors, I think the thing we wanna understand is the thing that makes... No animal can go to the moon or m- make semiconductors, so we want to understand what makes humans special.

Richard Sutton 19:42 So I like the way you consider that obvious, because I consider the opposite obvious. Yeah, I think we, we need to... We, we have to, we have to understand how we are animals and... If we understood a squirrel, I think we'd have a... We'd be almost all the way there to understanding human intelligence. The, the language part is just a small veneer on the surface. Okay, so this is great. You know, we're finding out the very different ways that we're thinking.

Dwarkesh Patel 20:10 Yeah. Maybe-

Richard Sutton 20:11 We're not arguing. We're trying to-

Dwarkesh Patel 20:13 Or even if we were.

Richard Sutton 20:13 ... share, share our different ways of thinking with each other.

Dwarkesh Patel 20:17 Yeah. And you... I think argument is useful so, um, uh-

Richard Sutton 20:20 Yeah.

Dwarkesh Patel 20:20 But I do want to com- complete this thought. So Joseph Henrich has this interesting theory that, uh, if you look, a lot of the, uh, skills that humans have had to master in order to be successful, and we're not talking about, you know, last 1,000 years or last 10,000 years, but hundreds of thousands of years. Uh, you know, the world is really complicated and it's not possible to reason through how to, let's say, hunt a, uh, seal if you're living in the Arctic. And so there's this many, many step-long process of how to make the bait and how to find the seal and then how to process the food in a way that makes sure you won't get poisoned. And it's not possible to reason through all of that. And so over time, yes, there's this like larger process of whatever analogy you want to use, maybe RL is something else, where culture as a whole has figured out how to, uh, find and kill and eat, uh, seals. But then what is happening when through generations this knowledge is transmitted is, in his view, that like there, you just have to imitate your elders in order to learn that skill because you can't, you can't think your way through how to hunt and kill and process a seal. You have to just watch other people maybe make tweaks and adjustments. Uh, and that's how cultural knowledge accumulates. But the initial step of the cultural gain has to be imitation. But maybe you think about it a different way?

Richard Sutton 21:48 No, I think about it the same way.

Dwarkesh Patel 21:49 Okay.

Richard Sutton 21:50 But still, it's a small thing on top of basic trial and error learning.

Dwarkesh Patel 21:56 Mm.

Richard Sutton 21:56 Prediction learning. And it's what distinguishes us perhaps from, from many animals. But we're an animal first.

Dwarkesh Patel 22:07 Yeah.

Richard Sutton 22:07 And we were an animal before we had language and all those other things.

Dwarkesh Patel 22:13 I do think you make a very interesting point that continual learning is a capability that most mammals have. I guess all mammals have. So it's quite interesting that we have something that all mammals have but our AI systems don't have, right? Whereas maybe like the ability to understand math and follow difficult math problems depends on how you define math. But like this is a capability our AIs have but that no, almost no animal has. And so it's quite interesting what ends up being difficult and what ends up being easy.

Richard Sutton 22:45 Morphic paradox.

Dwarkesh Patel 22:47 That's right. For the era of experience to commence, we're gonna need to train AIs in complex real-world environments. But building effective RL environments is hard. You can't just hire a software engineer and have them write a bunch of cookie cutter validation tests. Real-world domains are messy. You need deep subject matter experts to get the data, the workflows, and all the subtle rules right. When one of Labelbox's customers wanted to train an agent to shop online, Labelbox assembled a team with a ton of experience engineering internet storefronts. For example, the team built a product catalog that could be updated during the episode because most shopping sites have constantly changing state. They also added a Redis cache to simulate stale data since that's how real e-commerce sites actually work. These are the kinds of things that you might not have naively thought to do but that Labelbox can anticipate. These details really matter. Small tweaks are often the difference between cool demos and agents that can actually operate in the real world. So whether it's correcting traces that you already produced or building an entirely new suite of environments, Labelbox can help you turn your RL projects into working systems. Reach out at labelbox.com/dwarkesh. All right, back

Dwarkesh Patel 23:57 to Richard. This alternative paradigm that you're imagining...

Richard Sutton 24:00 The experiential paradigm.

Dwarkesh Patel 24:01 Yes.

Richard Sutton 24:02 Let's lay out a little bit about what it is.

Dwarkesh Patel 24:04 Yeah, let's talk about it.

Richard Sutton 24:04 It says that experience, action, sensation... Well, sensation, action, reward, and this happens on and on and on, makes for life. It's, it says that this is the, uh, foundation and the focus of intelligence. Intelligence is about taking that stream and altering the actions to increase-... the rewa- the rewards in the stream.

Dwarkesh Patel 24:28 Right.

Richard Sutton 24:29 So learning then is from the stream, and learning is about the stream. So it's, that, that second part is, is, is particularly telling. You know, the, what you learn, your knowledge, your knowledge is about the stream. Your knowledge is about if you do some action, what will happen? Or it's about, uh, which events will follow other events. It's about the stream. It's the content of the knowledge is- is statements about the stream. Um, and so because it, it's- it's a statement about the stream, you can test it by comparing it to the stream and you can learn continually.

Dwarkesh Patel 25:05 Mm-hmm. So when you're imagining this future continual learning agent...

Richard Sutton 25:10 They're not future. Of course we- they exist all- all the time. It's, I mean, this is what reinforcement learning paradigm is, learning from experience.

Dwarkesh Patel 25:17 Yeah. I guess the, maybe what I meant to say is, uh, human-level general continual learning agent.

Richard Sutton 25:22 Uh-huh.

Dwarkesh Patel 25:23 What is the reward function? Is it just predicting the world? Is it, uh, is it then having s- a specific effect on it? H- h- what would the general reward function be?

Richard Sutton 25:34 The reward, uh, function is arbitrary. And, um, so if you're playing chess, it's to win the game of chess. I- if you were to, um, uh, if you're a squirrel, maybe the- the reward has to do with getting nuts.

Dwarkesh Patel 25:48 Right.

Richard Sutton 25:50 Um, in general, for an animal, you would say the reward is to avoid pain and to, uh, uh, acquire pleasure.

Dwarkesh Patel 25:58 Right.

Richard Sutton 25:59 Uh, and this also would be a component having to do with, uh, I think there would be, should be a component having to do with your, uh, increasing understanding of your, of your environment that would be sort of an intrinsic motivation.

Dwarkesh Patel 26:14 I see. I guess this AI would be deployed to, like, lots of people would want it to be doing lots of different kinds of things.

Richard Sutton 26:23 Right.

Dwarkesh Patel 26:24 So it's performing the task people want, but at the same time, it's learning about the world from doing that task. And do you, do you imagine... Okay, so y- we get rid of this paradigm where there's training periods and then there's deployment periods. But then is there, do we also get rid of this paradigm when there's the model and then instances of the model or copies of the model that are, you know, doing certain things? H- h- how do you think about the fact that they're, we'd want this thing to be doing different things, we'd want to aggregate the knowledge that it's g- gaining from doing those different things?

Richard Sutton 26:59 I don't like the word model when used the way you just did.

Dwarkesh Patel 27:01 Interesting.

Richard Sutton 27:02 I- I think a better word would be, uh, the network, 'cause I think you mean the-

Dwarkesh Patel 27:07 Mm-hmm.

Richard Sutton 27:07 ... the n- the network. Maybe there's many networks. So anyway, things would be learned, and then you'd have copies and many instances. And sure, you'd want to share knowledge across the, uh, instances. And there would be lots of possibilities for doing that. Like there is not today, you can't have one child lear- grow up and- and learn about the world and then e- and then every new child has to repeat that process. Whereas with AIs, with a digital intelligence-

Dwarkesh Patel 27:36 Yeah.

Richard Sutton 27:36 ... you could hope to do it once and then copy it into the next one as a starting place.

Dwarkesh Patel 27:40 Right.

Richard Sutton 27:40 So this would be a huge savings. And I think actually it would be much more important than, uh, trying to learn from people.

Dwarkesh Patel 27:48 Mm-hmm. I agree that the kind of thing you're talking about is necessary regardless of whether you start from LLMs or not, right? If- if you want h- human- or animal-level intelligence, you're going to need this capability. Suppose a human is trying to make a startup, right? And this is a thing which has a reward on the order of 10 years. Once in 10 years, you might have an exit where you get, you know, paid out a billion dollars. But humans have this ability to make intermediate auxiliary rewards or have some way of, even when they have extremely sparse rewards, they can still make intermediate steps having an understanding of, like, what the m- next thing they're doing leads to this grander goal we have. And so how do you imagine such a process might play out with AIs?

Richard Sutton 28:31 So this is something we know very well.

Dwarkesh Patel 28:33 Hmm.

Richard Sutton 28:34 And it's, the basis of it is temporal difference learning-

Dwarkesh Patel 28:36 Mm-hmm.

Richard Sutton 28:36 ... where the same thing happens, um, in a less grandiose scale. Like, when you learn to play chess, you have the grand- the long-term goal is winning the game, and yet you ha- ha- you can't, you, um, you want to be able to learn from shorter term things like, you know, taking your opponent's pieces. Um, and so you do that by having a value function which predicts the long-term outcome.

Dwarkesh Patel 28:59 Right.

Richard Sutton 28:59 And then if you take the guy's pieces where your prediction about the long-term outcome is changed, it goes up, you think you're gonna win, and then that increase in your- in your belief immediately, quote, "reinforces" the, uh, the move that led to taking the piece.

Dwarkesh Patel 29:15 Mm-hmm.

Richard Sutton 29:16 Okay, so we have this long-term 10-year goal of making a startup and making a lot of money. And so when we make progress, we say, "Oh, I'm- I'm- I'm more likely to, uh, achieve the long-term goal," and that rewards the- the steps along the way.

Dwarkesh Patel 29:32 Right. And then you also want some mobility for information that you're learning. I mean, one, one of the things that makes humans quite different from these LLMs is that if you're onboarding on a job, you're- you're picking up so much context and information and that's what makes you useful at the job, right? You're, uh, everything from how your client, uh, preferences to how the company works, to everything. Um, and is the bandwidth of information that you get from a procedure like TD learning high enough to have this, like, huge pipe of, like, context and tacit knowledge that you'd need to be picking up in the way humans do when they're, when they're just, like, deployed?

Richard Sutton 30:10 Um, I think the crux of this, and I'm not sure, but, eh, the- the big world hypothesis seems very relevant. And the reason why humans become useful on their job is because they are encountering the particular part of the world.

Dwarkesh Patel 30:27 That's right.

Richard Sutton 30:27 And, um...And it can't have been anticipated and it can't all have been put in in, in advance, in, in, uh... The world is so huge that you can't... The, the dream, as I see it, the dream of large language models is you could teach the an- the agent everything, and it will know everything and it won't have to learn anything, uh, online.

Dwarkesh Patel 30:49 Right.

Richard Sutton 30:50 During its life. Okay? And, and your examples are all, "Well, really you have to because you can... there's a lot to..." You can teach it but there's all the little idiosyncrasies of the particular life they're leading and the, the particular people they're working with and what they like as opposed to what re- average people like.

Dwarkesh Patel 31:07 Right.

Richard Sutton 31:07 And so that's just saying the world is really big and so you're gonna have to learn it-

Dwarkesh Patel 31:11 Yeah.

Richard Sutton 31:12 ...uh, along the way.

Dwarkesh Patel 31:13 Yeah. So it seems to me you need two things. One is some way of converting this long-run goal reward into smaller auxiliary or, you know, um, uh, these, like, predictive rewards of the future reward or the future reward to at least the final reward. Then you need some other way... Initially, it seems to be you need some way of then, "Okay, I'm... I need to hold onto all this context that I'm gaining as I'm w- w- l- l- working in the world," right? "I'm, like, learning about my clients, my, my company, all this information and I'm..."

Richard Sutton 31:50 So I'm- I would say you're just doing regular learning.

Dwarkesh Patel 31:52 Yeah.

Richard Sutton 31:53 Maybe using context because in large language models all that information has to go into the context window.

Dwarkesh Patel 31:58 Right.

Richard Sutton 31:58 But in, in a continual learning setup it just goes into the weights.

Dwarkesh Patel 32:02 May- maybe... Yeah, so maybe "context" is the wrong word to use 'cause I mean a more general thing.

Richard Sutton 32:06 You learn a policy that's specific to the environment-

Dwarkesh Patel 32:09 Right.

Richard Sutton 32:09 ...that you're finding yourself in.

Dwarkesh Patel 32:10 Yeah. So the question I'm trying to ask is you need some way of getting... Like, how many bits per second are you picking u- like, is a human picking up when they're d- you know, out in the world, right? Um, if you're just, like, interacting over Slack with your work clients and everything?

Richard Sutton 32:26 So maybe you're trying to ask the question of it seems like the, the reward is too small of a thing to, to do all the learning that we-

Dwarkesh Patel 32:32 Yeah.

Richard Sutton 32:33 ...need to do. But of course we, we have the, uh, the sensations. Uh, we, we have all the other information we can learn from.

Dwarkesh Patel 32:40 Right.

Richard Sutton 32:41 We don't just learn from the reward. We learn from all the data.

Dwarkesh Patel 32:44 Yeah. So what is the learning process which helps you capture that information?

Richard Sutton 32:51 So now I wanna talk about the base common model of the agent with the four parts.

Dwarkesh Patel 32:57 Yeah.

Richard Sutton 32:58 So we need a policy. The policy says, "In the situation I'm in, what should I do?" We need a value function. The value function is the thing that is learned with TD learning. And the value function produces a number. The number says how well is it going.

Dwarkesh Patel 33:13 Mm-hmm.

Richard Sutton 33:13 And then you watch if that's going up and down and use that to adjust your policy. Okay? So the- those two things and, and then, uh, there's also the perception component, which is the construction of your, uh, state representation, this- your sense of where you are now. And the fourth one is what we're really getting at, most transparently anyway. W- the, the fourth one is the transition model of the world, um, that's why I am uncomfortable with just calling everything "models" because I wanna talk about the model of the world, the transition model of the world, your belief that if you do this, what will happen? What will be the consequences of what you do, so your physics of the world. But it's all- it's not just physics. It's also, um, abstract models like, you know, your model of how you traveled, um, from California up to Edmonton for this podcast.

Dwarkesh Patel 34:00 Right.

Richard Sutton 34:00 That was a model and that's a transition model and that would be, uh, learned. And it's not learned from rewards. It's learned from you did things, you saw what happened.

Dwarkesh Patel 34:08 Yeah.

Richard Sutton 34:08 You made that model of the world. That is- it will be learned very richly from all the sensation that you receive, not just from the reward. It has to include the reward as well but it- there's- it's- that's a s- a small part of the whole model.

Dwarkesh Patel 34:23 Right.

Richard Sutton 34:23 Small crucial part of the whole model.

Dwarkesh Patel 34:25 Yeah. One of my friends, Toby Ord, pointed out that if you look at th- the muzero models that Google DeepMind deployed to learn Atari games, that these models were initially b- not a general intelligent self but a general framework for training specialized intelligences to play specific games. That is to say that you couldn't, uh, using that framework, train a m- policy to play both chess and Go and some other game. You had to train each one in a specialized way. And he was wondering whether that implies that reinforcement learning generally, because of this information constraint, y- y- y- you can only learn one thing at a time, uh, th- the density of information is not that high, or whether it was just specific to the way that muzero was done. And if it's specific to, uh, AlphaZero, what, what, w- what needed to be changed about that approach so that it could be a general learning agent?

Richard Sutton 35:21 The, the idea is totally general. You know, uh, I do use all the time as my canonical example the idea of an AI agent is like a person.

Dwarkesh Patel 35:30 Yeah.

Richard Sutton 35:30 And, and people, uh, in some sense, they have just one world they live in and, um, that world i- may involve chess and it may involve Atari games, uh, but those are j- are, are not a different task or a different world. Those are different states-

Dwarkesh Patel 35:45 Right.

Richard Sutton 35:46 ...that they encounter. And so the, the general idea is not limited a- at all.

Dwarkesh Patel 35:51 So maybe it would be useful to explain what was missing in that architecture or that, that approach which this continual learn- m- learning AGI would have.

Richard Sutton 36:04 Well, th- they just set it up. They didn't... Th- it was not their ambition to, to have one agent across, across, uh, those games. If we wanna talk about transfer, we should talk about transfer not a- across games or across tasks but transfer between states.

Dwarkesh Patel 36:21 Yeah. I, I, I guess I'm curious about historically have we seen the level of transfer using RL techniques that would be needed to build this kind of-

Richard Sutton 36:34 Oh. Okay, good. Good. We're not seeing transfer anywhere-We're not seeing general... Critical to good performance is that you can generalize well from one state to another state.

Dwarkesh Patel 36:45 Yeah.

Richard Sutton 36:45 We don't have any methods that are good at that. What we have are people, um, try different things, eh, and they, they settle on something that, that, uh, a representation-

Dwarkesh Patel 36:56 Yeah.

Richard Sutton 36:56 ... that transfers well or that generalizes well. But we have no, we don't have any automated techniques to promote. We have very few automated techniques to promote transferring, and they're not, none of them are used in, in modern deep learning.

Dwarkesh Patel 37:10 Um, let me paraphrase just to make sure that I understood that correctly. It sounds like you're saying that when we do have generalization in these models, that is a result of some, uh, sculpted, uh-

Richard Sutton 37:27 Humans did it.

Dwarkesh Patel 37:28 Yeah.

Richard Sutton 37:29 The researchers did it, 'cause there's no other explanation. I mean, gradient descent will not make you generalize well. It will make you solve the problem.

Dwarkesh Patel 37:37 Right.

Richard Sutton 37:37 It will not make you, you know, get new data, you generalize in a good way. Generalization means train on one thing that'll affect what you do on the other things. So, we know deep learning is, is really bad at this. For example, we know that if you train on some new thing, it will often catastrophically interfere with all the old things that you s- that you knew.

Dwarkesh Patel 37:58 Mm-hmm.

Richard Sutton 37:59 So this is exactly bad generalization.

Dwarkesh Patel 38:01 Right.

Richard Sutton 38:02 Now, generalization, as I said, is some kind of influence of training on one state on other states. And generalization is n- n- not necessarily good or bad, right? Just the fact that you generalize is not necessarily good or bad. You can generalize poorly, you can generalize well.

Dwarkesh Patel 38:16 Right.

Richard Sutton 38:16 So you, you need... Generalization always will happen, um, but we need algorithms that will, uh, cause the, the generalization to be good rather-

Dwarkesh Patel 38:25 Yeah.

Richard Sutton 38:25 ... than bad.

Dwarkesh Patel 38:26 I'm not trying to kickstart this, uh, initial, uh, crux again, but I'm just genuinely curious because I, I think I might be using the term differently. I mean, one way to think about it is these LLMs are increasing the scope of generalization from, like, earlier systems which could not really even do a basic math problem to now they can do anything in this class of math olympiad type problems, right? So, you initially start with, like, they can generalize among addition problems at least, um, uh. Then you generalize to, like, they can generalize among, like, problems which require use of different kinds of mathematical techniques and theorems and, you know, conceptual categories, which is, like, what the math olympiad requires. And so it sounds like you don't think of th- being able to solve any problem within that category as an example of generalization? Or l- let me know if I'm m- uh, misunderstanding that.

Richard Sutton 39:18 Well, large language models, so complex. We don't, we don't really know what information they had prior. We are, we have to guess because they've been fed so much. This is one reason why they're not a good way to do science. Uh, it's just s- so uncontrolled, so unknown.

Dwarkesh Patel 39:37 But if you come up with an entirely new-

Richard Sutton 39:39 They're getting a bunch of things right-

Dwarkesh Patel 39:42 Right.

Richard Sutton 39:42 ... perhaps. And, uh, so the question is why? Well, it may be that they don't need to generalize to get them right because the only way to get some of them right is, is to form something which gets all of them right.

Dwarkesh Patel 39:53 Mm-hmm.

Richard Sutton 39:54 So, you know, if there's only one answer, uh, then, and you find it, I, we... That's not called generalization. It's just, it's the only way to solve it, and so they find the only way to solve it.

Dwarkesh Patel 40:03 Mm-hmm.

Richard Sutton 40:04 Now, generalization is when it could be this way, it could've been that way, and they do it the good way.

Dwarkesh Patel 40:08 My, my understanding is that they, um, this is working more and more, better and better with, uh, coding agents. So, engineers, obviously, if you're trying to program a library, there's many different ways you could achieve the end spec. And an initial frustration with these models has been that they'll do it in a way that's sloppy. And then over time, they're getting better and better at coming up with the design architecture and the abstractions that developers find more satisfying. And it seems th- an example of what you're talking about.

Richard Sutton 40:41 Well, there's nothing in them which will cause it to generalize well. The gradient descent will cause them to find a solution to the problems they've seen. And if there's only one way to solve them, then you know, they'll, they'll do that. But if there are many ways to solve it, some which generalize well, some which generalize poorly, there's nothing in them, in the algorithms, that will cause them to generalize well.

Dwarkesh Patel 41:03 Okay.

Richard Sutton 41:03 But people, of course, are involved and-

Dwarkesh Patel 41:05 Right.

Richard Sutton 41:05 ... and, you know, if, if it's not working out, you know, they fiddle with it, and until they find a way, perhaps until they find a way i- which it generalizes well.

Dwarkesh Patel 41:14 So to prep for this interview, I wanted to understand the full history of RL, starting with reinforce up to current techniques like GRPO. And I didn't just want a list of equations and algorithms. I wanted to really understand each change in this progression and the underlying motivation. You know, what was the main problem that each successive method was actually trying to solve? So I had Gemini Deep Research walk me through this entire timeline step by step. It explained the last 20 years of gradual innovation and explained how each step made the RL learning process more stable or more sample efficient or more scalable. I asked Deep Research to put all of this together, like an Andrej Karpathy style tutorial, and it did that. What was cool is that it combined this whole lesson together into one coherent, cohesive document in the style that I wanted. It was also great that it assembled all of the best links in the same place, so that if I wanted to understand any specific algorithm better, I could just access the right explainer right there. Go to gemini.google.com to try it out yourself. All right, back to Richard.

Dwarkesh Patel 42:17 I wanna zoom out and ask about... So, being in the field of AI for longer than almost anybody who is commentating on it, uh, or working on it now, I'm just curious about what the biggest surprises have been, how much new stuff you feel like is coming out, or does it feel like people are just playing with old ideas? Um, d- zooming out, you, you know, you're... You got into this even before, like, deep learning was popular. So-How, how do you see this trajectory of this field over time, and how new ideas have come about and everything? And what's been surprising?

Richard Sutton 42:51 Okay. So yeah. I, I, I, I, um, thought a little bit about this. There are many things, or a handful of things. Um, first, the large language models are surprising. It's surprising how, how effective, um, neural networks, artificial neural networks are at, at, uh, language tasks.

Dwarkesh Patel 43:12 Right.

Richard Sutton 43:12 You know, that, that was a surprise. It wasn't expected. Language seemed different. So that's impressive.

Dwarkesh Patel 43:17 Mm-hmm.

Richard Sutton 43:18 Um, there's a, a long-standing controversy in AI about, uh, simple basic principle methods. Uh, the, the general-purpose methods like search and learning, and compared to, um, human-enabled systems, uh, like symbolic methods. And, um, uh, so in the old days, it was interesting, because things like search and learning were called weak methods, because they're just... oh, they just use general principles. They're not using, uh, the power that comes from, uh, imbuing a system with human knowledge. So those were called strong. And, um, and so I think the weak methods have just, you know, totally won. That's, you know, that's, that's, that's the biggest, um, question from the old days of AI, what would happen, and, you know, yeah. Learning and search have just won the day.

Dwarkesh Patel 44:12 Right.

Richard Sutton 44:13 But there's a sense which that was not surprising to me, because I was always voting for the... or hoping or rooting for the, for the, uh, simple basic principles.

Dwarkesh Patel 44:21 Yeah.

Richard Sutton 44:21 And so even with the large language models, I, it's surprising how, how well it worked, but it was all, it was all good and gratifying. And, um, in things like AlphaGo, it's, it's sort of surprising how well that was able to work. Um, and AlphaZero in particular, how, how well it was able to work. Um, but it's all very gratifying, because again, it's simple basic principles are winning the day.

Dwarkesh Patel 44:44 D- d- h- h- have there felt like whenever the public conception has been changed because some new technique was... or s- sorry, some new application was developed. For example, when AlphaZero became this viral sensation, to you, as somebody who has literally came up with many of the techniques that were used, did it feel to you like new breakthroughs were made, or does it feel like, oh, we've had these techniques since the '90s, and people are simply combining them and applying them now?

Richard Sutton 45:13 So the whole AlphaGo thing had a precursor-

Dwarkesh Patel 45:16 Right.

Richard Sutton 45:16 ... which is TD-Gammon. Jerry Tesauro did exactly, um, reinforcement learning, temporal difference learning methods to, um, to play backgammon.

Dwarkesh Patel 45:28 Right.

Richard Sutton 45:28 And beat the, beat the world's best players. And it worked really well. And so in some sense, AlphaGo was, was merely a scaling up of that process. So there was quite a bit of scaling up, and there was also an additional innovation in how the search was done.

Dwarkesh Patel 45:45 Right.

Richard Sutton 45:45 But it made sense. It wasn't surprising in that sense. AlphaGo actually didn't use, uh, TD learning. It waited to see the final outcomes. Uh, but AlphaZero used TD. Uh, and AlphaZero was applied to all the other games, and that did extremely well. I was very... I've always been very impressed by the way AlphaZero plays chess, because I'm a chess player, and it, it just... it just sacrifices material for sort of positional advantages, and it's just c- it's just content and patient to, uh-

Dwarkesh Patel 46:18 (laughs)

Richard Sutton 46:18 ... sacrifice that material for a long period of time. And, um, so that was surprising that it worked so well, but also gratifying-

Dwarkesh Patel 46:27 Yeah.

Richard Sutton 46:27 ... and fitting into my worldview. So, so this has led me where I am. Where I am is, I'm in some sense a contrarian or somewhat d- thinking differently from the field is... And I'm, I am personally just kind of content being out of sync with my field for a long period of time, perhaps decades, uh, because occasionally I have improved, uh, right in the past. And the other thing that I do to help me not feel I'm, I am out of sync and thinking in a strange way is to look not at my, my local, uh, environment or my local field, but to look back in, in time, in- into history and see what people have thought classically about, about, um, about the mind-

Dwarkesh Patel 47:12 Yeah.

Richard Sutton 47:12 ... in many different fields. And I don't feel I'm out of sync with the larger traditions. It's, I, I really view myself as a classicist rather than as a contrarian. I go to what, what the larger community of, of thinkers about the mind have always thought.

Dwarkesh Patel 47:27 Hmm.

Dwarkesh Patel 47:28 Okay. Some sort of left-field questions for you, uh, if you'll tolerate them. Um, so the way I read The Bitter Lesson is that it's not saying necessarily that human artisanal researcher tuning doesn't work, but that it obviously scales much worse than compute, which is growing exponentially. And so you want techniques which leverage the latter.

Richard Sutton 47:51 Yep.

Dwarkesh Patel 47:52 And once we have AGI, we'll have m- researchers would scale linearly with compute, right? So we'll have this avalanche of millions of AI researchers, and their stock will be growing as fast as, uh, compute. And so maybe this will mean that it is rational or it will make sense to have them doing good old-fashioned AI and doing these artisanal solutions. Uh, does that... as a vision of what happens after AGI in terms of how AI research will evolve, I wonder if that's still compatible with The Bitter Lesson?

Richard Sutton 48:25 Well, how did we get to this AGI? You want to presume that it's been done.

Dwarkesh Patel 48:30 Let's suppose it started with general math- methods, but now we've got the AGI. And now we want to go-

Richard Sutton 48:35 Then we're done.

Dwarkesh Patel 48:37 Hmm?

Richard Sutton 48:37 We're done.

Dwarkesh Patel 48:38 Interesting. You don't think that there's a- anything above AGI?

Richard Sutton 48:43 Well, but you're using it to get AGI again.

Dwarkesh Patel 48:46 Well, I'm using it to get superhuman levels of intelligence or competence at different tasks.

Richard Sutton 48:50 Yeah.So these AGIs, if they're not superhuman already, then they, the, the knowledge that they might impart would be not superhuman.

Dwarkesh Patel 49:00 I guess there's different gradations if you're a human.

Richard Sutton 49:01 I'm not sure this, this, your, your idea makes sense, 'cause-

Dwarkesh Patel 49:05 So-

Richard Sutton 49:05 ... 'cause it seems to presume the existence of AGI, uh, and then, th- we've already worked that out.

Dwarkesh Patel 49:12 So may- maybe one way to motivate this is AlphaGo was superhuman.

Richard Sutton 49:15 Yeah.

Dwarkesh Patel 49:15 Um, it beat any Go player. AlphaZero-

Richard Sutton 49:18 Yeah.

Dwarkesh Patel 49:18 ... would beat AlphaGo every single time.

Richard Sutton 49:21 Yeah.

Dwarkesh Patel 49:21 So there's ways to get more superhuman than, than even superhuman.

Richard Sutton 49:25 Yeah.

Dwarkesh Patel 49:25 And it was a different architecture. And so it seems plausible to me that, well, the j- agent that's, like, able to generally learn across all domains, there would be ways to make that give it better architecture for learning, just the same way that AlphaZero was an improvement upon AlphaGo and MuZero was an improvement upon AlphaZero.

Richard Sutton 49:41 And the way AlphaGo was an improvement was it did not use the human knowledge, but just went from experience.

Dwarkesh Patel 49:49 Right.

Richard Sutton 49:49 So why do you, why do you say-

Dwarkesh Patel 49:52 But-

Richard Sutton 49:52 ... bring in other agents' expertise to teach it when it's, when it's been... it's worked so well from experience and not by help from another agent?

Dwarkesh Patel 50:04 I agree that in that particular case that it was moving to more general methods. But I meant to use that example to illustrate that it's possible to go superhuman, to superhuman plus plus, to superhuman plus, plus, plus, plus.

Richard Sutton 50:16 Yeah.

Dwarkesh Patel 50:17 And I'm curious if you think those gradations will continue to happen by just making the method simpler, or because we'll have the capability of these millions of minds who can then add complexity as needed, if that will continue to, if that will continue to be a false path even when you have billions of AI researchers or trillions of AI researchers.

Richard Sutton 50:35 I think, l- I think more interesting is just to think about that case-

Dwarkesh Patel 50:39 Mm-hmm.

Richard Sutton 50:39 ... which when you have many AIs, um, will they help each other the way cultural evolution works in people?

Dwarkesh Patel 50:46 Right.

Richard Sutton 50:48 L- let's just, maybe we should talk about that.

Dwarkesh Patel 50:50 Yeah, for sure.

Richard Sutton 50:50 The bitter lesson, oh, who cares about that?

Dwarkesh Patel 50:52 (laughs)

Richard Sutton 50:53 That's, that's an empirical observation about a particular period in history, 70 years in history, no longer... doesn't necessarily have to apply the next 70 years. So an interesting question is, you're an AI, you get some more computer power, should you use it to make yourself, you know, more computationally capable? Or should you use it to spawn off a copy of yourself-

Dwarkesh Patel 51:11 Right.

Richard Sutton 51:11 ... to go learn something interesting on the other side of the planet or on some other topic and then report back to you?

Dwarkesh Patel 51:16 Yep.

Richard Sutton 51:18 I think that's a really interesting question, um, that, that, uh, that will only arise in the age of digital intelligences.

Dwarkesh Patel 51:26 Mm-hmm.

Richard Sutton 51:27 I'm not sure what the answer is, but I think it, it will, uh, more questions. Will it be possible to really, you know, spawn it off, send it out, learn something new, some perhaps very new, and then will it be able to re- be reincorporated into the original?

Dwarkesh Patel 51:40 Mm-hmm.

Richard Sutton 51:41 Or will it, will it, uh, ha- will, will have changed so much that it, uh, it can't really be done?

Dwarkesh Patel 51:46 Yeah.

Richard Sutton 51:47 Is that possible or is it not? And, you know, you could carry this to its limit, as I, I, I saw one of your videos the other night that, that suggested-

Dwarkesh Patel 51:55 (laughs)

Richard Sutton 51:55 ... that it, that it could, where you spawn off many, many copies, do different things, it's highly decentralized, but report back to the, the, a central master. And th- this is, this will be such a powerful thing. Well, I think one thing that, uh... So this is my attempt to add something to this, this view, is that, uh, a big question, a big issue will become, uh, corruption. You know, if you, if you really could just get information from anywhere and bring it into your central mind, you can become more and more powerful. Uh, and i- it's, it's all digital and they all speak some internal digital language, maybe it'll be easy and possible, but it will not be, uh, that easy, as easy as you're imagining, because, uh, th- you could lose your mind this way. If you in- pull in something from the outside and build it into your, into your inner thinking, uh, it could take over you. It could change you. It could be, uh, your destruction rather than, uh, your inc- increment in knowledge.

Dwarkesh Patel 52:54 Mm-hmm.

Richard Sutton 52:55 And I think this will become a, a, a big concern, you know, particularly when you're, "Oh, he's figured all about, you know, how to play some new game," or figures out, he, he studied Indonesia and you want to incorporate that into your mind. Um, yeah, so you can't... So you could, you think, "Oh, just read it all in and that'll be fine," but no, you've just read a whole bunch of bits into your mind, and, uh, they could have viruses in them, they could have hidden goals, uh, they can, uh, warp you and change you. And this will be a- become a big thing. How do you have cybersecurity in the age of digital spawning and re- reforming again?

Dwarkesh Patel 53:34 Yeah. It's interesting that both quant firms and AI labs have a culture of secrecy, because both of them are operating in incredibly competitive markets and their success rest on protecting their IP. If you're an AI researcher or engineer and you're deciding where to work, most of the quant firms or AI labs that you'll be considering will be strongly siloing their teams to minimize the risk of leaks. Hudson River Trading takes the opposite approach. Their teams openly show their trading strategies, and their strategy code lives in a shared mono-repo. At HRT, if you're a researcher and you have a good idea, your contribution will be broadly deployed across all relevant strategies. This gives your work a ton of leverage. You'll also learn incredibly fast. You can learn about other people's research and ask questions, and you can see how everything fits together end-to-end, from the low level of execution of trades to the high level predictive models. HRT is hiring. If you want to learn more, go to hudsonrivertrading.com/dwarkesh. All right, back to Richard.

Dwarkesh Patel 54:35 I guess this brings us to the topic of AI succession.

Richard Sutton 54:38 Mm-hmm.

Dwarkesh Patel 54:39 Uh, you have a perspective that's quite different from a lot of people that I've interviewed and maybe, uh, a lot of people generally. So, eh, I also think it's a very interesting perspective. I, I want to hear about it.

Richard Sutton 54:48 Yeah. So I do think succession to digital...... or digital intelligence or augmented humans is inevitable. So the argument go, I have a four-part argument (laughs) . The argument, step one is, w- um, there's no government or organization that, that, uh, gives humanity a unified point of view, that dominates and that can, that can arrange ... There's no consensus about how the world should be run. And number two, um, we will figure out how intelligence works, researchers will figure it out eventually. And number three, we won't stop just with human-level intelligence, we will get, reach superintelligence. And number four is, that once ... It's inevitable over time that the most intelligent things around would gain resources and, and power. Uh, and, uh, so put all that together, it's, you know, you, um, it's sort of inevitable that you're going to have, um, succession to AI or to AI-enabled augmented humans. So within those, those four things seem clear and, and, and sure to happen. Uh, but within that set of possibilities, some ... There can be good outcomes as well as, as less good outcomes-

Dwarkesh Patel 56:10 Mm-hmm.

Richard Sutton 56:10 ... bad outcomes. And, um, so I'm just, just trying to be realistic about where we are, and, and ask how we should feel about it.

Dwarkesh Patel 56:20 Yeah. I, uh, I agree with all four of those arguments and the implication, and I also agree that succession contains a wide variety of possible futures. So curious to get more thoughts on that.

Richard Sutton 56:35 Right. And so then I do encourage people to think, um, positively about it. First of all because it's something we humans have always tr- tried to do for thousands of years, tried to understand themselves, tried to make themselves think better. And, um, you know, just understand themselves. So this is a great success from the, as science, humanities, uh, we're finding out what this essential part of, of, of humanness is, what it means to be intelligent. And then what I usually say is, is that this is all kind of human-centric. What if we look, you step aside from being a human and just, say, take the point of view of the universe? And, and this is, I think, a, a major stage in the universe, a major transition. A transition from replicators, we humans and animals, plants, we're all replicators, and that gives us some strengths and some limitations; and then we're entering the age of design where, 'cause our AIs are designed, our, our, our, our, all of our physical objects are designed, our buildings are designed, our, our technology is designed, and we're, we're designing now, uh, AIs. Things that can be intelligent themselves and that are themselves capable of design. And so this is, this is a, a key step in the world and I, and, and the universe, and I think, uh, it's the tr- it's the transition from the world in which most of the interesting things, uh, uh, that are, are replicated. Replicated means you can make copies of them, uh, but you don't really understand them. Like, right now we can make more intelligent beings, more children, uh, but we don't really understand how intelligence works.

Dwarkesh Patel 58:14 Right.

Richard Sutton 58:15 Whereas in, as we're, we're reaching now to having designed intelligence, intelligence that we do understand how it works and therefore we can change it in different ways and at different speeds, um, than otherwise. And, and our future, they might not be replicated at all. Like, we may just design AIs and those AIs will design other AIs and, um, everything will be done by design and construction rather-

Dwarkesh Patel 58:41 Mm-hmm.

Richard Sutton 58:41 ... than by replication. Yeah, I mark this as one of the four great stages of the universe. First there's, there's dust, ends with stars. Stars, we ... And, and then stars make planets and the planets give rise to life. And now we're giving li- life to, uh, designed entities. And so I think we should be proud and we should be, uh, uh, that we are giving rise to this great transition in the universe. Yeah, so it's an interesting thing. What should we ... Should we consider them h- uh, part of humanity or different from humanity? It's our choice. It's our choice whether we say, "Oh, they are our offspring and we should be proud of them and we should celebrate their achievements." Or we should, we could say, "Oh no, they're not us," and we should be horrified. It's, it's just, it's interesting that, that is, it feels to me like a choice, and yet it's such a strongly, uh, held thing that how could it be a choice? I like these sort of contradictory, uh, implications of thought.

Dwarkesh Patel 59:41 Hmm. Uh, I mean, it's interesting to consider if we were just designing another generation of hum- humans.

Richard Sutton 59:48 Yes.

Dwarkesh Patel 59:48 Maybe design's the wrong word, but we knew a future generation of humans are gonna come up. And forget about AI, we just know in the long run humanity will be more capable and maybe more numerous, maybe more intelligent. How do we feel about that? I do think there's potential worlds with future humans that we would be quite concerned about.

Richard Sutton 1:00:07 So are you thinking, like, maybe we are the, we are like the Neanderthals who give rise to Homo sapiens. Maybe Homo sapiens will give rise to a new group of people. That's what you're saying?

Dwarkesh Patel 1:00:17 So- something like that. Like, uh, I'm basically taking the example you're giving of like, okay, even if you consider them part of humanity-

Richard Sutton 1:00:22 Yeah.

Dwarkesh Patel 1:00:23 ... I don't think that re- uh, necessarily means that we should feel super comfortable-

Richard Sutton 1:00:27 Kinship.

Dwarkesh Patel 1:00:28 Yeah. Like, Nazis were humans, right? If we thought, like, "Oh, the future generation will be Nazis," I think we'd be, like, quite concerned about just handing off power to them. So, um, I agree that th- this is not super dissimilar to worrying about more capable future humans, but I don't think that that addresses a lot of the concerns people might have about this level of power being attained this fast with entities we don't fully understand.

Richard Sutton 1:00:54 Well, I, I think it's relevant to point out that, uh, for most of humanity-Um, they don't have much, uh, influence on what happens. Um, most of humanity doesn't influence-

Dwarkesh Patel 1:01:09 Mm-hmm.

Richard Sutton 1:01:10 ... who can control the atom bombs or who, uh, controls the nation-states.

Dwarkesh Patel 1:01:17 Right.

Richard Sutton 1:01:17 Even as a, as a citizen, I often feel that we don't control the nation-states very much. They're out of control. I- a lot of it has to do with just how you feel about change. Um, and if you think the current situation is really, really good, then you're, uh, more likely to be suspicious of change and averse to change than if you think, um, it's imperfect. And I think it's imperfect. In fact, I think it's pretty bad. (laughs)

Dwarkesh Patel 1:01:45 Yeah.

Richard Sutton 1:01:46 So I'm, I'm, I'm open to change, and I think humanity is not in a... has had good, super good track record. And maybe it's the best thing that there's been, but it- it- it's far from perfect.

Dwarkesh Patel 1:01:59 Yeah, I guess there's different varieties of change. Um, the Industrial Revolution was change. The Bolshevik Revolution was also change. And if you were around in Russia in the 1900s and you were like, "Look, things aren't going we- well. The tsar is kind of messing things up. We need change," I'd wanna know what kind of change you wanted before signing on the dotted line, right? And then similar with AI, where I'd want to understand, and to the extent it's possible to change its trajectory, to change its trajectory of AI such that the change is positive, um, for humans.

Richard Sutton 1:02:34 We- we are... We should be concerned about our future, the future, make s-... We should try to make it good. Um, we also- also, though, should recognize the limits, our limits. And we're... I think we want to avoid the feeling of entitlement, avoid the feeling, "Oh, well, we were here first. We should always have it in a good way." Um, how should we think about the future, and how much control, uh, a particular species on a particular planet should have over it? Uh, and how much control do we have? You know, a c- a counterbalance to our limited control over the long-term future of humanity (laughs) should be, how much control do we have over our own lives? Like, we have, uh, our own goals, and we have our- our families. And we... those things are much more controllable than, like, trying to control, um, the whole universe.

Dwarkesh Patel 1:03:32 Right.

Richard Sutton 1:03:32 Um, so I think it's appropriate, you know, for us to, to, uh, you know, really work towards our own local goals. And, uh, and it's, uh, kind of aggressive for us saying, "Oh, the future has to evolve this way that I want it to."

Dwarkesh Patel 1:03:49 Sure.

Richard Sutton 1:03:50 Because then we'll have arguments. Like, different people think the future, the global future should evolve in different ways, and then they have conflict.

Dwarkesh Patel 1:03:57 Yeah, and maybe-

Richard Sutton 1:03:58 And we want to avoid that.

Dwarkesh Patel 1:03:59 May- maybe a b- a good analogy here would be... Okay, so suppose you're raising your own children.

Richard Sutton 1:04:06 Mm-hmm.

Dwarkesh Patel 1:04:06 It might not be appropriate to have extremely tight goals for their own life or also have some sense of like, "I want my children to go out there in the world and have this specific impact, you know? My- my son's going to become president, and my daughter's going to become CEO of Intel, and like, together, they're going to have this effect on the world." Um, but i- people do have the sense, and I think this is appropriate, of saying, "I'm going to give them good, robust values such that if and when they do end up in positions of power, they do reasonable, pro-social things." And I think maybe a similar attitude towards AI makes sense. Not in the sense of, "We can predict everything that they will do," um, where we have this plan about what the world should look like in 100 years. But it's quite important to give them robust, and steerable, and pro-social values.

Richard Sutton 1:04:58 Pro-social values?

Dwarkesh Patel 1:05:00 Maybe that's the wrong word.

Richard Sutton 1:05:02 Are there universal values that we can all agree on?

Dwarkesh Patel 1:05:06 I don't think so, but that doesn't prevent us from, uh, giving our kids a good education, right? Like, we have some sense of, "We want our children to be a certain way."

Richard Sutton 1:05:14 Yeah.

Dwarkesh Patel 1:05:15 And maybe pro-social's the wrong word, actually. High-integrity is maybe a better word. Where if there's a request or if there's a goal that seems harmful, they will refuse to engage in it, um, or they'll be honest, um, things like that. And we have some sense that we can teach our children things like this, even if we don't have some sense of what true morality is, or everybody doesn't agree on that. Um, and maybe that's a reasonable target for AI as well.

Richard Sutton 1:05:41 So- so you're saying s-... We're trying to design the future and the- the principles by which it will evolve and come i- into being.

Dwarkesh Patel 1:05:48 Right.

Richard Sutton 1:05:49 And so you're saying... The first thing you're saying is, "Well, we will... We try to teach our, our children, um, general principles which will promote more likely evolutions."

Dwarkesh Patel 1:06:00 Yeah.

Richard Sutton 1:06:00 Um, maybe we should also seek for things being voluntary. If there is change, we want it to be, uh, voluntary, rather than imposed on people.

Dwarkesh Patel 1:06:09 Yeah.

Richard Sutton 1:06:09 I think that's a- a pretty important point.

Dwarkesh Patel 1:06:11 Yeah.

Richard Sutton 1:06:11 Um, and yeah, that's all good. And I think, I think this is like a big, um, you know, the big, the big or one of the really big human enterprises to design society, and that we... that's been ongoing for- for thousands of years again. And so- so it's like the more things change, really the more things s- they stay the same. We still have to figure out how to be. Uh, the children will still come up with different values that seem strange to their parents and their grandparents, and uh, and things will evolve.

Dwarkesh Patel 1:06:42 The- the more things change, the more they stay the same also seems like a good capstone to the AI discussion, because the AI discussion we were having was about how techniques which were, um, invented even before their application to deep learning and back propagation was evident hav- are, you know, central to the progression of AI today. So maybe that's a good place to wrap up the conversation.

Richard Sutton 1:07:04 Okay.

Dwarkesh Patel 1:07:05 Awesome.

Richard Sutton 1:07:05 Thank you very much.

Dwarkesh Patel 1:07:06 Thank you for coming on.

Richard Sutton 1:07:07 My pleasure.