Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again

Rich Sutton 0:00 People think I'm, have a radical point of view sometimes. They, they, they start questions saying how what I'm thinking is so different from everyone else. But I don't see it that way at all. I see it as like I'm thinking the ordinary way. It's just everyone else that's thinking a bit weird. [laughs]

Sonya Huang 0:16 [laughs]

Rich Sutton 0:17 And I mean that. Like, you know, it's just the recent times people are thinking weird. Before there was all this AI craziness, uh, you talk about... You wouldn't have to say continual learning 'cause it wouldn't make any sense to talk about learning that wasn't continual. All learning is continual. We always act and we learn. That's just the normal way of thinking. I'm not weird. [laughs] The field is weird. The field, they need to call it continual learning. It's just learning. [upbeat music]

Sonya Huang 0:59 We are honored to have the great Rich Sutton with us here today. Rich, you invented reinforcement learning. You wrote the seminal textbook. You had the key students in the field, uh, folks like Dave Silver. You wrote the essay, The Bitter Lesson, that I believe is the bible of the field. And, and you have just been one of the greats in propelling the field forward, so thank you for taking the time to join us today. Um, Rich is joined by Khurram Javed, his co-founder, uh, and former student from the University of Alberta. Um, the two of you have set off to found Oak Lab. I'm very excited to talk to you about that today. So for today's session, we're gonna start talking about The Bitter Lesson, the state of the world as we know it today, whether LLMs will get us there or not, and then we're g- we're gonna transition to start talking about your, your research agenda and your plan for Oak. Um, Rich, maybe take us back. I was gonna start with The Bitter Lesson, but I actually wanna start earlier than that. Decades ago, you decided to dedicate your career to reinforcement learning, to deep reinforcement learning in particular, and you established the University of Alberta as a bastion of that back when I think the field was very much in its infancy. What gave you the conviction to do that?

Rich Sutton 2:10 What else were you gonna do?

Sonya Huang 2:11 [laughs]

Rich Sutton 2:12 We're trying to figure out the mind, and learning is a central part of the mind, and having a goal is a central part of the mind, central part of intelligence. Yeah, so I was just doubling down on what I was always thinking.

Sonya Huang 2:26 Did people think you were crazy at the time?

Rich Sutton 2:30 Um, it was a winter. It was an AI winter. Um...

Sonya Huang 2:34 [laughs] What year was this?

Rich Sutton 2:35 It was in 2003.

Sonya Huang 2:37 Okay.

Rich Sutton 2:38 And it's kinda crazy actually, the truth, uh, 'cause I was, like, really sick. I was dying. I was actually dying of cancer in 2003. [laughs]

Sonya Huang 2:48 Oh my gosh.

Rich Sutton 2:48 And, uh, but I, I wasn't quite dead, you know. I, I've been trying for a number of years. And I, I wasn't dead. I was in another m- another remission, and so, so I said, "Well, I'm not dying. I, I haven't succeeded in dying, so I might as well, you know... It's going on long enough. I might as well just try to get another job." And so I [laughs] so I went to Alberta and, and, and started teaching there. And then in the end, I didn't die. It's kind of amazing. It's 'cause, you know. You know, it's, it's like that. I'm joking about it now, but it was quite serious. And, um, it, it's an even more poignant question. Why, why did I [laughs] continue, uh, to work on this research stuff when I was, you know, I only had a few months? I would always keep reminded what... I think it's Benjamin Franklin who's supposed to have said that, you know, if you ever wonder why someone is doing something, uh, it's almost always one of two things. It's either habit or vanity, okay? So I think it was, I think it was probably true. Maybe it was my habit to just kept doing what I'd always been, was doing, or maybe it was vanity. I, I don't know. I think it was more like habit 'cause I was, I was dying.

Sonya Huang 3:55 Wow.

Rich Sutton 3:56 Um.

Sonya Huang 3:56 Wow. Divine intervention.

Rich Sutton 3:57 Yeah. It's always been easy for me to ke- be very determined. Um, and, uh, I'm, I'm gonna go even longer on this answer.

Sonya Huang 4:06 Please do.

Rich Sutton 4:07 Uh, people think I'm, have a radical point of view sometimes. They, they, they start questions saying how, how what I'm thinking is so different from everyone else. But I don't see it that way at all. I see it as, like, I'm thinking the ordinary way. It's just everyone else that's thinking a bit weird. [laughs]

Sonya Huang 4:25 [laughs]

Rich Sutton 4:26 And I mean that. Like, you know, it's just the recent times people are thinking weird. If you could look back at what, what, what people thought about the mind for, you know, even just a decade, you'll find the kind of thoughts that, you know, learning is important. You've gotta have a goal. Um, and, you know, perception is important, and we have a, we are, we are low lev- we are low-level beings. We are generating actions and receiving data at a fast speed, and yet we have to think at higher levels. And, you know, go back a few... Before there was all this AI craziness, uh, you talk about... You wouldn't have to say continual learning 'cause it wouldn't make any sense to talk about learning that wasn't continual. All learning is continual. You know, you, it's not a special phase. We always act and we learn. That's just the normal way of thinking. I'm not weird. [laughs] The field is weird. The field, they need to call it continual learning. It's just learning.

Sonya Huang 5:22 I'm not weird. Everybody else is. That's a-

Rich Sutton 5:24 [laughs]

Sonya Huang 5:24 ... good, good motto to live by.

Alfred Lin 5:26 We're gonna have to send out a, an X post about that.

Sonya Huang 5:29 [laughs]

Alfred Lin 5:30 We're very happy that you s- you, you lived on. We're... The field is happy that you lived on, and thank you for pushing the frontier of AI.

Rich Sutton 5:36 I'm really happy. [laughs]

Alfred Lin 5:38 Thank you for pushing the frontier of AI. I'm sure really happy, and thank you for all of that, and push... And you've, like, been able to sort of educate a lot of students who push the frontier as well. How'd you pick them? How'd you f- over the last 20, 30 years?

Rich Sutton 5:51 Oh, well, you are giving me opportunities to be, uh, to be humble. Like, I like to be humble and point out how all these great decisions are, are just happen. And it's that, that's the way I feel about students. I don't feel that I choose them very well. I've just... Sometimes I'm lucky, sometimes I'm unlucky. I don't feel I'm particularly good at, at picking my students.

Khurram Javed 6:12 [laughs]

Rich Sutton 6:14 I'm looking at Khurram now. I think sometimes you end up with really great ones. David picked, David Silver picked me.

Khurram Javed 6:20 Yeah.

Rich Sutton 6:20 How is it that, that I got you, Khurram?

Khurram Javed 6:23 Yeah, that was also... So I finished my master's, not with you, uh, and I was planning to join industry, and then we were collaborating on a project, which also just ha- started organically. Like, there was something I worked on that Rich was in a meeting, then they mentioned that I worked on it, so I got pulled into it. We started collaborating. It went really well. Like, I felt so happy with that collaboration. Rich also felt really good about it. And then six months down the road, we had made some progress, and it just made sense to convert that into a thesis proposal. So at no point did I apply, at no point did I ask, "Should you be my PhD advisor?" We worked together, then we decided this would be a pretty good thesis, and then, then after that, I applied

Khurram Javed 7:07 for the PhD. Life works in unexpected ways. Uh, take us to 2019. You wrote The Bitter Lesson, which has become the, the modern tome. 2019 was a funny time to be writing that piece because ImageNet was 2009, AlphaGo was 2015. What caused you in 2019 to reflect and, and to write that? 'Cause it was before the current kind of scaling paradigm around large language models had taken off, but it was after deep learning had really proven itself.

Rich Sutton 7:36 Well, it was a long time coming. You know, as The Bitter Lesson expresses, it's something that you can observe for a long time, for many decades, and it's definitely at least as much due to the round of symbolic AI, which I lived through. It's all about not getting distracted by trying to put in your human knowledge and just paying attention to what the problem needs and how you can scale with computation. I know I, I, I wrote versions of it, uh, at least a year before, and I, I gave talks. I gave a talk a year before, and, um, it wasn't a particular response to the moment. It was a particular response to my, my long experience. Different people trying to think in different ways about how you can make smart systems.

Khurram Javed 8:28 Hmm. What is the essence of The Bitter Lesson? And, you know-

Rich Sutton 8:33 The essence of The Bitter Lesson

Khurram Javed 8:33 ... you know, it may be the phrase that I hear used the most in my meetings these days is, "Bitter Lesson pilled," "Is it not Bitter Lesson pilled?"

Rich Sutton 8:40 Yeah.

Khurram Javed 8:41 I would imagine, given the popularity of the fa- phrase, it's probably been tortured and misused in different ways that you didn't originally intend it. So what do you think, what is the essence of it, and where do you think people go wrong in their tr- in their attempt to understand it?

Rich Sutton 8:53 Yeah. You're, you're making me think about X now and my... I recently made a post where I tried to do the, The Bitter Lesson in, in 26 words.

Khurram Javed 9:02 [laughs]

Rich Sutton 9:03 It goes something like, uh, "Don't be distracted by human knowledge, as AI traditionally has been many times. Instead, focus on learning methods that will scale with computation," yeah, like search and like learning. So it's really all about, uh, focusing on algorithms and improvements. It's not, it's not saying you don't need f- fancy algorithms. You need fancy algorithms, but you want fancy algorithms that will scale with, scale with, uh, computation.

Khurram Javed 9:40 Rather than scaling with data.

Rich Sutton 9:42 Rather than scaling with human input. Yeah, and then the, the question, if I can anticipate, um... Yeah, what about large language models?

Khurram Javed 9:53 Yeah.

Rich Sutton 9:53 Are they-

Khurram Javed 9:54 Are they consistent or inconsistent with your-

Rich Sutton 9:56 Yeah

Khurram Javed 9:57 ... your essay?

Rich Sutton 9:57 And, and I thought about this, and I think there's an X, there's another X post about it. But the conclusion is that it's both a, a positive example and a negative example of The Bitter Lesson. First, uh, large language models enabled, uh, enormous scaling with computation, and you could just drink in the internet and scale so much. So it was a, it was a way of getting a much more capable system just by methods that scale. Then after that, as you go on further, um, it eventually gets limited by, by that information. The, the internet is finite, and it's hard to get more examples, and, uh, the world is big, and the world is m- massively bigger than everything we stored or on the internet. And so in the end, it seems like it could be, uh... I guess that would be a, a positive example of when, you know, we, we relied too much on human knowledge, and it eventually h- ho- holds us back.

Khurram Javed 10:59 Hmm. Can I just push on this a little bit?

Rich Sutton 11:02 Yeah.

Khurram Javed 11:03 It seems like a lot of what the foundation model labs are working on right now is synthetic data generation in order to kind of get us beyond the fossil fuel that is the existing human internet. Um, is synthetic data generation, kind of as part of this LLM scaling paradigm, is that a general method that leverages computation?

Rich Sutton 11:24 No, that's-

Khurram Javed 11:25 Why?

Rich Sutton 11:25 ... that's just a big mistake. [laughs]

Khurram Javed 11:27 Why?

Rich Sutton 11:29 Well, it's such a big, it's such a... Maybe it's the next, the next big lesson. Um, it's been floating around, uh, Alberta for five or 10 years.

Khurram Javed 11:40 Okay.

Rich Sutton 11:40 And, uh, we call it the big world perspective or the big world hypothesis. Khurram, who eventually wrote it up as a paper, there's a little paper called The Big World Hypothesis.

Khurram Javed 11:50 So the big world is that the world is incredibly big. There are infinitely many things to learn, and you can have people generating these synthetic data sets, but there will always be more things to learn. And because of that, if you could just

12:07 Learn from experience. If you could remove the humans from the loop, then you would have systems that can do everything because, you know, the world is big. There are many tasks that wa- we want them to do, and they would be able to do anything by learning from their own experience. Going back to the synthetic data question too, uh, who decides what's a good synthetic data and what's a bad synthetic data? Because I can write a program that can output a lot of synthetic data which would hurt programs. Right now, I would say humans decide, and that's the bottleneck where, okay, you can have humans deciding how to generate these data sets, but you need human experts who know what's a good data set and what's a bad data set for that approach to scale. So it is bottlenecked by humans.

Sonya Huang 12:45 Doesn't my loss curve decide, like how much better did I get with this data set versus that better data set?

Khurram Javed 12:50 Right, but if all the engineers, OpenAI, Anthropic, or all the big neo labs, their engineers went on vacation, who would generate the synthetic data? That's the question. It doesn't, doesn't come from agent's experience. It's not something that the agent is generating itself. Some human has to decide what is the right synthetic data to generate, and that requires human expertise. So for example, if you want a system to do something very challenging from a physics point of view, maybe you want a drone that flies with echolocation, like a bat, for example, um, what's the right synthetic data for that? I think you would need to hire domain experts to go figure out what is the right data and generate it, and then maybe you would be able to learn from that. But the domain expert has to exist first, so we are bottlenecked by human expertise at that point.

Sonya Huang 13:37 But you can have infinite synthetic worlds. The existing world is finite.

Khurram Javed 13:42 But let's go back to the echolocation thing, right? That's what I want. I want a drone that can, uh, localize itself and move with echolocation. That's my goal. Um, the robot th- that... It's a robot. It's generating its own experience, so it could totally learn from its own experience, but it wouldn't be able to. It doesn't matter how much synthetic data you generate. Doesn't matter if you generate synthetic data that captures 50 different universes. It would not be, not allow you to do that task without humans figuring it out first.

Rich Sutton 14:12 But first just say it's, it's the synthetic data is wrong. I mean, it won't be correct. It'll be a synthetic world. It won't be the real world, and it will matter. The world is incredibly complex. If you write a little program, 'cause it's gonna be a little program that'll generate the synthetic data, it'll be a very, uh, it'll be a small world.

Sonya Huang 14:35 Hmm.

Rich Sutton 14:36 So f- for example, what's important to me is what's going on in your mind right now, okay? And why don't... You're saying, "Why don't I get some synthetic data to tell me [laughs] what's going on in other people's minds?" No, there's no way we can have synthetic data for other people's minds, and other people's minds matter to us. You know, like I talked to you guys about investing today, so I care what you... going on in your minds. And how, how can I get synthetic data on such a thing? Really, you can't even get synthetic data on anything. You can't get synthetic data on, on how the drone is gonna interact with, with its environment in the physical world and the, the, the, the, uh, the friction and wear in the motors of this, of this robot. The world is infinitely complex, and any simulation of it is, like, microscopic. The, the big world hypothesis, let's say what it is, is that the world is massively more complex than your mind, than any agent's, any agent. And this is obvious because the world contains many other agents. So because the world is, is massively complex, has ma- th- you, there's no way you can do anything like, uh, anything that, that might claim to be optimal or perfect. Uh, you're going to be imperfect, and you have to have approximations, and those approximations will, will be severe. And so because of that, that is the ultimately the reason why we have to continue learning, if you wanna think of it as a reason. Um, we have to continue learning because we'll encounter some particular part of this immense world, and we'll have to learn an approximation that's tuned to the part of the world we're in, not to the, all the other parts that we're not in.

Sonya Huang 16:19 Yeah. I'm, I'm gonna push on this one more time. Um, and sorry, I'm being argumentative-

Rich Sutton 16:24 No, good

Sonya Huang 16:24 ... for the sake of being argumentative, but I'm trying to understand. My understanding is that the newest cohort of self-driving car companies, many of them were primarily trained in sim, and then they, you know, have to do some, some sort of, uh, post-training, I guess, to, to make sure they work in the real world, but, but that it's been a very effective pipeline.

Khurram Javed 16:43 Yeah. So I think the important question to ask here is how many eng- engineers were involved in building that simulation, and are we ready to say that the only problem worth solving are those where we can hire a large team of engineers to first make a simulation? And I'm sure they had to do multiple iterations where they made the simulation, they learned in it, they realized there was a sim-to-real gap that was not acceptable, then they fixed it. So there is this human in the loop fixing the simulation. Like, they're getting feedback from the real world, humans, and then they're fixing the simulation. Why can't we just remove the human and let the agent do it itself? And-

Rich Sutton 17:18 And then when it actually drives, again, something unexpected will happen.

Sonya Huang 17:22 And that's when you actually-

Rich Sutton 17:22 They just have to-

Sonya Huang 17:23 ... wanna learn from experience.

Rich Sutton 17:24 Yeah.

Sonya Huang 17:25 So your point is there's just so much more data that's gonna come from experience than there possibly can be from humans curating, creating data.

Khurram Javed 17:34 Yeah, and I think there is a, there is obviously value in learning from simulation, and, and there is a way of doing it. The agent can learn a model from its own experience, and when the agent learns it, it's very, it's much better because if the model is incorrect, it can fix it by continual learning. If the humans are making the simulator, then the model only gets updated when the humans figure out that something's wrong. So yes, planning is important. The agents should learn from, uh, simulators, but simulators they make themself.

Sonya Huang 18:01 Okay, I wanna move to another part of The Bitter Lesson, removing human knowledge. From your essay, quote, "Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain. But the only thing that matters in the long run is the leveraging of computation. And if I... You know, your former student, Dave, uh, with AlphaGo and AlphaZero, for me, that was a tr- an example of a triumph of removing human priors. Did that result surprise you? I guess why or why not?

Rich Sutton 18:31 Of course, it made me very happy. It made me, you know, feel vindicated. Um, you know, it could've gone either way. It wasn't that, uh, 'cause, 'cause prior knowledge can help. You know, there's nothing wrong with prior knowledge. You know, and, and I say this right at the very beginning of the be- right at the very beginning of The Bitter Lesson. I say there's no reason why there has to be a conflict between prior knowledge and then learning knowledge. You know, you can put some prior in there and then start learning. There's no reason in principle why these have to be opposed. In fact, they're all about knowledge. You know, life is gaining knowledge and having knowledge, and why are these... How somehow, you know, nature and nurture became enemies, but really, you know, prior learning is what you already... Yeah, and then, and then you learn more, and it's... They should be friends. But as I say at the beginning of The Bitter Lesson, in practice, they have been enemies. In practice, people who, who had a, an affection for existing human knowledge ended up, you know, wanting that to win, and so they wanted to, to minimize or, or dismiss learning. And so now I'm sure your sense of me is that I'm someone who, who loves learning and w- wants to dismiss prior knowledge. Um, but, i- you know, I'm really an, uh, someone who's interested in the mind. The mind is you have prior knowledge, and then you get more, and then once you've gotten more, then that becomes your prior knowledge as you get more and more and more. And this, these two things work together. I end up appearing to be, uh, someone who's, who's interested in learning primarily because all the rest of the world is, is, is, is talking about all you need is enough knowledge. You don't need to learn. You know, large language models are, "Oh, we're gonna put all this knowledge in the age into the system," and the large language model will not learn when it runs. Inter-- You know, it's talking to people. It's interacting. It is absolutely... The weights never change. So, you know, I am not the weird one. It's you guys that are the weird one that think that that's possible, that you could possibly... You know, they claim they can make, like, a PhD level experience and expertise out of something that doesn't, uh, learn at all anymore. You know? So, you know, I'm not the weird one. [laughs]

Alfred Lin 20:47 So your recommendation is let it, let the algorithms run for a much, much longer period of time before feeding it-

Rich Sutton 20:52 Continually learn

Alfred Lin 20:53 ... before you feed it data.

Rich Sutton 20:56 Well-

Alfred Lin 20:56 Or the prior data.

Rich Sutton 20:58 So-

Alfred Lin 20:59 Drip or drip the prior data along the way, prior knowledge.

Rich Sutton 21:04 So both are important, um, but in the long run, we've gotta gain and structure the gaining of new knowledge, and that's, that's what all that matters in the long run. And as you are doing this, yeah, there'll be some that you got previously. Um, like, how it work if we, you know, look into the future when we have intelligent robots, will, will, will we, uh, have them all learn from scratch, or will we, like, copy them and ask them to keep learning from wherever they are? I mean, we- they'll be digital, and it'll be easy to copy them. And so instead of having, like, this huge thing where we're spending zillions of dollars to retrain them from the internet, we'll just copy the agent and keep learning from there. And, and so in some sense, the prior knowledge will be, should be dismissed 'cause you're just gonna copy it from the previous robot.

Alfred Lin 22:02 So why don't you des- describe for us what you think a machine or a computer that learns from experience looks like?

Rich Sutton 22:09 Well, it could look like a robot. It could... But it also could live entirely on the internet. You could, like, for example, rou- routing of packets through the internet and do that in a way that's sensitive to experience and, and becomes better over time, or you can interact, be the user interface that's interacting with people, like on your phone or on your computer, and, uh, it becomes better over time. You know, like an intelligent assistant, you know, has to become better over time, has to know what you want.

Sonya Huang 22:37 Would your contention be that the current paradigm of, you know, the popular A- LLM-based assistants, would your contention be that these are not experiential learners or continual learners, and if so, what is the fundamental gap?

Rich Sutton 22:52 Are you serious?

Sonya Huang 22:53 [laughs]

Rich Sutton 22:55 I mean, obviously they-

Sonya Huang 22:56 They, they have mem-- They learn memories about me. They're, you know, they're, they're, they're doing some in-context learning.

Rich Sutton 23:04 Their weights never change.

Sonya Huang 23:05 And, and by the way, is a small number of the weights changing sufficient, or do you need all the weights to be changing?

Rich Sutton 23:10 Well, so all... Think of all the structuring and generation of new concepts that went into creating the large language models. All that is the weight learning. The... And you, you wanna continue, be able to continue doing that. You don't want that to happen just once.

Alfred Lin 23:28 Is another way of saying is we do too much pre-training and post-training before we launch the mo- the models, and they don't learn after that?

Khurram Javed 23:35 The only point that the big disagreement is we don't let them learn after that.

Alfred Lin 23:38 Yeah.

Khurram Javed 23:38 Right? So-

Alfred Lin 23:39 We don't let them learn after that

Khurram Javed 23:39 ... we can do as much pre-training as we want. That's okay. Post-training is fine. But then when I'm, when I'm using the model, I cannot... It, it, it stops learning. Um, you can give it more context. You can change the state of the model by giving it more context. And so it al- had already learned that if the state is different, if the state says something new, then it'll use that to make the next prediction. But the model is unlearning.

Sonya Huang 24:01 Cursor's tab autocomplete model.

Khurram Javed 24:03 Yeah.

Sonya Huang 24:03 It is, you know-

Khurram Javed 24:04 Yeah, so those-

Sonya Huang 24:04 It does get updated based on...

Alfred Lin 24:07 Those models-

Khurram Javed 24:07 Those two-

Alfred Lin 24:08 ... those weights change.

Khurram Javed 24:09 Those weights change. Those are two example, like Cursor's tab, and I think the Composer, they were also updating.

Sonya Huang 24:13 Yeah.

Khurram Javed 24:13 Those are two example of continual learning.

Sonya Huang 24:15 Okay.

Khurram Javed 24:16 It's can be much better. So the, the way they do it, as far as I understand, is a lot of people are using Tab. They collect all this data, so coming from millions of users or thousands of users, and then they do one update of the policy from this batch data. Um, so ima- uh, this could work, but imagine I want to teach this model something specific. I don't wanna fight with 100,000 other peoples about what they wanna create, teach their models. I wanna teach my model something very specific, and I want to do it to my version of the model. I... Like, I don't care about the shared knowledge that the model has coming from other people, and so it's a very inefficient way of doing it.

Sonya Huang 24:50 Hmm. It seems like the way that this is currently done is that there's fundamental skills maybe that are learned in the weights that are common to everybody, and then there's personalization that happens in the form of context.

Khurram Javed 25:03 Right.

Sonya Huang 25:03 Right?

Khurram Javed 25:04 Yeah, yeah.

Sonya Huang 25:05 Is that not the right mental model for how learning should work? Like, should, should all the context live in the weights themselves?

Khurram Javed 25:11 So context can be in the state, too.

Sonya Huang 25:14 Right.

Khurram Javed 25:14 Could be both, but you still need to be able to update the weights. So if I give you an example, some really good use studies are for, uh, with human disabilities. When human go through something that changes their mind or, or some sensors, you can see them adapt. So for example, we have proprioception. We have internal sensors that tell us where the... how the body's positioned, and we use this for walking. There are cases where people lose this ability completely, and then they can't walk at all because that is literally the foundation of their walking policies. It is ingrained in the brain. But then over the course of two, three years, they can learn to walk again by looking at their feet, so visual feedback through that. So brain is insanely plastic in the sense that it, it can learn a lot of things. Something that has been true for 20 years, when it stops being true, it can go and update that and get rid of that, and that is the capability I think that's extremely useful we would want in our systems.

Sonya Huang 26:09 Hmm. What is there for us to learn from how human babies or animals learn, and how much inspiration do you take from that?

Rich Sutton 26:17 Well, we take, uh, a lot of inspiration. We don't take it as a requirement that the AI has to behave like, uh, the natural system, like babies or people or animals. Um, but it's, it's a source of inspiration.

Sonya Huang 26:32 Yeah.

Rich Sutton 26:32 Inspiration, but not constraint from animal learning.

Sonya Huang 26:35 Be consistent with the bitter lesson. Uh [laughs]

Rich Sutton 26:37 Yeah, yeah.

Sonya Huang 26:39 Where do you think we should most seek to draw inspiration from the way that biological learnings works that is not present in today's systems?

Rich Sutton 26:47 [laughs] I feel like I'm just giving opinions now, but they're just obvious opinions. So, so I think it's apparent that no animal learns by supervised learning, because we don't get examples of w- uh, how our muscles should twitch, and that's our output.

Sonya Huang 27:07 But all of school is supervised learning.

Rich Sutton 27:10 I, I know. Absolutely not. Uh, but even if it was, school is, like, a tiny fraction of what we learn. Like, we learn to see, and we, we learn to walk, and we learn, um, how the world works. But ev- yeah, even in school, you know, no one tells us how we should twitch our muscles.

Sonya Huang 27:31 The knowledge skills I acquire were, were from supervised learning in school.

Rich Sutton 27:35 So I don't wanna say that the, the learning from, from others, transmission from others is not important. It's, like, extremely important, and language is extremely important. Um, but what are we, what are we missing? You know, there is, there is no supervised learning. There's no targets that are given to us. You know, you, you hear the right answer is, you know, where, where is... what's the capital of France? And we know the answer is Paris, okay? But no one tells me how I should pronounce Paris. You say the answer is Paris, and I listen to you, and I hear your words, and, you know, I will make some other, uh, muscle motions to produce the answer Paris. It's not literally supervised learning. Um, anyway, yeah, so I, I think it's really true. I mean, uh, eh, well, anyway, the first thing is that school is, is irrelevant. Like, you know, squirrels don't go to school and, and, and learn that.

Khurram Javed 28:37 [laughs]

Sonya Huang 28:37 They might.

Rich Sutton 28:38 Animals don't learn that way. It's... and school is a very special thing that, that even we didn't have up until, you know, I don't know, a few hundred years ago.

Sonya Huang 28:46 But isn't it something we've evolved to?

Rich Sutton 28:47 It's not part of, it's not part of the essence of intelligence, and it's a distraction to think of that as your primary example of learning as this thing which we didn't do as animals.

Khurram Javed 28:59 I wish you had been around to tell my parents that and force me to have g- go to school and deal with all the structure.

Sonya Huang 29:08 But, but, but look, the, the thing is, like, squirrels are wonderful at jumping off trees, but squirrels can't prove math theorems. And if I wanna learn how to prove a math theorem, I go to school.

Rich Sutton 29:14 Yeah. Uh, they also don't have, uh, DVDs and-

Sonya Huang 29:20 [laughs]

Rich Sutton 29:21 ... and iPods. You know, there are a lot of things. Th- they can do things that we can't do. Um, but math theorems, uh, yeah, and they don't play chess. You know, sort of like Moravec's paradox. They're, uh, they're these advanced things that we think of as really intelligent, but, uh, they're sort of easy for computers to do as opposed to all these regular things that are hard, like moving and seeing with attention and everything. Um, I think supervised learning is a, is a good thing. You know, just mentions... I like to think, look for obvious things. No one tells us how to twitch our muscles by giving us examples, 'cause they couldn't possibly, 'cause we have had to twitch our muscles. We've had to figure that out.

Sonya Huang 30:06 Yeah.

Khurram Javed 30:06 And their on- answer would be wrong, right? So if I move my mouth and my tongue and my vocal cords exactly the same way that Rich does to pronounce Paris, I'm sure a very different sound would come out. So in some sense, Rich or no one knows the right way of producing a sound with my body. Only I know that.

Sonya Huang 30:24 Yeah. It seems to me that many of the most, I guess, the most raw, like, sensory motor capabilities, especially related to movement in the physical world, I agree with you that that seems something that's inherently learned from experience. It seems to me, though, that there are higher levels of abstraction that bring us closer to, you know, what makes humans great, and much of that doesn't live in this le- low level of sensory motor learning. Does your world model, I guess, span sensory motor learning all the way up?

Rich Sutton 30:56 Yeah, that's the ambition. Absolutely. And squirrels, by the way, can do some enormously abstract things.

Sonya Huang 31:03 What's the coolest thing a squirrel can do?

Rich Sutton 31:05 Well, it can always get into your bird feeder [laughs]

Sonya Huang 31:08 [laughs]

Rich Sutton 31:08 No matter what obstacles you put in the way, you know, it can find new ways to jump and climb and-

Sonya Huang 31:14 Okay

Rich Sutton 31:14 ... and do lots of things.

Alfred Lin 31:16 They calculate trajectories pretty well. Animals are pretty good at understanding the physical world without the mental calculations that we think we are doing when we think about launching ourselves into space.

Khurram Javed 31:29 Breaking a fall, they can do it in real time, the right way to prevent injuries.

Sonya Huang 31:33 Okay, fair enough.

Rich Sutton 31:34 I think it's just a question of degree between... You know, I like to think that animals, other animals, are, are very close to humans. I think it's hubristic to try to emphasize what we do differently, you know, how we're different from animals. It's better to see the commonalities, and I, I think we are just a question of degree. It's, it's degree and, of course, society and culture give us big advantages. Language gives us big advantages.

Alfred Lin 32:00 Can I just push on some of

Alfred Lin 32:02 this?

Rich Sutton 32:02 Yeah, good.

Alfred Lin 32:03 Because I wanna back up Sonya. So I believe animals and children learn from experience and do incredible things learning from experience. And when my son was two or three or four, I'm like, wow, this is really interesting that they're... my son can learn these things without nobody really teaching him how to do these things. But at the same time, what Sonya's saying is, like, what makes human uniquely human, to be able to go to outer space, build a rocket, those are not things that are learned 100% from experience. Because before you launch the rocket, you actually have to abstract thinking through it in a way that is not learned from, quote-unquote, experience, because you don't know if it's gonna work or not. You have to imagine it. How do you... we teach a machine to imagine things that were not available before? That's probably the thing that we're trying to, like, push on because that we're not quite understanding that.

Rich Sutton 33:04 We're, we're absolutely gonna agree with you there. You have to be able to plan. You have to be able to imagine.

Alfred Lin 33:09 Yeah.

Khurram Javed 33:10 Would you say that humans a thousand years ago, before they had done all, most of the things that we're talking about, were they as intelligent? Um, if, for example, someone from that era was exposed to this new culture, would they be able to get the same skills and, and start doing useful things?

Alfred Lin 33:27 Even over the last 10,000 years, I don't think the human brain has evolved that much because evolution-

Rich Sutton 33:31 Fundamentally the same machine

Alfred Lin 33:33 ... fundamentally the same machine, but we've built up 10,000 years of knowledge.

Khurram Javed 33:37 Yes, so-

Alfred Lin 33:37 And I get to learn 10,000 years of knowledge by going to school through supervised learning.

Khurram Javed 33:42 Right.

Alfred Lin 33:42 And I get all that much, much faster than trying to learn through experience.

Khurram Javed 33:47 So-

Rich Sutton 33:47 Right

Khurram Javed 33:47 ... so I think you're, like, totally right. So we, we would want our systems to learn from experience, and part of their experience would be getting exposed to our culture and then learning from, about our culture. They should learn from that. That's all good. But let's talk about when someone goes and does a paradigm-shifting thing. So everyone gives the example of Einstein, but I think there are many examples. Learning is that too, like looking at learning thing versus, uh, programming things. So when these paradigm shifts happen, I would say it's a human who has accumulated all this knowledge, and then from their experience, they're building new abstractions, they're planning with them, and then they're discovering new knowledge. And that skill of, of coming up with new abstraction and then learning more models and planning with them, that problem is to- that skill is totally missing in our current systems. And you can expose this at the edge of human knowledge, but you can also study this problem at the sensory motor stream level.

Rich Sutton 34:41 So we're not arguing with the, with the, uh, principle. We need to form abstractions, so we can reason at a high level. You guys are, are coming close to doing that thing that I said we should never do, which is argue is prior knowledge important or gaining knowledge important? You know, that's what you guys just said. You said, "It's a... You're still gonna have to learn things." And you were saying, "Oh, I can get things from my culture and from prior knowledge." But these should not fight for each other.

Alfred Lin 35:10 On the exact thing around paradigm shifts, how do we create a machine that understands when to shift the paradigm?

Khurram Javed 35:17 Yeah, I think through its experience, right? So it, it would have to, through its own experience... It can't rely on human knowledge because we're assuming the humans see one paradigm, and we want a different way of looking at things. And so through its experience, it has to find something that is better. Maybe it generalizes better and makes better predictions. Maybe it's better in some other ways, but it has to be through its own experience.

Rich Sutton 35:41 The big challenge that we don't see in our field, the ability we don't see in our field yet, is the ability to learn a model and then plan with a model. We can do the, the math things, and we can do AlphaGo because the games, we know the model. We know how the moves work. And in math, we know what the operators are. We, you know, we know lean will take us from one state of knowledge to the, to the state of the proof to the next state. But if we have to learn the models, there are no... I'm, I'm gonna say it. It's probably maybe a weird example, but a counterexample. But I'm gonna say there's no instances of learning the model and then planning with the model in our field.

Khurram Javed 36:23 At least not with, uh Like self-discovered em- abstractions. So there are people who say, "I'm just gonna learn a model of what happens in the next second or next millisecond," but that's not how our models work. Our models are more abstract. Our models are, um, quite different.

Rich Sutton 36:42 Mm-hmm.

Sonya Huang 36:42 So one of the things I like about what you're doing here is you're not just sitting around pontificating or lamenting the state of the world as it is. You're very action-oriented. It's why you started the company. So let's, let's start talking about that a bit. Uh, in 2022, Rich, you laid out a very specific 12-point plan, the Alberta Plan for AI Research. Maybe tell us about that.

Rich Sutton 37:04 So the Alberta Plan came about because we did have general ideas, but we also needed to convert them into smaller chunks. And so the 12 steps are the attempt to crystallize particular chunks. There's a very important early step, step two, uh, and which is continual deep learning, and w- and we think that one is, like, almost the most important because it unlocks everything else. If you could do continual deep learning, you could then continually update your model of the world, and then if you knew how to do the abstraction rights and, like, the second half of, of the, the steps are all about how to get the abstractions right.

Sonya Huang 37:47 Mm.

Rich Sutton 37:47 So, and not only by abstractions right, what I mean by, I don't mean get the right abstractions, 'cause no one can say what the right abstractions are. That depends on the worlds you're in. Your agent would have to learn the correct abstractions for whatever world it's in. And so, you know, if you, maybe those are the two key things. You have to find the right abstractions, and then you have to do, able to do continual deep learning.

Khurram Javed 38:10 I think that a lot of the people in the field realize that we need models, we need to plan with them, but the abstractions tell us what the model should be conditioned on. So what should you, what should the model predict? What are you gonna do, and then something is gonna happen? And more importantly, where would that come from? So I really like the example of, um, elite athletes. If you ask elite athletes about how they do certain things, they would have weird niche terminologies for doing very specific things. They were like, "You know, I do this thing," and they would have a name for it if they communicate. Sometimes they don't even have a name for it if they're just doing it alone. So how did they come up with those abstractions? That's, in some sense, a crucial thing that's missing that the later half of Alberta Plan

Khurram Javed 38:53 answers.

Sonya Huang 38:53 Can we talk about the continual deep learning part? Is it an algorithmic gap that exists today, or is it a, just a practical deployment infrastructure data privacy gap? Because if I wanted to do, call it naive updating of weights based on user interaction, I can do that today, right? And so w- what, what in your opinion is the biggest thing that we're missing to kind of get to continual deep learning?

Khurram Javed 39:18 Yeah, so it's absolutely an algorithmic gap. You, you can do the naive thing, but then you'll see all sorts of problems. So for example, if you say, um, "I'm gonna take, uh, one sample and then I'm gonna update my whole model with that one sample," you will run into this problem that now all of the previous knowledge in the model, it's impacted negatively. And the way currently we s- we get around this is exactly what Cursor does. They don't use one example. They use a large batch coming from a lot of users. So in use cases where you can have that, you can do continual learning, but most use cases you don't have that. Most ca- use cases you have a single stream of data, and then if you apply it to the naive thing, it just completely destroys your prior knowledge in a very, um, destructive way.

Rich Sutton 40:00 Catastrophic forgetting.

Sonya Huang 40:02 Yeah.

Khurram Javed 40:02 That is so, yeah.

Rich Sutton 40:04 But it's totally curable.

Khurram Javed 40:05 [laughs]

Rich Sutton 40:06 You have to have the right algorithm.

Sonya Huang 40:07 What's the cure?

Khurram Javed 40:07 Well, yeah, exactly. [laughs] What is the cure?

Rich Sutton 40:09 Well, you know, first you need to do st- what we call step size optimization, and it means every weight in your network has to have a, a separate step size. So some will move fast, some will move slow, and you will n- we, you will have to meta learn these step sizes for each weight. Most of your network will be, have, have weights that have tiny step sizes, so then when you train on a new example, they don't get destroyed.

Sonya Huang 40:36 Mm-hmm.

Rich Sutton 40:37 Which happens just to the right places. And then secondly, you have to use some form of generate and test, um, which is, uh, in feature space. So you come up with new features or new units and, and without following gradients, 'cause gradients is a very slow process. You only move in a direction if you know it's the helpful one, and, uh, that's always gonna be very slow and doesn't give you a path to grow more and more complex and, and to have sustained learning. You need to have something that just proposes, uh, a bunch of new units and, uh, a- and, and, and then goes from there. I guess, so there is a specific thing I can say that, to make it at least concrete, which is to say we have this algorithm called continual backprop we used, uh, published in, in Nature a couple years ago, and it is exactly like backprop, but every, but you also plant new seeds of units that are re- newly initialized with random weights. Backprop only has random weights at the beginning of time, and then as you go on, all that randomness, all that variety from the randomness gets used up. And with continual backprop, you keep injecting a bit of randomness, a bit of generate and test, a bit of generate, and then the, the, the operation of, of backprop is the tester. So you need s- you need that, and, and if you put those together really well, I think you'll have a new generation of massively, um, superior, uh, continual deep learning.

Sonya Huang 42:07 Mm.

Rich Sutton 42:07 And that's what we hope to do in the next couple years.

Sonya Huang 42:10 Wonderful. Do you think that these algorithms can be applied to the current state of affairs with people scaling LLMs and trying to get them to do continual learning without catastrophic forgetting?

Khurram Javed 42:22 Yeah, absolutely. I think it, uh... So I don't think that you could take an existing model and say, "I'm gonna just start updating it with these algorithms," because these algorithms meta learn how to learn. So really you have to say, "I'm gonna learn from scratch," so let's say learn a new foundation model, but I'm gonna learn with this, these new algorithms. These new algorithms, in addition to Learning the knowledge, they're also going to learn how to learn future things. So they're learning two things at the same time, and then, um, then I think you would be able to learn new things without catastrophic forgetting.

Alfred Lin 42:57 Is that the most radical thing in, that you're trying to do in your company, in terms of from the current state of affairs, yeah, to try to do these two things at the same time?

Rich Sutton 43:06 Most radical thing. Is... I think that's [laughs]

Sonya Huang 43:09 This goes back to I'm not crazy, everyone else is crazy.

Rich Sutton 43:11 [laughs] Yeah.

Khurram Javed 43:13 Well, that's perhaps not totally radical. There was a point in, like, 2016 to 2018 where a lot of people were exploring these ideas, uh, quite a bit. They were doing it in a much more limited setting, so they would say, "We have a distribution of problems, and then in this specific case, we'll do it," whereas we wanna do it from a single stream of experience, so our method should be more generally applicable. So I think pe- many people have explored this, but no one has explored this in the general setting where the resulting algorithm would be applicable

Khurram Javed 43:43 everywhere.

Alfred Lin 43:43 So what would be the most radical thing that your company, your new company is trying to do that other people are not doing?

Rich Sutton 43:49 What's the most ambitious thing? Uh, remember, I don't think I'm weird, so I don't wanna say it's radical.

Alfred Lin 43:55 All right. Let's, what's the most ambitious thing?

Rich Sutton 43:56 The most ambitious thing, I think, is to try to have the full spectrum of knowledge, both about the tiny things and about the big things. You know, like thinking about how you take an airplane from one city to another. That's a, a very big... You know, it's, it's more, it's like your, your, uh, space flight example, but it's just kind of more commonsensical to think about, because we all, many of us take airplanes, and e- all of us use abstractions in all kinds of our life, and even, even the squirrels use abstractions. So to have that spectrum of, of knowledge, uh, from the small to the big, and to treat it in a uniform way, and to be able to hel- have it self, uh, maintaining. You know, the big question is always you have your knowledge-based system, and what keeps the knowledge in it correct? Well, what keeps the knowledge correct in a large language model is, well, people did a lot of post-training a- and they, they, they made it sure it was correct, and then they freeze it after that. So that's what keeps it correct. But really, our minds, we are, we're always changing things, and yet something keeps it organized and coherent and, and, and settling back into a good place rather than drifting off into crazy land. That is in, I think, our, our biggest ambition, to have a mind that is self-consistent a- a- and can keep training itself and making it coherent.

Sonya Huang 45:26 Hmm. I love that. Can I ask, it almost seems that it's, it's such a ambitious vision, and the idea that all these things can be unified into a single mind is so ambitious.

Rich Sutton 45:37 It's within reach.

Sonya Huang 45:39 Well-

Rich Sutton 45:39 I think it's within reach. It's, here it's 2026-

Sonya Huang 45:41 Yeah

Rich Sutton 45:41 ... and our computers are so fast. You know, is it, is it so ambitious that it's out of reach, uh, or, or do we have already inklings of how all the steps can be done? And we, I, I think we have a vision and inklings. Um, I don't think it's, I don't think it's inappropriate.

Alfred Lin 46:02 Your vision involves a trillion parameter model with t- 20 watts. [laughs] That seems pretty ambitious.

Khurram Javed 46:10 That is ambitious. Uh, in some sense, with current technology, I would say it's also impossible. Like, just storing a trillion parameters in memory would probably use more than 20 watts of energy with current memory tech- technologies. But it's, we are really thinking of, okay, things are getting better, computation is getting cheaper, it is getting more energy efficient, so where would be, would be in five to 10 years? And I think five to 10 years with the right algorithms, and, um, we can totally be in a world where this would be possible.

Rich Sutton 46:44 So five to 10 years is two orders of magnitude of Moore's Law. The, that's the standard improvement if we double every 18 months. 10 years would give you two orders of magnitude. And so for Khurram's statement to be plausible, then today you should be able to do it for, for, what, 20 watts, two orders of magnitude, two-

Khurram Javed 47:08 Two thousand

Rich Sutton 47:08 ... 2,000 watts.

Khurram Javed 47:09 Yeah.

Rich Sutton 47:09 If you can do it with 2,000 watts today-

Khurram Javed 47:11 Yep

Rich Sutton 47:11 ... then in 10 years you'll be able to do it for 20 watts.

Alfred Lin 47:14 You think you can do it for 2,000 watts? You have-

Khurram Javed 47:17 I think-

Alfred Lin 47:17 ... lots of people at research labs that have access to way more than that.

Khurram Javed 47:21 Yeah, I think we can, can be more efficient than that even now with the right algorithm.

Sonya Huang 47:28 If we can be more efficient than that, then why aren't we? It's not like people just want to spend all their money, um, spend all our money.

Rich Sutton 47:35 Sometimes it seems like they-

Sonya Huang 47:35 I'll be like, "I'll give my, I'll give money." [laughs]

Rich Sutton 47:38 Sometimes it seems like they want to.

Khurram Javed 47:39 Yeah. I, I-

Rich Sutton 47:40 Doesn't it?

Khurram Javed 47:42 I think-

Rich Sutton 47:42 I mean, that's how they show they're, they're real men, by using lots of energy. [laughs]

Khurram Javed 47:46 At least when I look at different research groups, I don't even see anyone believing in it's possible, and I think if you don't believe in it, you're just not gonna work on the technical problems and work

Khurram Javed 47:56 through them. Um-

Alfred Lin 47:56 Is it that it's not possible, or it's that there's so much waste in the system? Like, one, which one is it? Or is it, is, is there someone who knows how to do it efficiently-

Khurram Javed 48:05 Yeah

Alfred Lin 48:05 ... and then there's 10 times the number of people in the same lab doing all these other things, and so-

Khurram Javed 48:13 There's-

Alfred Lin 48:13 ... nine out of 10 people are wasting?

Khurram Javed 48:16 In some sense, I, the way I think about it is that we are stuck in a local minimum, so if we want to move towards these new kind of algorithms, it is almost impossible that things will not get worse before they get better. So when we start exploring these new directions, you're not gonna get state-of-the-art performance from day one, but it is, because it is a different paradigm. Um, but that path leads to similar performance at a higher energy, uh... And these big labs, they are so locked into a product that they Like, it, it is not possible for them to pursue a path where things get worse, worse

Alfred Lin 48:49 Because their current paradigm allows them to keep scaling, and this new paradigm, they have to take a bet, and they-

Khurram Javed 48:56 And they have to figure out some, some technical things that are difficult that we have thought about it for many years. We know people who have thought about these things for many years, and when I talk to them, it makes sense that it's doable. But you need to think about those challenges for a long period of time.

Alfred Lin 49:13 So if everything goes right with Oak, what happens with the company? What do you-- what kind of company are you building?

Rich Sutton 49:20 Uh, if everything goes right, we, uh, implement the architecture. We can have genuine, uh, continual learning, and we can form abstractions so that we can do planning and reasoning, and, and we ha- so, sort of like true intelligence. And then, you know, it's hard to imagine just exactly what will happen by then. But I think-

Alfred Lin 49:41 Humans will become irrelevant? [laughs]

Rich Sutton 49:43 I, I, I don't think that's true at all. I, I, I-

Alfred Lin 49:46 We don't either

Rich Sutton 49:47 ... I, I think the world becomes exciting and in- and even more exciting and interesting, uh, for humans. But in particular, I think there are the-- You have to wonder about the large language models. They might be at risk, uh, when this eventually happens. You know, I'm sure they'll get a good run. They've already had a good run. You know, they've been very successful. And let me say, just for, for clarity, that, uh, large language models are, are an amazing scientific breakthrough, a breakthrough in the skillful use of language by neural networks. It's totally unanticipated. You know, it was a, it was always a holdout for, uh, symbolic methods in language, and the, the, they have totally changed how that's thought about now. Yeah. It's a, it's a big breakthrough. So f- it's frustrating to me that we have to, you know, just celebrate that we've made this great progress in this sub-subset of the problem of AI and enjoy that. Instead, it has to pretend to be all of AI. All of intelligence is not f-fluid, capable use of language. There's so much more. It's an important part. You know, it's like 20% or a quarter of intelligence. There's m- there's more.

Sonya Huang 50:59 Yeah.

Rich Sutton 51:00 We're not done.

Sonya Huang 51:01 Yeah. If everything goes right, are you imagining that there's a single mind that can do everything from learn how to swing from tree branches, to make a spaceship, uh, to, you know, all these various things we've talked about today? Is it a single mind, and is it a single set of weights that can, that can do all these things, or is it a s-

Rich Sutton 51:22 It's a single design.

Sonya Huang 51:23 Okay.

Rich Sutton 51:24 There'll be many different... There'll be many minds.

Sonya Huang 51:26 Hmm. Okay, so it's a single design-

Rich Sutton 51:28 Decentralization

Sonya Huang 51:28 ... that reacts to different environments.

Khurram Javed 51:31 And it, it, it... And different versions of that mind would learn different things because they have different experience. This sort of goes back to the big world hypothesis that there are infinitely many things to learn, so one system cannot learn infinitely many things. And, like, I think Rich already mentioned this, but if you have two of these systems, if you have two of the largest systems in the world, then it is trivial that they cannot model each other because they're equally complex. So a single system would never be able to get to a point where it can learn everything. It would always be multiple systems that are learning from their own experience.

Alfred Lin 52:04 You guys are hiring. What kind of people are you looking for?

Khurram Javed 52:06 We are hiring. Uh, the initial team, most of it we already have in our mind, so these are people who have thought about these ideas in the pa- in many, uh, you know, for a long time. And we are gonna take a slightly different approach because this is a different paradigm. It doesn't make sense to become large very quickly because, uh, in some sense, everyone we hire has to, uh, come to see what we see, and not everyone sees that. So we're gonna start small, slowly grow to maybe a handful or two or three, uh, and then go from there.

Rich Sutton 52:41 We want to be super aligned.

Khurram Javed 52:42 Want to be super aligned.

Rich Sutton 52:43 So that we can be, um, very productive working together and, and scaling the progress.

Sonya Huang 52:50 Absolutely.

Alfred Lin 52:51 Very, very cool.

Sonya Huang 52:52 Wonderful. I loved this conversation. Thank you for taking the time to share what you're up to.

Rich Sutton 52:58 Thank you, guys.

Sonya Huang 52:58 You are, you know, un- an unusually deep thinker about where reinforcement learning and algorithmic design will go, and it was a true pleasure to get to explore it together with you today, so thank you.

Rich Sutton 53:10 Thank you very much.

Alfred Lin 53:11 Yeah, thank you for having us.

Rich Sutton 53:11 Our pleasure. [outro music]