Building AlphaGo from scratch – Eric Jang

1. 0:00 – 8:06

   Basics of Go

   1. DPDwarkesh Patel0:00

      Today I'm here with Eric Jang, who was most recently vice president of AI at 1X Technologies. Before that, senior research scientist at what is now Google DeepMind Robotics, and you've been on sabbatical for the last few months. One of the things you've been doing is rebuilding, and improving, and hacking on AlphaGo. And so we're, today what we're gonna do is you're gonna explain building AlphaGo from scratch and what it tells us about the future of AI research and development. But, uh, before we get to that, why is AlphaGo interesting? Why is this, why is this the project you decided to do on sabbatical rather than just hang out at the beach?

   2. EJEric Jang0:33

      Sure, yeah. Um, I like making things, and AlphaGo and Go AI is one of those things that really got me into the field. Uh, when I saw the kind of early breakthroughs, um, on AlphaGo in 2014, 2015, 2016 and so forth, it was just profound to see, you know, how smart AI systems could become and the, the kind of computational complexity class that they could tackle with deep learning. Um, this is a problem that has, you know, long been understood to be kind of intractable for a search, and yet, um, it was solved, um, through, through deep learning. And so, so that was quite mysterious to me, and I've always wanted to understand that phenomena a little bit better. My training is often in deep neural nets for robotics, where it's, uh, the, the decisions made by the neural networks are a bit more intuitive, but AlphaGo is a sort of problem where the, the decisions are actually the result of a very, very deep search. And it's always been very mysterious to me how, like, a 10-layer network can sort of-

   3. DPDwarkesh Patel1:30

      Mm-hmm

   4. EJEric Jang1:30

      ... amortize the simulation of-

   5. DPDwarkesh Patel1:31

      Yeah

   6. EJEric Jang1:31

      ... something so, so, uh, so deep in the, in the game tree.

   7. DPDwarkesh Patel1:35

      Yeah, interesting.

   8. EJEric Jang1:35

      So if you plot out how much compute it took to build various iterations of strong GoBots over the years, you can see that in 2020 there was a open source project called KataGo, um, by David Wu from Jane Street, who, who basically achieved a 40x reduction in compute needed to train a really strong GoBot, uh, tabula rasa. Um, I'm not certain if it's stronger than AlphaGo Zero or AlphaZero or MuZero, uh, but it's very, very strong, and this is what most Go practitioners today train against when they're, when they're playing an AI. And thanks to LM coding, what took a whole team of research scientists at DeepMind and, you know, millions of dollars of research and compute can now be done for, you know, a few thousand dollars of, of rented compute.

   9. DPDwarkesh Patel2:14

      Okay. I guess we should first discuss how Go works.

   10. EJEric Jang2:16

       Great.

   11. DPDwarkesh Patel2:16

       So, yeah, how, how does the game work?

   12. EJEric Jang2:18

       Um, so the game of Go is a very simple one that can be implemented, uh, quickly and easily in a computer. The, the objective of the game is basically to put down black and white stones and try to occupy as much territory in the game as possible. So I might start by putting down a black stone. Uh, black always goes first. So go ahead. And so the way you capture an opponent's stones is, like, for every, um, intersection, if you can surround all four of its neighbors with, um, with your stones, then, um, then this one is sort of cut off from oxygen, if you will, and then it, uh, and it, it, it, it is a dead, dead stone. So, so then now I control these four stones as well as this empty intersection here.

   13. DPDwarkesh Patel2:57

       Mm-hmm.

   14. EJEric Jang2:57

       So there's, like, slight variations between Chinese, Japanese, and, uh, what is called Trump-Taylor rules. Um, Trump-Taylor rules are designed to be completely unambiguous for Go, so this is what all Go AIs train against-

   15. DPDwarkesh Patel3:08

       Mm

   16. EJEric Jang3:08

       ... and resolve against. So in typical Go, like when, like, humans play, you're actually not allowed to put this white stone down here.

   17. DPDwarkesh Patel3:15

       Mm.

   18. EJEric Jang3:15

       It would be instant suicide. Um, in Trump-Taylor, it's actually fine. You put it down, and then it immediately resolves to death.

   19. DPDwarkesh Patel3:20

       Mm.

   20. EJEric Jang3:20

       So the outcome is sort of the same. Let's go ahead and start over and, and play, play a few stones, and then I'll explain some more.

   21. DPDwarkesh Patel3:26

       Cool.

   22. EJEric Jang3:26

       So I'll just start there.

   23. DPDwarkesh Patel3:32

       All right. I'm, like, basically playing randomly here, but I'm trying to get around your stones and see if I can-

   24. EJEric Jang3:37

       Sounds good

   25. DPDwarkesh Patel3:37

       ... surround them.

   26. EJEric Jang3:37

       Yeah.

   27. DPDwarkesh Patel3:40

       Hmm.

   28. EJEric Jang3:45

       So this move, um, basically exposes one empty neighbor for your white stone, and it's very akin to a check in chess-

   29. DPDwarkesh Patel3:53

       Hmm

   30. EJEric Jang3:53

       ... where if you don't respond immediately by putting one here, then I can immediately capture this.
2. 8:06 – 31:53

   Monte Carlo Tree Search

   1. DPDwarkesh Patel8:06

      Let's understand how, um, AlphaGo actually works and how somebody in the audience might be able to implement it.

   2. EJEric Jang8:12

      Great, yeah. Let's start with, um, kind of an intuition about the underlying, um, you know, search process used to make moves, and we'll layer on, uh, ideas from deep learning to make it much more efficient-

   3. DPDwarkesh Patel8:25

      Mm

   4. EJEric Jang8:25

      ... and tractable. So Go is a game where there's just two players. We're gonna draw a person here, and we're gonna draw an AI here. And, um, let's say this person is playing black, so they go first. So we're gonna draw. They go here. And then now the AI, um, is going to make a move based on what it sees here. So there's a question of, like, how you encode these inputs into the AI. Maybe you could use ones and zeros, but you wanna represent, um, you know, black, white, and empty. So, so you would need at least three different values here, right? So maybe you could use zero, ones, and twos or something. So, so the AI might see something like, you know, zero, zero, zero, zero, uh, one. Great. So, so this is the input to the AI, uh, on its turn.

   5. DPDwarkesh Patel9:25

      Yeah.

   6. EJEric Jang9:26

      So, so the AI can choose, let's just pick three possible random moves that it can go, and I just drew these at random. And so which, which move is best here, right? Well, we don't know until the game ends. Um, there's no-- Go does not have any kind of local reward of which move here is good. And this is what makes Go a very difficult game, is that you don't actually know who won until you really get to the end of the game. So how deep is this tree, right? Well, in a 19-by-19, um, Go board, there are, uh, you know, roughly to the order of 361 moves on any given, uh, move, and of course, as it fills up, you have less moves. Um, and, and the, the number of steps in the game can be somewhere from 250 to 300 moves, and maybe experts might, uh, decide to end the game, uh, well before that. But, uh, you know, under Tromp-Taylor scoring, you actually have to play things all the way to the end, so this could be like 300 moves or something. Right. So, like, 300, um, like, depth of the tree.

   7. DPDwarkesh Patel10:18

      Yeah.

   8. EJEric Jang10:19

      So if you keep on expanding possible moves here, so th- in, in this move, the AI is going, and then, you know, here the human would go. And then, you know, there's, there's some... and so forth. You can find that, like, essentially what you end up with is an enormous explosion in the possible game outcomes originating from just this one s-state. So this is something to the order of, like, you know, 361 to 300, power of 300, which is far more than the number of atoms in the universe, right? Like, it's, it's just, uh, it's just... And, and of course, actually there are redundancies and symmetries, so it's not actually 300, but, but that's sort of the... If you were to do a naive tree where there were no merging of children, then actually you end up with a tree about this big.

   9. DPDwarkesh Patel11:09

      What do you mean by merging of children?

   10. EJEric Jang11:10

       Right. Let me, uh, use this board here. So if we start here, and then you play here, and then, um, I play here, and then you play here, that is equivalent to I start here, you play here, I play here-

   11. DPDwarkesh Patel11:25

       Yeah

   12. EJEric Jang11:25

       ... and then you play here, right?

   13. DPDwarkesh Patel11:27

       Yeah.

   14. EJEric Jang11:27

       So, so both of them arrived at the same spot but through different paths. So this child node can be thought about as a shared ancestor.

   15. DPDwarkesh Patel11:34

       Backtrack.

   16. EJEric Jang11:34

       Yeah.

   17. DPDwarkesh Patel11:34

       And I guess it's not 361. It starts at 361, but it dec-decreases by one each time.

   18. EJEric Jang11:38

       Yes, and the branching factor decreases by one each time.

   19. DPDwarkesh Patel11:39

       Yeah.

   20. EJEric Jang11:40

       Yes, yes. But in, in any case, this is a very, very, very large tree.

   21. DPDwarkesh Patel11:43

       Yeah.

   22. EJEric Jang11:43

       And, um, this is also why, you know, computer scientists for many years thought that Go was not a tractable problem this century-

   23. DPDwarkesh Patel11:50

       Yeah

   24. EJEric Jang11:50

       ... because the amount of compute you would need to exhaustively search every possible possibility, um, is just too large.

   25. DPDwarkesh Patel11:56

       Yeah.

   26. EJEric Jang11:57

       Um, if you could, Go is actually a deterministic game. So, um, on any given state, you can actually compute what the, uh, best possible strategy you can, uh, you can make is-

   27. DPDwarkesh Patel12:06

       Mm

   28. EJEric Jang12:07

       ... uh, to, in order to win the game. You can search all the possible futures where you win and then just make sure you always stay in, in, in that, you know, set of futures.

   29. DPDwarkesh Patel12:13

       Mm-hmm.

   30. EJEric Jang12:15

       So AlphaGo's kind of core conceptual breakthrough was using neural nets to make this search problem tractable. So before we get into, you know, how neural networks are involved, let's talk a little bit about how we can, you know, a- assuming we had a very powerful enough computer, um, search this, uh, this tree to find the best move, right? So in the beginning, um, you're not gonna build out the whole tree, uh, bec-because storing that tree would be very expensive. Instead, you might do something like interactively figure out which, um, which l-leaves of this tree are worthy of exploring and expanding into the future to see, you know, what else is there. So, um, there are some early algorithms in, uh, bandit literature like, you know, UCB, uh, one, which is not exactly appropriate for a, you know, sequential game like Go, but very much inspired the action selection, um, algorithm used in, uh, in AlphaGo. So, so UCB one looks like on, on every move we're gonna take the best action, uh, or, you know, the argmax over A that maximizes, um, um, you know, the Q of A, and I'll explain what Q of A is in a moment, plus some sort of exploration bonus. So on every node, we're going to track a few quantities. So, so let's, you know, consider each of these a node. This is, this is the, the root node, um, where you're making decisions from, and these are the children of the root node. And, um, we're gonna say each node is basically a data structure that is, um-- it stores a visit count of this, um, this node, this, this child node.
3. 31:53 – 1:00:22

   What the neural network does

   1. EJEric Jang31:53

      Okay, so n- now we have a basic intuition of how moves are made-

   2. DPDwarkesh Patel31:56

      Yeah

   3. EJEric Jang31:56

      ... with search. We're gonna talk about how neural networks can speed this up, uh, by providing an analog to, like, the human intuition here.

   4. DPDwarkesh Patel32:02

      Mm.

   5. EJEric Jang32:03

      So there's two networks. There is the value network, which takes in a state, and it predicts, you know, am I gonna win or lose? It's a binary classification problem. Then we're gonna have a policy network, which, uh, induces a distribution over good actions to take.

   6. DPDwarkesh Patel32:24

      Mm-hmm.

   7. EJEric Jang32:25

      So, um, I'm gonna draw a one-dimensional flattened m- move distribution, but this is really like, you know, a, a, a square kind of grid, right? So, um, so maybe, like, it thinks actions are like... Th- these are the kind of probability distribution over good actions.

   8. DPDwarkesh Patel32:41

      Mm.

   9. EJEric Jang32:42

      And both of these are, uh, categorical classification problems, right? So you can train this like any classifier in, uh, you know, with deep learning, um, uh, you know, c- cross entropy, loss, that kind of stuff. So the, um, the specific architecture does not actually matter too much. I, I tried a few different architectures, transformers work, ResNets work.

   10. DPDwarkesh Patel33:00

       Hmm.

   11. EJEric Jang33:01

       For small data regimes, um, my experience is that ResNets still kind of outperform transformers-

   12. DPDwarkesh Patel33:05

       Hmm

   13. EJEric Jang33:05

       ... and, um, and they kind of give you more bang for the buck at, at lower budgets, but this may not be true.

   14. DPDwarkesh Patel33:10

       Oh, really? Why is that?

   15. EJEric Jang33:11

       Um, they, they provide the inductive bias of, like, local convolutions.

   16. DPDwarkesh Patel33:14

       Yeah.

   17. EJEric Jang33:14

       And t- generally, transformers start to outperform, uh, residual convolutional networks when you want more global context.

   18. DPDwarkesh Patel33:21

       I see. Yeah.

   19. EJEric Jang33:21

       So, so one, um, interesting finding from the Katago paper was that they found it actually quite useful to pool together global features together, um, t- at aggregate global features, like, uh, throughout the network, um, to kind of give the network a global sense of how to, like, connect value from one side of the board to another side of the board.

   20. DPDwarkesh Patel33:40

       But what does it mean to aggregate global features?

   21. EJEric Jang33:42

       Yeah. So if you have a, um, Go... A, a, a very large 19 by 19 Go board-

   22. DPDwarkesh Patel33:47

       Yeah

   23. EJEric Jang33:48

       ... and you, you know, you've got some, some sort of battles going on here, and you've got some battles going on here, um, when you pass this through a convolutional neural network-

   24. DPDwarkesh Patel33:57

       Yeah

   25. EJEric Jang33:57

       ... the receptive fields of the convolutional network are going to be good at computing local things and making that sh- invariant, but, um, they won't be able to kind of connect these two features easily, right? They need to sort of be pooled together and attend to each other somehow. So the argument about, you know, why transformers are good for computer vision tasks, like with, uh, you know, vision transformers and so forth, is that because they have a sort of global attention across the whole thing-

   26. DPDwarkesh Patel34:25

       Yeah

   27. EJEric Jang34:25

       ... they can more easily draw these kind of connections.

   28. DPDwarkesh Patel34:27

       Right.

   29. EJEric Jang34:27

       But you do need more data there so that you can kind of, uh, learn through data the, the sort of invariant local, local features.

   30. DPDwarkesh Patel34:35

       Got it. Yeah.
4. 1:00:22 – 1:25:27

   Self-play

   1. EJEric Jang1:00:22

      Okay, so, um, we now talk about the RL part of, like, how this thing gets stronger by playing itself, right? Um, let's say we play a game where we have the AI. So you make a moveAI, AI, um, will, will kind of compute the search and then th-this is the sort of visit count distribution. Um, let, let's say this is your policy, your policy, initial policy recommendation at the, at the, at this node.

   2. DPDwarkesh Patel1:00:51

      Mm-hmm.

   3. EJEric Jang1:00:52

      And then after MCTS, it, uh, gets more confident about one of these actions, right? And, and so maybe the, the distribution looks a bit more peaky like this based on the, the search. Now, of course, you can tune the search process so that it ends up more diffuse, but that's probably not a good idea. MCTS should get more confident about specific actions, um, than others, but it, of course, might place a lot of weight on, you know, other actions initially-

   4. DPDwarkesh Patel1:01:16

      Yeah

   5. EJEric Jang1:01:16

      ... and then as you increase the number of sims, it should converge to a very peaky distribution. Um, so, so this is your new, uh, let's call this, like, pi... Let's, let's wrap this in, like, a MCTS operator of, you know, A given S, right? So after applying MCTS process, your, your policy-recommended distribution looks like this. It's, it's a bit more peaky than, than the previous one. Um, and so then you take the argmax, or maybe you just sample from this, uh, it doesn't have to be the argmax, and then you, you make your move. And then, um, and then you throw away the tree, and then you, you begin anew on the next move, right? So again, like, you, um, you know, you compute a new distribution. So initially, maybe your guess looks like this, and then you refine it through MCTS.

   6. DPDwarkesh Patel1:02:16

      There should be one more X on the board, right?

   7. EJEric Jang1:02:18

      I'm sorry. That's correct, yes. Um, to something that looks like this, right? So, so on every move, you have your initial guess from your policy network, um, and then the search process that combines your policy network and your value network arrives at a more confident action that you take.

   8. DPDwarkesh Patel1:02:37

      Mm-hmm.

   9. EJEric Jang1:02:38

      And, and then so on and so forth, and then the game ends, and one person wins and one person loses. So a, um, the way that, uh, the, the beauty of, of how AlphaGo trains itself is that it actually can take this final search process, the outcome of the search process, and tell the policy network, "Hey," like, you know, "instead of having MCTS do all this, you know, legwork to arrive here, why don't you just predict that from the get-go?" Right? Like, "Why don't you, like, you know, not use this guess and just predict this to begin with?" And if you have this guess to begin with in your policy network, then MCTS has to do a lot less work to, to get things to work. And so if we draw like a sort of test-time scaling plot, um, so, so let's say, like, this is, like, number of simulations. Um, let's say, you know, at, at zero simulations, your, your sort of implicit win rate is like, um, is like, I don't know, here, and then, and then, um, w-without any simul- if you just take this raw action, th-this is what your w-win, win rate is. And let's say as we increase the number of sims, um, maybe, maybe you kind of have a, a win rate that looks like this, right? So, um, when you search for, let's say, a thousand simulation steps, that gets you to a, um, a policy here that gets you to here, which is great. But if you were to distill this MCTS policy network back into your sort of, uh, shoot-from-the-hip policy network, then you could actually, um, uh, you know, start here, like, uh, if, let's say this was, you know, zero, um, for, uh, by, by distillation, then if you spend another one thousand sim steps, then you actually kind of get to here. It's almost like if you could just, you know, am-, um, amortize the, the first one thousand steps actually into the policy network-

   10. DPDwarkesh Patel1:04:26

       Yeah

   11. EJEric Jang1:04:26

       ... instead of the search process, then you could begin at a much better starting point and then get a much better result for, for your, uh, for the number of sims that you put in.

   12. DPDwarkesh Patel1:04:33

       The same more type nature of test-time scaling as the number of simulations increases, the, the increase in win rate is, is, uh, smaller. Is that true even for the distilled network? That is to say, is there some gain of, like, okay, we start from the distilled, we get these early gains again, or is that just inherent to, like, the nature of-

   13. EJEric Jang1:04:50

       Yeah

   14. DPDwarkesh Patel1:04:50

       ... MCTS?

   15. EJEric Jang1:04:51

       To be honest, I actually don't know the test-time scaling behavior of MCTS simulations, and I, I believe it might actually be quite sensitive to how strong this one is in practice.

   16. DPDwarkesh Patel1:05:00

       Yeah.

   17. EJEric Jang1:05:00

       I'm just drawing a monotonically increasing function that gets to one.

   18. DPDwarkesh Patel1:05:02

       Okay, cool.

   19. EJEric Jang1:05:03

       So yeah. So, so don't pay too much attention to the shape of the curve. Just know that it's monotonic with respect to sim. Um, okay, so, um, so, so the idea of MCTS is very brilliant, which is like we're gonna... We got something better by applying search.

   20. DPDwarkesh Patel1:05:17

       Yeah.

   21. EJEric Jang1:05:18

       And, um, we're going to now on our next iteration of updating this network, just train this to approximate the outcome of a thousand steps of search.

   22. DPDwarkesh Patel1:05:27

       Yep.

   23. EJEric Jang1:05:27

       And so instead of starting here, we get to now have our neural network start here, and then, and then, you know, the, the play gets stronger once we then apply another thousand steps on top of it.

   24. DPDwarkesh Patel1:05:35

       Yeah.

   25. EJEric Jang1:05:35

       And you can keep going, right? So, so the training algorithm for AlphaGo is to basically take the games where you've applied the search on every move that the policy encountered, uh, whether you won or lost, and that's quite important. Um, and you're just gonna train, uh, the, the model to imitate the search process. So, um, there's an analogy to robotics, actually, which is the DAGGER algorithm.

   26. DPDwarkesh Patel1:05:57

       Hmm.

   27. EJEric Jang1:05:57

       Um, uh, fir-first I'm gonna draw like a, uh, schematic of like, let's say, you know, the states, right? So S0, S1, S2, S3. So let's say, you know, we, we took a series of actions in an MDP to, to get, uh, a trajectory, and these actions may be suboptimal, right? Maybe we lost at the end of this game. So, um, there is a family of algorithms that basically take trajectories, uh, and relabel the actions to better trajectories. So maybe a better action here would have been to take, you know, A0 prime. A better action here would have been to take A1 prime, and yet another one, like A2 prime.A3 prime. So, um, what MCTS is doing is basically saying, like, you played this game, where you eventually lost, but on every single action, I'm gonna give you a strictly better action that you should take instead. It does not guarantee that you are going to win, but it does guarantee that, you know, if you take these tuples as training data so that you retrain your, uh, your, your policy network to predict these ones instead of these ones, you're gonna do better. And, uh, this is very related to DAgger in, uh, robotics and, and imitation learning, where you want to collect a intervention here, and even if you're in a, in a not great state, for example, like a self-driving car that, you know, veers off the side of the road, there is still a valid action that kind of corrects you and brings you back.

   28. DPDwarkesh Patel1:07:30

       Yeah. Okay, so, um, pedantic question, but is there a guarantee that MCTS must be better than the policy? For example, you could imagine early on in training, because MCTS is informed by the value network-

   29. EJEric Jang1:07:43

       Yeah

   30. DPDwarkesh Patel1:07:43

       ... early on in training, uh, when the value network hasn't been well-trained on finished, uh, games, um, that, like, MCTS is worse than sort of randomly initialized policy. So is it just, like, a heuristic that MCTS is better than the policy, or is that, like, is there some guarantee?
5. 1:25:27 – 1:45:36

   Alternative RL approaches

   1. EJEric Jang1:25:27

      Okay, so, uh, it, it's worth kinda thinking a little, a little bit about, like, okay, what are some alternatives we could do to train self-play agents instead of MCTS, right? Like, you know, we, we use a lot of, uh, LLM style RL these days. Like, is that relevant? Could we do that instead? Um, so, so let, let's think through this a little bit. L-let's suppose we have a very naive algorithm where we take a league of agents of different checkpoints, and we play them against each other, and, um, for, for the games where a single player wins, uh, we're gonna reinforce those actions up, and then, and then, uh, and, and train-- retrain the policy network to imitate those, those guys instead of, uh, instead of the MCTS objective. Um, so what ends up happening is, let's say you have a chain of actions that led to a win, and you're, you're-- you have a match-up between two agents that are basically the same. So in fact, let's just assume that, like, uh, you know, policy A and policy B are, like, evenly matched, right? So their true, their true win rate is, is, like, fifty percent. Um, so let's say you play a hundred games, and then, uh, each game, let's say, lasts, you know, three hundred, um, moves. And, um, you're doing some sort of, like, evolution strategy or some way to perturb these things to get them, get them to do different things, or maybe you don't, and you just play them against each other, and you see, like, occasionally this one might actually have a better strategy than this one, right?

   2. DPDwarkesh Patel1:26:59

      Yeah.

   3. EJEric Jang1:27:00

      And so, so let's say, um, you know, fifty-one games, um, policy A wins. And then forty-nine games, policy B wins. And this is just due to random luck, or maybe you perturbed policy A-

   4. DPDwarkesh Patel1:27:16

      Yeah

   5. EJEric Jang1:27:16

      ... in some way that let it do this. And just to have a very, very simple model, let's pretend that for, uh, for, like, uh, forty-nine of the games, they played exactly equally. Um, I'm sorry, for, for fifty of the games, they, they played exactly equally, right? Um, and on that one game where this one won, it, it played slightly differently. It made, like, one critical move that, like, you know, normally it would have done differently, but, uh, due to some exploration or some random noise, it just happened to make a smarter move than it did previously.

   6. DPDwarkesh Patel1:27:46

      Yeah.

   7. EJEric Jang1:27:46

      So you have one supervision signal, like one true supervision signal for your policy network. Um, and then you have, uh, ninety-nine games times three hundred moves for which, uh, imitating those actions gives you exactly the same policy you had before. And so the, um, the scale of your variance is actually very bad because-

   8. DPDwarkesh Patel1:28:10

      Mm-hmm

   9. EJEric Jang1:28:10

      ... it's like you only have one label out of this enormous data set of, of actions, of supervision actions where you want-- Actually, sorry, let, let me, let me clarify a little bit.

   10. DPDwarkesh Patel1:28:19

       Okay, so we're just talking about how the good move, the out-of-distribution move, is a small fraction of all the moves that are played across all the games on which you'd want to train. And, um, this of course reminds me of how LLMs are trained with policy gradient methods. Uh, Karpathy was-- when he was on the podcast called it, like, sucking supervision through a straw. Um, and uh, and so yeah, it's interesting that this, like, this thing you're saying, which would be intractable and prevents you from actually getting beyond a certain level in Go is just by default how LLMs are trained, question mark?

   11. EJEric Jang1:28:51

       R-right. So, um, in this case, th-this is not to say it doesn't work, right? Like if you imagine increasing the number of games to like, you know, millions of samples, you actually can get some meaningful supervision, like, s-samples, so long as you find a way to sort of mask out the supervision from these guys. And then this is where things start to get pretty related to RL in terms of advantage and baselines and so forth.

   12. DPDwarkesh Patel1:29:12

       Yeah.

   13. EJEric Jang1:29:13

       So, so let's, um, let's look at the eq- you know, the gradient variance of a very naive approach like this where, um, I'm just gonna call it, like, gradient RL, and it's basically the, you know, sum of rewards.

   14. DPDwarkesh Patel1:29:38

       Okay, I see what you're saying.

   15. EJEric Jang1:29:52

       So, so the sum of rewards is the return, right? So, so, like, uh, in, in our naive setup here, we only have a indicator variable for the return, where either you won or lost. So, um, so in the case where you lost, well, you just don't train on-- your gradient is zero. You don't train on those examples. And when you won, you try to predict those, those things, right?

   16. DPDwarkesh Patel1:30:13

       Yeah.

   17. EJEric Jang1:30:13

       So, so you can think about this setup as a, as a special case of-

   18. DPDwarkesh Patel1:30:16

       Yeah

   19. EJEric Jang1:30:16

       ... this general formula here. Um, the, um, the trouble here is that this is very high variance because when you multiply these terms out, when you take-- when you try to compute the variance of this, and so, so variance of the gradient is equal to expectation of squared minus... And just for simplicity, we can pretend this is like, you know, on average zero or something, we c-- if, if you're centering it at, at, you know, no signal. Um, and uh, the variance here basically means that you're, you know, taking the square of this product term, and so you end up with, uh, a term that kind of grows quadratically with the, with T.So, so variance, um, when you have a setup like this, th-this thing acts as a coupling effect on top of, of these terms here. So, um, um, let's actually map this to an LLM case, and we can answer like why do LLMs only do one-step RL instead of a multi-step RL scenario? Um, in LLMs you have a decoder that might, you know, predict some words like, "Hello, world." And so in current LLM RL they treat this entire sequence as a single action, just at, and big T is just one, right? And so yes, it is true that, you know, the, uh, because of how, you know, transformers are formulated, uh, th-through the sort of p-product of conditional probabilities, um, we do have, uh, you know, probability of this sequence is equal to the sort of sum of... Lo-log probability of the whole sequence is equal to the sum of the probabilities of like, you know, uh, individual tokens, right?

   20. DPDwarkesh Patel1:32:06

       Yeah.

   21. EJEric Jang1:32:06

       So, so in this case I would, um, I would say something like, you know, log hell plus log lo plus log world. So, um, this is true, and if this term were one, then they would be the same thing. However, um, in sampling things if you have a reward term assigned to every specific token, now you have these interaction effects between the cross-multiplication of these terms and these terms.

   22. DPDwarkesh Patel1:32:42

       Right.

   23. EJEric Jang1:32:42

       And so the problem becomes how do you ascribe the credit associated with every episode to all these different terms here?

   24. DPDwarkesh Patel1:32:50

       I guess the thing I'm confused on is h- what would that even look like to do it that way in, um-

   25. EJEric Jang1:32:55

       In LLMs

   26. DPDwarkesh Patel1:32:56

       ... in LLMs because you do, you only do get a reward at the end of the episode, so...

   27. EJEric Jang1:33:01

       You could imagine a reward that says like, "I'm gonna give you some process supervision-

   28. DPDwarkesh Patel1:33:06

       Yeah

   29. EJEric Jang1:33:06

       ... uh, where you get a reward for each of these actions on every step."

   30. DPDwarkesh Patel1:33:11

       Okay, so you're saying it, instead of doing it that way where you, um, well I guess the way you've written it would be a sum at the end anyways, so it would, they, they wouldn't have to be multiplied. But you're saying instead of doing it that way you would just add up this process rewards at the end and then treat that as one single reward signal?
6. 1:45:36 – 2:00:58

   Why doesn’t MCTS work for LLMs

   1. DPDwarkesh Patel1:45:36

      Okay, so this is very interesting. And then to unify this with our discussion of LLMs, so with LLMs, you're doing something, you don't have Q values, but you're doing this sort of backwards learning where, hey, let's find the trajectories which pass some unit test in some coding environment, and then let's reinforce those trajectories.

   2. EJEric Jang1:45:54

      Mm-hmm.

   3. DPDwarkesh Patel1:45:54

      And then there's a huge difference between that and this forward approach with MCTS, and the reason you can do MCTS, and it- it's much more preferable to do MCTS because you can do it per move, uh, and make each move better rather than having to learn per trajectory, um, and hope, you know, as Karpathy said, hope to learn this like-

   4. EJEric Jang1:46:11

      Through a straw

   5. DPDwarkesh Patel1:46:12

      ... yeah, so you get this supervision through a straw. Uh, basically just upgrade all the tokens r- in the trajectory that might or might not have been relevant to getting the answer right. The reason you can do this much more sort of sample efficient, uh, much more favorable thing with Go is that because MCTS works in Go, you basically know that, hey, if I just do search locally here, and this search is sort of truncated at the end by this value function that, uh, works even, even if I haven't unfolded my whole trajectory, I can just say, "This is my new policy," um, and I can improve in a more iterative, like, local way rather than having to ha- having to unfold all these trajectories.

   6. EJEric Jang1:46:54

      So there was some research, I think, from Google in 2030, 2023, 2024, where they did try to apply tree structures to reasoning.

   7. DPDwarkesh Patel1:47:02

      Yeah.

   8. EJEric Jang1:47:03

      And I think it's, you know, the jury is still out as to whether this can ever work. So I, I would say like, uh, it-- we probably will see like, you know, revisiting of this idea of forward search, um, in, in the future. But there's two things that make MCTS very simple for Go, which is that value estimation is kind of concrete and you can determine it-

   9. DPDwarkesh Patel1:47:25

      Yeah

   10. EJEric Jang1:47:25

       ... for real, and then you can kind of, uh, uh, sort of use it to truncate depth, as you said.

   11. DPDwarkesh Patel1:47:30

       Yeah.

   12. EJEric Jang1:47:30

       And then the, uh, breadth is also, uh, determined, and what's kind of critical is that the action selection algorithm where you iteratively visit and grow the tree is, um, well suited for the size of problem that Go is-

   13. DPDwarkesh Patel1:47:43

       Yeah

   14. EJEric Jang1:47:43

       ... and the depth of the problem. But for something like LLM reasoning, um, you know, PUCKT might actually not be a good enough heuristic.

   15. DPDwarkesh Patel1:47:49

       Right.

   16. EJEric Jang1:47:49

       It might be too greedy with local tokens, and it might do something like, oh, only give you, you know, uh, sort of obvious thoughts that are correct, but not really solve your final problem.

   17. DPDwarkesh Patel1:47:59

       Yeah.

   18. EJEric Jang1:47:59

       So I, I would say the jury is probably still out on how, like, what the final instantiation of reasoning for LLMs would look like, and I wouldn't rule out that, like, this stuff could, you know, come back-

   19. DPDwarkesh Patel1:48:08

       But d-

   20. EJEric Jang1:48:09

       But it's been hard

   21. DPDwarkesh Patel1:48:09

       ... do- don't LLMs sort of ni- natively learn to do MCTS, where they'll try and approach and be like, "Oh, that doesn't work. Let's back up. Let's try this other thing-"

   22. EJEric Jang1:48:19

       Mm-hmm

   23. DPDwarkesh Patel1:48:19

       ... and then go in the direction that proves to be more fruitful?

   24. EJEric Jang1:48:22

       Uh, yeah, cer- certainly I think the LLMs manage to do something that looks like real human reasoning-

   25. DPDwarkesh Patel1:48:26

       Yeah

   26. EJEric Jang1:48:26

       ... without having to do an explicit tree structure.

   27. DPDwarkesh Patel1:48:28

       Yeah.

   28. EJEric Jang1:48:29

       Um, that being said, I think the idea of doing forward search and simulation to get a better sense of what is valuable might make a comeback, um, even though not exactly in the same instantiation as, as-

   29. DPDwarkesh Patel1:48:42

       Interesting

   30. EJEric Jang1:48:42

       ... uh, AlphaGo.
7. 2:00:58 – 2:11:51

   Off-policy training

   1. EJEric Jang2:00:58

      seeing, like, what really matters.

   2. DPDwarkesh Patel2:00:58

      Wait, this is a sort of tangential question, but why is it okay to have a buffer in AlphaGo? 'Cause every time I talk to an AI researcher, they're telling me about how bad it is to be off policy.

   3. EJEric Jang2:01:06

      Mm-hmm.

   4. DPDwarkesh Patel2:01:07

      But then the way a naive implementation of AlphaGo Zero would work is that most of the moves in a given backward step or in a batch of backward steps, um, would be n- not, not among the ones that were made by the most recently trained model. So why is that okay?

   5. EJEric Jang2:01:26

      Great question, yeah. And this, this gets into the sort of fundamental off-policy versus on-policy, uh, reinforcement learning kind of questions. So, uh, as you recall, in MCTS, you take actions that you took and you relabel them to, uh, to take different actions on the same states.

   6. DPDwarkesh Patel2:01:43

      Mm.

   7. EJEric Jang2:01:43

      Right? So, so the off-policy part here comes where, um, what if you're relabeling states that your new policy would never visit?

   8. DPDwarkesh Patel2:01:50

      Yeah.

   9. EJEric Jang2:01:50

      Right, like, what's the point? You're kinda wasting capacity, and in the extreme limit, imagine your distribution of states in your training buffer are all states that you would never visit. Then you're basically supervising them, uh, to, uh, take good actions on states you would never achieve-

   10. DPDwarkesh Patel2:02:04

       Yeah

   11. EJEric Jang2:02:04

       ... and therefore your policy can get really bad, right? So this is where off-policy can really hurt, um, uh, uh, AlphaGo. Um, however, if you interpret this sort of from, like, the dagger perspective, which is basically saying, like, a way to kind of correct yourself back to the optimal trajectory, uh, given some, some data, what, what you kind of want in a algorithm like this is to have mostly states that you would visit, but then you have a small percentage or maybe a reasonable percentage of states in this kind of high-dimensional tube around your optimal-

   12. DPDwarkesh Patel2:02:34

       Mm

   13. EJEric Jang2:02:34

       ... you know, tr- trajectories, and any of those states are given a supervision target to kind of f- uh, sort of funnel you back into your optimal trajectory. Uh, so maybe I can just draw-

   14. DPDwarkesh Patel2:02:44

       Yeah, yeah

   15. EJEric Jang2:02:44

       ... it out quickly here. Great. So in, um, sort of a dagger-style setup, what your kind of optimal training data distribution is, is that here is your optimal states and actions, so this is like, you know, you wanna be in this state, you wanna be in this state, you wanna be in this state, and then you win here. Um, and then these are your optimal policy actions. So th- these are the, these are the things that you definitely wanna train on, but to make it robust to disturbances, um, you want to make sure that if you happen to drift off into some other states-

   16. DPDwarkesh Patel2:03:14

       Mm

   17. EJEric Jang2:03:15

       ... you can kind of funnel yourself back into-

   18. DPDwarkesh Patel2:03:17

       But why isn't this a fully general argument for off-policy training?

   19. EJEric Jang2:03:20

       This is actually why you wanna do off-policy training sometimes-

   20. DPDwarkesh Patel2:03:23

       Mm

   21. EJEric Jang2:03:23

       ... is that, like, you, you don't want to have a compounding error where if you make a mistake, you don't have the data of how to return back to your optimal distribution.

   22. DPDwarkesh Patel2:03:30

       Yeah.

   23. EJEric Jang2:03:31

       And so, um, optimal control does not really say too much about, like, uh, y- you know, how to, uh, you know, not accidentally get here, because it's sort of making the assumption that, like-

   24. DPDwarkesh Patel2:03:42

       Right

   25. EJEric Jang2:03:42

       ... once you learn the policy, you're gonna get here.

   26. DPDwarkesh Patel2:03:43

       Yeah.

   27. EJEric Jang2:03:44

       But- but in applications like robotics, right, like, like, I don't know, a, a gust of wind blows you slightly off, and then now you need to, like, correct, right?

   28. DPDwarkesh Patel2:03:50

       Yeah.

   29. EJEric Jang2:03:51

       Um, or the friction on one of your tires is kind of a little bit, like, lower than the other wheel, and then now, now your, your car is drifting, and you gotta, gotta, like, correct it. So, so these kind of things in, in, like, more real environments often happen, where, like, um... A- actually, there was a funny, uh, quote about chess and, and also Go. It's like, "The problem with, uh, the, the problem with Go and chess is that the other player's always trying to do some shit," right?

   30. DPDwarkesh Patel2:04:11

       [laughs]
8. 2:11:51 – 2:22:05

   RL is even more information inefficient than you thought

   1. DPDwarkesh Patel2:11:51

      I, I'm reminded now of our earlier conversation of why MCTS is so favorable as compared to the kind of r- you know, reinforce or policy gradient kind of thing LLMs do. And this might be totally wrong, but I wrote a blog post a few months ago about, um, how RL, at least policy gradient RL, is even more, uh, inefficient than you might think. And so the inefficiency one thinks about naively is the fact that you have to roll out a whole trajectory in order to get any learning signal at all, a- at all. And so as these trajectories become longer and longer, as an agent has to, instead of just previously like complete the next word in the sentence, it has to go instead to, "Hey, sp- do two days worth of work to figure out even if you even did this project correctly." The amount of information per, uh, FLOP has been decreasing. As you had to unroll two days worth of thinking in order to see if you even did something correctly to like, did I, uh, implement this feature, the amount of samples per FLOP has been decreasing. But so you can think of, um, uh, you're trying to maximize as you're learning bits per FLOP, right?

   2. EJEric Jang2:12:53

      Mm-hmm.

   3. DPDwarkesh Patel2:12:54

      Um, and this is, you can think of bits per FLOP as, um, samples per FLOP times, um, uh, bits per sample. And what I just mentioned, uh, a second ago is that the samples per FLOP go down as RL becomes more and more long horizon.

   4. EJEric Jang2:13:21

      Mm-hmm.

   5. DPDwarkesh Patel2:13:21

      But, um, at least this kind of naive RL is also terrible from a bits per sample perspective, and here's what I mean, at least compared to supervised learning. So early on in training, let's say you have a, a vocabulary size for an LLM that is 100K long, so th- there's 100K possible, you know, tokens that one could answer. And you have a totally untrained model, and you have a prompt like, "The sky-"Is, um, with supervised learning, you, what, what, what would happen is that the model would have some probability distribution over all the things it could say. Um, there's a label that says actually the term here is blue, and it would figure-- it would learn basically through cross-entropy loss exactly how far its distribution is from correctly saying blue. Now, if you were doing this through RL, um, you would say the model would try, "The sky is halicon." Nope, that's wrong. "The sky is told." Nope, that's wrong. This is a totally untrained model, right? And so you would have to do this on the order of a hundred thousand times in order to just stumble on blue, then get some learning signal off of that. So if you're in the supervised learning regime, and you just get-- you have your distribution of probabilities, you get told, uh, that it's blue, and you figure out how far off you were, the amount you learn is, um, is a function of your pass rate. So, like, the further away you are from blue, the more you've learned to go towards blue, uh, u-using cross-entropy loss. And so you can think of it as, like, your pass rate, your, like, prior probability of having said blue. And, um, as a function of that, like, in supervised learning, uh, through cross-entropy loss, you would, you would learn negative log P, P being pass rate-

   6. EJEric Jang2:15:04

      Mm-hmm

   7. DPDwarkesh Patel2:15:04

      ... uh, bits once you get this label. Whereas in RL, if you're just randomly guessing shit and seeing if it works or not, that's, um, that's just basically going to be the entropy of a binary random variable, which is-

   8. EJEric Jang2:15:25

      And what's also tough here is that actually the distribution that you're sampling under is your policy's distribution.

   9. DPDwarkesh Patel2:15:31

      Mm.

   10. EJEric Jang2:15:32

       Right. So, so it's like if your policy has no chance of sampling blue, then you will never get a signal.

   11. DPDwarkesh Patel2:15:37

       Exactly. Right, right. So that's, that's being modeled by the fact that your probability of sampling blue is extremely low. If you do sample it, you do learn as much as you would have learned under supervised learning. In all other cases, like, you know, ninety-nine point nine nine nine percent of in an untrained model, you're, um, you're just learning incredibly little from, like, seeing halicon is not the correct word or told is not the correct word.

   12. EJEric Jang2:15:59

       Yep.

   13. DPDwarkesh Patel2:15:59

       Um, and that's what happens most of the time. So you're just like, um, learn very little. So if you try to graph, um, if you put on the X-axis your pass rate, um, and, uh, here you put the, like, sort of the bits you, bits you're learning from a sample.

   14. EJEric Jang2:16:18

       Mm-hmm.

   15. DPDwarkesh Patel2:16:19

       If you have, like, zero percent here, fifty percent here, and a hundred percent here, so the end of training you're here. Um, if you have, um, supervised learning, negative log pass rate would look something like this. And then the, uh, entropy binary random variable would look like this. Um, and this is, uh, depending on whether you're doing natts or bits-

   16. EJEric Jang2:16:48

       Mm-hmm

   17. DPDwarkesh Patel2:16:49

       ... uh, yeah, if you do bits, it's like one, right-

   18. EJEric Jang2:16:50

       Mm-hmm

   19. DPDwarkesh Patel2:16:50

       ... here at the, at the peak. Um, this is like a coin flip. You learn the most from a coin flip.

   20. EJEric Jang2:16:55

       Mm-hmm.

   21. DPDwarkesh Patel2:16:56

       Uh, this is supervised learning. This is RL. However, the problem is you spend most of training in this regime, right? Like in the, in the low pass rate regime. And, um, in fact, of how fast you're learning is a function of how many bits per sample you're getting, uh, and you're getting very little signal here. If you chart the pass rate on a log scale, so you put the X-axis on a log scale where, like, at the beginning of training with a vocab size of one hundred K, the pass rate is one hund- one over one hundred thousand, then one over ten thousand, one over one thousand, uh, one over one hundred, and then, um... Okay, what this graph looks like here, where supervised learning would look like this, and, and then RL, if you just basically cr-crunch what I just sh-uh, showed there, it would look like that.

   22. EJEric Jang2:17:58

       Yeah. And arguably, you spend all your time here-

   23. DPDwarkesh Patel2:18:02

       Exactly

   24. EJEric Jang2:18:03

       ... potentially e-never even getting a single success, right?

   25. DPDwarkesh Patel2:18:05

       Exactly.

   26. EJEric Jang2:18:05

       Like, uh, so, so it's, it's a sort of depressing plot in the sense that, like, once you're here, it's not at all obvious how you get to here.

   27. DPDwarkesh Patel2:18:12

       Yeah.

   28. EJEric Jang2:18:12

       Um, you know, once you're here, you have something-

   29. DPDwarkesh Patel2:18:14

       Yeah

   30. EJEric Jang2:18:14

       ... but, like, you actually, in many RL problems, spend all the time here.
9. 2:22:05 – 2:30:11

   Automated AI researchers

   1. EJEric Jang2:22:05

      Sounds good.

   2. DPDwarkesh Patel2:22:06

      One thing I really wanted to talk to you about is that you did a bunch of the research for this project through this kind of automated, uh, LLM coding assistant loop, and, um, there's an idea that if you fully automated AI research, you could have some sort of singularity. Uh, obviously we're not there yet, but to the extent that we have early indications of what this process might look like, I am curious what your observations about, um, what the AI is good at, what it's not good at, what you think about this scenario's likelihood eventually, what thoughts you have about this in general.

   3. EJEric Jang2:22:42

      For sure, yeah. I think automated scientific research is one of the most exciting, um, uh, skills that, you know, that frontier labs are developing right now, and I think it's important for everyone who's doing any kind of research to get a good intuition of, like, what it can do now and what it can't, and how might the sort of science process work in the future once we're having AIs automating a lot of this, this investigation. Um, so, uh, in brief, I mostly use Opus four point six and four point seven throughout the, uh, working on this, and, um, what works is that the models can do a very good job of doing hyperparameter optimization.

   4. DPDwarkesh Patel2:23:16

      Mm.

   5. EJEric Jang2:23:16

      So in the past, people would kind of come up with a search base of hyperparameters like learning rate and, you know, weight decay and maybe how many layers are in your network, and, um, they would just kind of do a grid search or a sort of Bayesian hyperparameter optimization, uh, approach, and then it would find some tuned parameters. Um, the kind of really cool thing that automated, uh, you know, uh, coding can do now is that it can search a much more open-ended set of problems, right? It can say, like, "Well, um, I've identified that, like, the gradients are kind of small on this layer, so let me change it up here. Let me rewrite the code so the data later- data loader has a new augmentation I came up with. Let's, uh, let's, uh, sort of try to find the, the best way to kind of fit the constraints of the optimization problem." And, and you end up with this much more flexible and kind of high level, almost like grad student-like, uh, ability to just, you know, grind a performance metric.

   6. DPDwarkesh Patel2:24:03

      Mm.

   7. EJEric Jang2:24:03

      And, and so this can squeeze out quite a lot of performance. You can, you know, f- on a fixed dataset with a fixed time budget, um, improve perplexity by quite a lot on, on a sort of classification problem like LLMs or, um, or Go. Um, and, uh, it is also fantastic now at basically executing any experiment, right? So I have a, a Claude skill that I wrote called Experiment where, um, I give it a description of what I want it to plot, and like I just describe, "Here's the X-axis I want, here's the Y-axis. Answer this question for me." And it'll go run off and do all the experiments, compile the plot, make a report, and suggest like, you know, what might have caused it or, or so forth. Um, so, so that's what works quite well today, and I think we can expect that these abilities get better in the future. But it's also kind of useful to know, you know, what, what is it not doing so well today. Um, so on my, uh, blog version of this tutorial, I have a, a plot of basically all the kind of experiments I did grouped in a sort of tree where, um, you know, every node kind of represents a failed, successful, or sort of mixed experimental result, and then from there it branches off into a child where it's like the follow-on experiment. Um, occasionally I'll kind of rabbit hole down a track like this off-policy MCTS relabeling, do a few experiments, and then realize it's probably not worth it, so then I'll kind of jump to a completely different track, right? And I call these kind of things like rows, right? So, so what, what I find is that current, uh, you know, closed models that we can access, the public can access today, um-They don't seem to be that great at selecting what the next experiment should be in a given track, and they don't seem to be able to kind of step back and do the lateral thinking of like, "Wait a minute, this track doesn't really make sense. Like, let's go back to sort of first principles and, and think about, you know, what the bottleneck might be, or like what are we trying to achieve?" Right? And, and so often I had to catch infra bugs myself by prompting the right question to Claude to like investigate, you know, why, why, what, what is causing this discrepancy, and then it'll answer the question. Um, I think with like, you know, Mythos class models or Mythos plus plus models coming online, um, maybe this just completely changes, and these, these problems just fall to, to just improved scaling. Um, but at the same time, I think there's a lot of like rich opportunity to, um, develop RL environments that might incentivize this kind of lateral thinking. And, and so one of the motivations for setting up this Go environment was that I think that, you know, Go captures a lot of very interesting research problems, often overlapping with, you know, LLMs or robotics. Uh, and yet it's like very quick to verify. Um, the outer loop is ultimately like, does the agent do what I think it does? And, and you can just kind of check the outcome of a Go game quite easily. Um, and then the inner loop involves all this kind of like, you know, research engineering around distributed systems, uh, predicting whether an idea is gonna work or not, um, predicting the, you know, the difference a particular modification to your training algorithm might make. Um, and I think there's a rich library of subtasks and sub-environments that you can kind of train an automated scientist to work on, uh, with Go as a sort of outer verification loop that then once you acquire these skills, maybe you can apply them to like other domains like, uh, you know, um, biosciences or robotics.

   8. DPDwarkesh Patel2:26:55

      Or automating AI research.

   9. EJEric Jang2:26:57

      Or automating AI research.

   10. DPDwarkesh Patel2:26:57

       Which is, which is the real crux or the, um, scary/uh, incredible thing of just making AIs making future versions of AIs. And you're suggesting the outer loop here could just be your win rate against KataGo, basically?

   11. EJEric Jang2:27:10

       Um, that's one of them. Um, I think there's a lot of deeper questions that one could tackle, right? So, for example, um, let's say you have an idea on, um, how to improve a scaling law compute multiplier.

   12. DPDwarkesh Patel2:27:22

       Yeah.

   13. EJEric Jang2:27:22

       Um, the outcome isn't necessarily like I, I, uh, achieve the best Go bot ever. The outcome might just be like, can I predict what the win rate of my Go bot will be?

   14. DPDwarkesh Patel2:27:32

       Yeah, yeah.

   15. EJEric Jang2:27:33

       Uh, or can I predict the scaling law plots that emerge from my idea? But then you can verify that you haven't kind of reward hacked anything by using a very verifiable game like Go on the outer loop.

   16. DPDwarkesh Patel2:27:41

       I, I, I think there's a couple of interesting follow-on questions. There's questions on the inner loop and the outer loop. On the inner loop, there's a question of how locally verifiable any modification you might make is. That is to say, could-- Would you know whether something is actually improvement or a degradation, some idea you try out? Would you know that if something isn't working as a result of, um, a bug, or is it the result of the idea itself being wrong? Um, Ilya was talking about why having-- One of the reasons he, he thinks he's a good researcher is-- He is a good researcher. One of the things he thinks makes him a good researcher is that, um, he has intuition about-- He has strong belief in what the correct idea is, and he is able to persevere through bugs and know which things are bugs versus mistakes in the fundamental idea based on his high-level belief about this idea should work, so therefore it's, this, there has to be bug versus the other way around. Why don't we start with that question, actually? Yeah. How locally verifiable are things which are good ideas?

   17. EJEric Jang2:28:44

       Yeah. I think as in the case of the success story for deep learning, you can think about this as like a decades-long idea that took, like, took a lot of faith-

   18. DPDwarkesh Patel2:28:52

       Mm-hmm

   19. EJEric Jang2:28:52

       ... to get it to work.

   20. DPDwarkesh Patel2:28:53

       Mm-hmm.

   21. EJEric Jang2:28:54

       Um, and so this presents a very challenging long-horizon, you know, RL problem where, you know, every, every step of the way you have like a committee telling you that this is a bad idea, and then ultimately you break it through, right?

   22. DPDwarkesh Patel2:29:05

       Yeah.

   23. EJEric Jang2:29:05

       And so, like, how do you design RL environments that maybe give you some feedback, uh, uh, earlier? Um, and, and I think this is a very tough open question that I don't have an answer to. But, um, but, you know, ultimately, to play a very strong Go bot, you probably did need to discover deep learning.

   24. DPDwarkesh Patel2:29:22

       Yeah.

   25. EJEric Jang2:29:22

       Right? And so, um, um, I think that, like, having a challenging game that cannot be, you know, cheated easily on the outer loop could be used as a sort of outer loop signal for something like discovering the principles of deep learning. Now, of course, like, to make it tractable, and this is where research taste really matters, um, like, you have to come up with ways to initialize your problem so that you don't solve a sort of very intractable problem, right? Like, like, maybe you can leverage LLMs as, as a sort of a universal grammar in the middle to kind of g-give, give you some sort of local feedback.

   26. DPDwarkesh Patel2:29:55

       Yeah.

   27. EJEric Jang2:29:55

       Um, um, the, the, the fact that LLMs are universal grammar means that they can kind of move at almost any level of the stack, right? They can think very locally as well as step back and think, like, in very broad steps. And I think that's where a lot of, uh, um, the, the lateral thinking ability of humans kinda come from.

Episode duration: 2:37:17