Dwarkesh PodcastWhat rebuilding AlphaGo teaches us about self-play, RL, and future of LLMs - Eric Jang
CHAPTERS
- 0:00 – 2:14
Why rebuild AlphaGo: compressing intractable search into a small network
Dwarkesh introduces Eric Jang and the sabbatical project of rebuilding (and hacking on) AlphaGo. Eric explains why Go/AlphaGo is still intellectually important: it demonstrates how a relatively small neural network can effectively amortize a massive search process, and how open-source work like KataGo made the project accessible to individuals.
- •AlphaGo’s original mystery: how shallow networks approximate very deep game-tree reasoning
- •KataGo’s ~40× compute reduction and why modern hobbyists can reproduce strong Go bots
- •Compute history: from multi-million-dollar efforts to a few-thousand-dollar projects
- •Motivation: understanding the phenomenon, not just replicating a milestone
- 2:14 – 5:08
Go rules that matter for AI: captures, threats, and how games actually end
Eric gives a practical walkthrough of Go: placing stones, capturing by liberties, and why local tactics resemble ‘check’-like forcing moves. They also cover when games terminate (resign or consecutive passes), setting up why ‘end conditions’ are subtle for both humans and computers.
- •Captures are determined by orthogonal liberties (not diagonals)
- •Forcing moves: immediate capture threats and tactical sequences
- •Strategic tradeoffs: sacrificing stones to win elsewhere (micro vs macro)
- •Game termination: resigning or two consecutive passes
- 5:08 – 8:06
Scoring ambiguity vs Tromp-Taylor: why computer Go needs unambiguous endgames
They compare human conventions for ending/scoring games with Tromp-Taylor rules used by AIs. Eric highlights edge cases where humans ‘know’ a group is dead, but a ruleset like Tromp-Taylor scores purely by algorithmic connectivity—creating situations that are important for training value functions and avoiding implementation bugs.
- •Different scoring systems (Chinese/Japanese vs Tromp-Taylor) and why AIs standardize
- •Human consensus-based resolution vs algorithmic resolution
- •Counting stones + empty intersections not connected to opponent stones
- •Why these scoring details create tricky training/engineering edge cases
- 8:06 – 12:15
The core difficulty: enormous game trees, branching factor, and transpositions
Eric reframes Go as a search problem: enormous branching factors and long horizons make naive search intractable. They discuss transpositions (different move orders reaching the same state), and why Go was historically viewed as out of reach for brute-force methods.
- •Rough scale: ~361 legal moves early; game length ~250–300 moves
- •Combinatorial explosion: why exhaustive search is impossible
- •Transpositions: different action sequences can reach identical board states
- •AlphaGo’s key idea: make search tractable with learned guidance
- 12:15 – 21:38
Monte Carlo Tree Search mechanics: nodes-as-states, Q values, and exploration bonuses
Eric builds MCTS from first principles: the tree data structure, visit counts, Q estimates, and how UCB-style exploration balances trying new moves vs exploiting known-good ones. They clarify where ‘probabilities’ come from in a deterministic game: randomness is induced by the sampling/search process, not the environment.
- •Tree node stores: visit count, mean value (Q), prior probability (P), children map
- •Nodes represent states; actions are implied by choosing a child (deterministic dynamics)
- •UCB intuition: exploration bonus shrinks as an action is visited more
- •Probabilities arise from Monte Carlo sampling distributions, even in deterministic Go
- 21:38 – 31:53
From terminal outcomes to backups: how MCTS turns win/loss into value estimates
They explain how leaf evaluations (win/loss at the end) propagate upward through backups to update Q values along the selected path. Eric emphasizes why pure MCTS without neural guidance remains expensive: most actions are low value, so efficient sampling is crucial.
- •Terminal utility provides ground-truth return only at the end of the game
- •Backup step: running averages propagate values up the tree
- •Why sampling is hard: only a few paths matter; most are ‘dead ends’
- •Breadth vs depth: both must be reduced to make Go tractable
- 31:53 – 42:56
What the neural network does: policy + value heads, architectures, and inductive bias
Eric introduces AlphaGo’s two learned components: a value network (win probability) and a policy network (move distribution). They discuss practical architecture tradeoffs (ResNets vs Transformers), why local convolutions help in low-data regimes, and how KataGo adds global feature pooling to connect distant board interactions.
- •Value head: binary win/loss prediction; Policy head: categorical move distribution
- •ResNet vs Transformer: local convolutional bias vs global attention tradeoffs
- •Global pooling features help link distant fights across the board
- •Practical note: architecture matters less than getting the training setup correct
- 42:56 – 1:00:22
Neural MCTS end-to-end: selection, expansion, evaluation, backup (and why rollouts disappeared)
They walk through the four-step neural MCTS loop and how priors from the policy network shape exploration (PUCT). Eric explains AlphaGo Lee’s original hybrid of value net + full rollout-to-end (policy self-play rollout), and why later systems dropped rollouts for speed and simplicity.
- •Four steps per simulation: selection → expansion → evaluation → backup
- •PUCT uses policy priors (P) to focus search on promising actions early
- •Value estimates are flipped across players in a zero-sum game
- •AlphaGo Lee averaged value-net estimate with rollout result; later work removed rollouts
- 1:00:22 – 1:25:07
Self-play as policy improvement: distilling MCTS into the network (DAGGER analogy)
Eric explains the key reinforcement learning loop: MCTS produces a stronger policy distribution than the raw network, and training distills that improved distribution back into the policy head. This creates an iterative bootstrapping process where the network gradually internalizes what search used to do, reducing required test-time simulations.
- •MCTS transforms an initial policy into a sharper, improved distribution (π)
- •Training target is the full MCTS visit-count distribution (soft labels), not just argmax
- •Iterative ‘amortization’: push search compute into the forward pass over time
- •Connection to imitation learning: DAGGER-style relabeling of better actions per state
- 1:25:07 – 1:46:54
Alternative RL approaches: policy-gradient variance, best-response training, and NFSP
They compare AlphaGo’s low-variance per-move supervision to ‘win-trajectory reinforcement’ approaches that suffer severe credit assignment and variance issues. Eric describes neural fictitious self-play (NFSP): train best responses against fixed opponents with model-free RL (e.g., PPO), then distill across a league to approximate a robust mixed strategy.
- •Naive ‘reinforce winners’ provides sparse, high-variance learning signals
- •Advantage estimation and baselines try to isolate which actions truly helped
- •NFSP approach: train best response vs fixed opponent, then distill across opponents
- •Why this matters in domains where tree search is hard (e.g., StarCraft, Dota)
- 1:46:54 – 1:51:03
Why MCTS doesn’t map cleanly to LLMs: branching explosion and non-revisiting children
They discuss why Go-style PUCT/MCTS is difficult for language: the effective action space is enormous and search rarely revisits the exact same ‘child’ completion, breaking key assumptions behind visit-count-driven exploration. Eric notes the jury is still out on forward-search-style reasoning for LLMs, but the direct transplant of Go MCTS heuristics is likely wrong.
- •Language ‘actions’ (token sequences) create a vast, open-ended branching factor
- •PUCT relies on revisiting actions; LLM reasoning rarely samples identical branches
- •Evaluation/truncation is harder: no clean value function like endgame scoring
- •Forward search may return in other forms, but not as a simple AlphaGo clone
- 1:51:03 – 2:00:58
Compute and scaling in Go: test-time vs training-time tradeoffs and modern efficiency gains
Dwarkesh brings up scaling laws in board games (Andy Jones) and the analogy to inference-time scaling in LLMs. Eric describes lessons from his rebuild: scaling-law studies require a working, bug-free system; many ‘tricks’ matter less with better hardware and stronger initializations; and modern approaches can drastically reduce compute compared to AlphaGo-era runs.
- •Test-time compute (sims) can trade off with training compute for similar strength
- •Scaling analysis is meaningless if data/pipeline are broken or mislabeled
- •Hardware + better initialization reduce need for complex infrastructure
- •Practical tricks: pretraining on smaller boards (e.g., 9×9) to warm-start value learning
- 2:00:58 – 2:11:51
Off-policy training and replay buffers: when they help, when they waste capacity
They dig into why AlphaGo-style systems can tolerate some off-policy data and how this resembles DAGGER-style ‘tube around the optimal trajectory’ training. Eric explains an experiment: relabeling random historical states with fresh MCTS can work and can saturate GPUs, but fails if the buffer drifts too far from states the current policy will actually visit.
- •Core off-policy risk: training heavily on states the current policy never reaches
- •Why some off-policy is good: robustness and recovery from drift (DAGGER intuition)
- •Offline relabeling: re-run MCTS on stored states to refresh targets
- •Stability vs efficiency: modern RL often prefers more on-policy updates for stability
- 2:11:51 – 2:22:05
RL is more information-inefficient than it seems: bits-per-sample, pass rates, and soft targets
Dwarkesh presents a ‘bits per FLOP’ and ‘bits per sample’ perspective: policy-gradient RL learns little when success probabilities are tiny (common early in training), making exploration brutally inefficient. Eric connects this to distillation: soft-label distributions carry far more information than one-hot outcomes, and AlphaGo benefits by training on the full MCTS distribution rather than only the chosen move.
- •Long-horizon RL reduces samples-per-FLOP because rewards arrive only after long rollouts
- •When pass rates are low, RL’s learning signal is weak for most samples
- •Soft targets (logits/distributions) contain much higher information than one-hot labels
- •AlphaGo trains policy on the MCTS distribution (dark knowledge), improving sample efficiency
- 2:22:05 – 2:37:17
Automated AI researchers: what LLM agents do well, where they fail, and why Go is a good sandbox
Eric shares practical observations from using LLM coding assistants to build and experiment on AlphaGo: strong at hyperparameter/implementation iteration and executing experiments, weak at choosing the right next experiment and doing ‘lateral thinking’ across research branches. They propose Go as a rich, verifiable outer loop for training automated research agents, potentially transferring skills to harder domains.
- •LLMs excel at: hyperparameter search, code refactors, running/plotting experiments
- •LLMs struggle with: experiment selection, stepping back, diagnosing ‘track is wrong’
- •Go provides fast, verifiable outer-loop feedback (e.g., win rate, scaling predictions)
- •Vision: train automated scientists on structured sandboxes, then transfer to real science/AI R&D
