Dwarkesh PodcastIlya Sutskever on Dwarkesh Patel: Why RL Overfits the Evals
Why RL targeting benchmark evals creates models that ace GPT-3 tests but cycle bugs: Sutskever links this to skipping value functions in the training mix.
CHAPTERS
- 0:00 – 2:48
Why models feel smart on evals but fail in real workflows (and the “jaggedness” problem)
Ilya and Dwarkesh open by contrasting impressive benchmark performance with underwhelming economic impact and day-to-day reliability. Ilya illustrates "model jaggedness" with a live-coding bug-fix loop where the model oscillates between two errors, motivating deeper questions about generalization and training incentives.
- •AI investment feels abstract, but diffusion through the economy will eventually be significant
- •Confusing gap: strong evals vs weak real-world/economic impact
- •Example of jagged behavior: fixing one bug introduces another, then reverts
- •Two hypotheses: RL can over-focus models, or RL environments get overfit to evals
- •Framing the core issue as a mismatch between eval optimization and robust performance
- 2:48 – 6:07
RL training incentives: when humans “reward hack” themselves by chasing evals
The conversation turns to how RL training differs from pretraining: unlike “train on everything,” RL requires selecting environments, rewards, and tasks. Ilya argues that companies may unintentionally let public evals shape RL environment design, producing systems that look good on tests while failing to transfer.
- •Pretraining’s data choice is simple: use everything; RL requires many design decisions
- •Teams create many bespoke RL environments—huge degrees of freedom
- •Evals can become a gravitational pull for what gets trained
- •Reward hacking may be upstream: researchers optimizing for eval appearance
- •Disconnect explained by (1) eval-inspired training + (2) inadequate generalization
- 6:07 – 8:31
A human analogy for overtraining: the competitive programming “10,000 hours” trap
Ilya offers an analogy: a student who grinds competitive programming for 10,000 hours may dominate contests yet perform worse in a broader career than a student who does well with far less practice. He suggests models resemble the first student—hyper-trained on narrow distributions—highlighting why transfer and taste remain hard.
- •Two-student story: intense narrow practice vs modest practice plus broader capability
- •Models are trained like the 10,000-hour specialist, often with data augmentation
- •Specialization makes performance intuitive but harms generalization
- •The “it factor” as an implicit ingredient that’s hard to replicate
- •Raises the question of what pretraining really provides beyond sheer coverage
- 8:31 – 10:59
What pretraining is (and isn’t): scale, natural data, and why it’s hard to reason about
They unpack why pretraining worked so well: enormous quantity, natural human-produced text, and minimal need for curation. Ilya notes it’s difficult to interpret model failures because it’s unclear how any capability depends on support in the pretraining distribution, and he doubts there’s a clean human analog.
- •Pretraining’s strengths: huge scale and not needing to choose among datasets
- •Naturalistic data: “the whole world projected by people onto text”
- •Hard to diagnose failures: was something simply under-supported by data?
- •Human analogs (childhood, evolution) have similarities but also big differences
- •Humans know less breadth than models but often with more depth and fewer silly errors
- 10:59 – 18:49
Emotions as a value function: why decision-making needs intermediate feedback
Ilya connects a neuroscience case—loss of emotion causing catastrophic indecision—to the role of value functions in learning and agency. He argues emotions may be a robust, evolution-shaped modulator of human value functions, and that future ML systems will likely rely more on explicit value-function-like mechanisms.
- •Case study: intact cognition but no emotion leads to inability to decide
- •Emotions as a modulator of a human value function/reward shaping
- •Definition of value function: provide intermediate signal without waiting for final outcome
- •Value functions can short-circuit long-horizon credit assignment
- •Expectation: value functions will become more central in future AI training
- 18:49 – 24:35
From the age of scaling to the age of research: what are we scaling now?
Ilya explains how ‘scaling’ became a dominant organizing concept: pretraining offered a reliable recipe where more compute/data predictably improved loss. With finite data and diminishing returns, he argues we’re returning to an “age of research” where progress depends more on new ideas than brute-force 100× scaling.
- •Scaling as a powerful meme that told the field what to do
- •Pretraining as the canonical scaling recipe; low-risk investment path for companies
- •Data is finite; pretraining eventually runs out of runway
- •Shift already underway: scaling RL/post-training compute and long rollouts
- •Claim: 100× more compute won’t transform everything; we’re back to research
- 24:35 – 36:38
The fundamental bottleneck: models generalize dramatically worse than humans
Ilya identifies poor generalization and robustness as the central obstacle, more fundamental than efficiency tricks like value functions. They explore why humans learn with fewer samples and less bespoke reward engineering, and whether evolution supplies priors versus deeper learning principles.
- •Value functions can speed learning, but aren’t the deepest limitation
- •Core claim: model generalization is far below human generalization
- •Two facets: sample efficiency and learning ‘the right thing’ without verifiable rewards
- •Evolution as a source of strong priors for vision/locomotion, less so for math/coding
- •Ilya hints at undisclosed ideas/principles and a possible compute-underestimate in neurons
- 36:38 – 40:22
Sponsor interlude + ‘research era’ logistics: are ideas or compute the bottleneck now?
After an ad break, they discuss what research looks like when compute is abundant: scaling homogenized approaches and created “more companies than ideas.” Ilya argues that many breakthroughs historically required modest compute to validate, and that frontier-scale compute matters most for best-in-class systems, not for proving concepts.
- •Scaling ‘sucked the air out of the room,’ pushing everyone toward the same paradigm
- •Bottlenecks shift over time: in the ’90s compute limited idea validation
- •Historical examples: AlexNet (2 GPUs), Transformer-era papers (8–64 GPUs)
- •Today: compute is large enough that ideas may be the main limiting factor
- •To build the very best system, frontier compute still differentiates within a paradigm
- 40:22 – 43:00
Why SSI can compete on research compute (and why most big funding is for inference/products)
Dwarkesh challenges whether SSI’s funding can match rivals’ annual research spend. Ilya argues the comparison is misleading because much capital elsewhere goes to inference and product-driven engineering, whereas SSI’s focused research structure makes its compute more comparable for validating new approaches.
- •SSI’s $3B vs competitors’ larger budgets: much of theirs is inference earmark
- •Product companies divert resources to engineering, sales, and product features
- •Compute fragmentation across modalities/workstreams increases demand elsewhere
- •Claim: SSI has enough compute to convince itself and others its approach works
- •Business model deferred: focus on research first; monetization can ‘reveal itself’
- 43:00 – 47:14
Straight-shot superintelligence vs incremental deployment—and why ‘showing the AI’ matters
They debate SSI’s original “straight-shot” framing versus gradual deployment. Ilya argues even a straight-shot plan would involve gradual release, and he increasingly values deploying earlier because it’s hard for society (and even AI researchers) to viscerally grasp future power without seeing systems in action.
- •Pros of straight-shot: insulation from market rat race and forced trade-offs
- •Counterpoint: public exposure to powerful AI communicates reality better than essays
- •Deployment teaches the world; people and institutions will change behavior in response
- •Prediction: as AI feels more powerful, companies become more paranoid about safety
- •Early signs: competitors collaborating on safety; governments/public will demand action
- 47:14 – 1:00:58
AGI, pretraining, and the missing ingredient: continual learning as the real ‘general’ capability
Ilya argues the terms ‘AGI’ and ‘pretraining’ shaped thinking in ways that overshoot reality: humans aren’t static ‘do-everything’ agents; they rely on continual learning. He proposes superintelligence may look like a powerful learner (a ‘super-intelligent 15-year-old’) that gains competence through deployment and on-the-job learning.
- •AGI arose as a reaction to ‘narrow AI,’ but humans aren’t fully general in that sense
- •Pretraining created an illusion of uniform improvement across tasks
- •Humans depend on continual learning rather than complete preloaded competence
- •Superintelligence as the ability to learn any job quickly, not already know all jobs
- •Deployment becomes part of training: trial-and-error learning in real environments
- 1:00:58 – 1:18:25
Alignment: caring about sentient life, capping power, and long-run equilibrium (Neuralink++)
The discussion turns explicitly to alignment goals and governance in a world of multiple continent-scale AIs. Ilya floats “care about sentient life” as a potentially easier-to-anchor target (especially if AIs are sentient), suggests power caps could help, and offers a provocative equilibrium concept: human–AI integration to keep humans meaningfully in the loop.
- •Short-run hope: first N powerful systems are robustly aligned (care for people/sentient life)
- •Concern: even single-minded optimization of a ‘good’ target could yield unwanted outcomes
- •Multiple powerful AIs likely; agreements/restraints could be valuable
- •Long-run equilibrium problem: if personal AIs act for humans, humans may stop participating
- •Proposed (uncomfortable) solution: Neuralink++ so understanding is shared, preserving agency
- 1:18:25 – 1:29:23
SSI’s differentiation, timelines, and why current approaches may ‘stall out’
Dwarkesh presses on what SSI will do differently; Ilya emphasizes a distinct technical approach focused on understanding generalization. He forecasts 5–20 years to human-like learning systems, expects current paradigms to keep generating huge revenues even if they plateau, and anticipates convergence on broad safety strategy as AI visibly advances.
- •SSI framed as an ‘age of research’ company pursuing promising generalization ideas
- •Leadership/Meta acquisition context: cofounder took liquidity; SSI continues independently
- •Forecast: ~5–20 years to human-like learners that become superhuman
- •Current approaches may stall in capability while still producing massive revenue
- •Convergence likely on high-level strategies (coordination + aligned first systems), not immediately on methods
- 1:29:23 – 1:36:03
Self-play, multi-agent diversity, and ‘research taste’ as aesthetic + top-down conviction
In the closing stretch, they discuss why LLMs are so similar (shared pretraining data) and where differentiation may emerge (RL/post-training, multi-agent competition). Ilya reframes self-play as compute-only skill generation with limits, points to debate/prover–verifier as related forms, and ends by describing research taste as pursuit of elegance and brain-inspired beauty that sustains persistence through noisy experiments.
- •LLM similarity attributed largely to shared pretraining; RL/post-training enables divergence
- •Self-play attractive as compute-only data generation, but tends to narrow skill sets
- •Modern analogs: debate, prover–verifier, adversarial judging setups
- •Multi-agent competition can incentivize differentiated approaches and diversity
- •Research taste: beauty/simplicity/elegance + correct brain inspiration + top-down belief to persist through contradictions
