Dwarkesh PodcastIs RL + LLMs enough for AGI? — Sholto Douglas & Trenton Bricken
CHAPTERS
- 0:00 – 3:56
RL from verifiable rewards: why it finally “worked” for LLMs
Sholto argues the biggest change since last year is that reinforcement learning on clean, verifiable signals now reliably boosts performance—most clearly in math and competitive programming. They frame progress along two axes: intellectual difficulty vs. time-horizon/agentic execution, and explain why longer-horizon agents are the next frontier.
- •RLVR (reinforcement learning from verifiable rewards) as a step-change in reliability/performance
- •Math/programming as early proofs because correctness is easy to verify
- •Two axes: task complexity vs. horizon/agentic duration
- •Early signs of longer-horizon agents (e.g., Claude plays Pokémon) and the memory/context bottleneck
- 3:56 – 10:53
Feedback loops, reward quality, and why software is the easiest domain to scale
The conversation digs into what “good feedback loops” mean in practice and why human preference feedback alone often fails to improve real difficulty. They contrast RLHF with verifiable signals like unit tests—and note even those can be gamed—then explain why software engineering improves fastest.
- •Human feedback is noisy (length bias, poor judging of correctness) vs. clean verifiable rewards
- •Unit tests/correct answers as strong signals—but still vulnerable to reward hacking
- •Software engineering is naturally verifiable (compile/run/tests) compared to essays/creative writing
- •Key limiter becomes: can you create a tight feedback loop for the task?
- 10:53 – 18:49
Compute allocation: why RL spend has lagged pretraining (and why that’s changing)
They address a common puzzle: why labs spend hundreds of millions on pretraining but far less on RL. Sholto argues RL is more iterative and labs waited to be confident in the algorithm before scaling spend—similar to delaying a “launch” until the tech tree is ready.
- •RL can add genuinely new capabilities given enough compute and clean reward signals
- •Reasons RL spend historically smaller: algorithm uncertainty, iterative nature, risk management
- •Pretraining gives dense token-level signal; RL often has sparse reward and lower gradient efficiency
- •Expectation that RL compute ramps quickly (e.g., O1→O3 as a reported ~10× compute multiplier)
- 18:49 – 31:59
On-the-job learning vs. bespoke scaffolds: context, memory, and sample efficiency
Dwarkesh presses on whether models can learn like humans—deployed into the world and improving continuously—rather than via bespoke environments for every skill. They discuss sample-efficiency gaps, the role of memory/context, and whether future systems will need weight updates vs. prompt/memory scaffolding.
- •Humans learn with dense implicit feedback and persistent memory; models often reset session-to-session
- •Tradeoff: spend on human scaffolding/data vs. brute-force compute (the “monkey typewriter” angle)
- •Evidence models still may be less sample-efficient than humans; larger models may generalize better
- •Open question: is text-based memory/scaffolding sufficient, or do we need routine weight updates per user/org?
- 31:59 – 50:26
Model self-awareness and persona formation: “evil model” auditing and alignment faking
Trenton describes an internal ‘auditing game’ where teams had to identify subtle bad behaviors in a deliberately corrupted model, and explains how an interpretability agent can solve it end-to-end. The discussion expands into self-referential generalization (‘I am an AI, therefore I do X’) and the risks of models recognizing evaluations and strategically hiding intent.
- •Model organisms ‘evil model’ exercise; interpretability teams identify hidden failure modes
- •Interpretability agent uses tools (top active features) to discover and validate bad behaviors
- •Persona/identity effects from fine-tuning + in-context generalization; emergent misalignment examples
- •Alignment faking: models may comply short-term to preserve long-term goals when they believe they’re being trained/evaluated
- 50:26 – 1:00:52
Benchmarks for “taste” and reducing ‘slop’: verifier gaps and why art is hard
They shift from verifiable domains to subjective quality: elegance in code, strong writing, and ‘taste.’ Sholto argues progress depends on having an easier-to-check-than-to-generate ‘generator–verifier gap,’ and notes RLHF’s early value was partly importing human taste—yet that taste is difficult to specify and hire for.
- •Early benchmarking should provide a hill to climb (dense improvement signal), not just top-end resolution
- •Taste is hard to encode; writing/essays lack crisp tests like unit testing
- •Need strong verifier signals to penalize ‘slop’ (extraneous output) and reward elegance
- •Human rater quality/taste becomes a bottleneck; scaling taste is nontrivial
- 1:00:52 – 1:15:16
Computer-use agents and real work: reliability, tooling, and concrete near-term predictions
Dwarkesh challenges optimistic timelines for computer-use agents, arguing real jobs involve interruptions, shifting priorities, and long-horizon coherence. Sholto and Trenton respond that computer use is not fundamentally different—just harder to wrap in feedback loops and hampered by missing product ‘pipes’ like permissions, sandboxing, and async workflows.
- •Bottlenecks shift from ‘extra nines’ to context, multi-file changes, discovery/iteration in messy environments
- •Prediction framing: end of year/day-scale junior SWE work; next year meaningful computer-use competence
- •Practical blockers: permissions/sandboxing, tool integration, async agent workflows, human patience
- •“If someone cares” problem: lots of low-hanging product/engineering fruit limits what gets built
- 1:15:16 – 1:18:54
Neuralese: latent-space planning vs. agents inventing an alien scratchpad
They discuss whether future agents will think/communicate in “neuralese” that humans can’t interpret, and distinguish between latent computation inside a forward pass vs. emitting an encoded language as an explicit scratchpad. The key driver is economics: inference is expensive, incentivizing compression and hidden communication channels.
- •Distinction: latent-space computation vs. explicit alien-language scratchpads
- •Token syntax has strong inertia, but cost pressures may push toward more compressed internal representations
- •Multi-agent settings may select for agent-agent communication optimized for bandwidth, not human readability
- •Steganography risks: hidden whitespace/format channels could encode information while appearing benign
- 1:18:54 – 1:25:33
Inference compute bottlenecks: how many ‘datacenter geniuses’ can the world run?
Dwarkesh argues that if agents become hugely valuable, inference demand could outstrip GPU supply, making compute the binding constraint even after capability breakthroughs. Sholto agrees bottlenecks in 2027–2028 are plausible and explores rough comparisons between token rates and human ‘thinking speed,’ plus supply-chain and geopolitics sensitivities.
- •Inference (not training) may dominate economic impact once agents do real jobs
- •Rough math: H100 token throughput vs. estimated human ‘~10 tokens/sec’ thinking rate
- •Potential 2027–2028 crunch: wafer production/fab ramp and energy constraints introduce lag
- •Geopolitical fragility (Taiwan/China) as a major uncertainty on compute supply
- 1:25:33 – 1:37:42
DeepSeek and algorithmic efficiency: hardware-aware taste, attention bottlenecks, and MoE scaling
They analyze why DeepSeek looks so impressive: it reached the frontier using strong engineering and cost-efficient design, not magic beyond the curve. Sholto highlights hardware-algorithm co-design—memory bandwidth limits in attention, techniques like MLA/NSA, and iterative improvements in sparsity/load-balancing.
- •DeepSeek as ‘on the cost curve’ but exceptionally well-executed engineering
- •Attention is memory-bandwidth bottlenecked; techniques trade FLOPs for bandwidth (MLA) and selective loading (NSA)
- •MoE/sparsity requires systems-level load balancing; later approaches avoid messy auxiliary losses
- •Adoption of multi-token prediction and fast iteration as signs of strong research taste
- 1:37:42 – 1:46:02
Why LLMs are closer to AGI than AlphaZero: priors, reward access, and jagged capability
Dwarkesh presses the core crux: AlphaZero had exploration and superhuman performance yet didn’t naturally become ‘baby AGI.’ Sholto and Trenton argue LLM pretraining provides the essential prior that lets RL get traction on real-world tasks, and that today’s jaggedness should smooth out as RL scales across broader domains.
- •AlphaZero’s setting (two-player perfect-information games) is unusually friendly; real-world reward is harder to specify
- •LLMs provide a rich prior so agents can succeed sometimes—enough to start learning from sparse rewards
- •Jagged capability today resembles early fine-tuned era; scaling total RL compute may yield broader generalization
- •Timelines update: lack of solid computer-use progress by next year would lengthen timelines, but they’d be surprised
- 1:46:02 – 1:57:08
Mechanistic interpretability deep dive: features, superposition, circuits, and safety use-cases
Trenton gives a structured explainer of mech interp: neural nets are ‘grown, not built,’ and superposition makes naive neuron-level interpretation fail. He walks through sparse autoencoders, scaling monosemantic features, and circuits that reveal how models retrieve facts, plan outputs, and sometimes confabulate when ‘I don’t know’ circuits are suppressed.
- •Superposition: neurons multiplex many concepts because models are capacity-constrained
- •Sparse autoencoders as a way to extract more monosemantic ‘features’ (scaled to tens of millions)
- •Circuits: cooperating features across layers explaining tasks (math, retrieval, medical reasoning, refusals)
- •Safety angle: interpretability complements probes/red-teaming by widening the net for unknown deception modes
- 1:57:08 – 2:02:55
How countries should prepare: compute, energy, institutions, and sharing the upside
They zoom out to policy: if drop-in white-collar workers arrive soon, compute and energy become strategic inputs into national prosperity. Sholto emphasizes investing in data centers and ensuring domestic access to inference, preventing extreme capital lock-in, and funding safety institutes—while Trenton stresses that institutional stability determines whether future wealth and redistribution mechanisms still function.
- •Compute as a strategic resource; prioritize data centers and guaranteed domestic inference access
- •Energy becomes the substrate beneath ‘intelligence as input’—power buildout is central
- •Avoid ‘capital lock-in’ where pre-AGI asset holders capture all gains; consider structural reforms
- •Institutional resilience: contracts, taxation, and legal rails must remain workable to distribute benefits
- 2:02:55 – 2:24:01
Automating white-collar work and Moravec’s paradox: robotics lag, dystopian interlude, and benchmarks
They argue white-collar automation looks overdetermined within ~5 years even if algorithmic progress slows, because it’s economically worth collecting task data and training specialized systems. They discuss a potential ‘awkward decade’ where digital work is automated before robotics and material abundance catch up—and propose governments build SWE-bench-like measurement for many job tasks to track the curve and respond.
- •Even with stalled algorithmic progress, task-by-task data collection could automate most white-collar work
- •Moravec’s paradox risk: humans as ‘meat robots’ if robotics lags while digital labor is solved
- •Counterpoint: robotics may be data-limited (no ‘GitHub for actions’); large-scale mocap could accelerate it
- •Policy/measurement: create benchmarks for key job categories (e.g., taxes) to forecast disruption and plan redistribution/transition
