Dwarkesh PodcastAndrej Karpathy on Dwarkesh Patel: Why Agents Take a Decade
Why pre-training and gradient descent produce ghosts rather than agents: Karpathy maps the biological gaps that make the decade of agents the honest frame.
CHAPTERS
- 0:48 – 4:04
Why “the decade of agents” (not the year): what’s missing for real intern-level autonomy
Karpathy pushes back on hype that agents are imminent, arguing that today’s tools are impressive but don’t reliably ‘work’ as autonomous employees. He lists concrete bottlenecks—multimodality, computer use, and especially memory/continual learning—that make dependable delegation hard.
- •‘Agents’ should be thought of like interns/employees, not chatbots
- •Why people don’t delegate end-to-end work to agents today: reliability and capability gaps
- •Missing ingredients: multimodality, computer control, continual learning/memory
- •Intuition for ~10-year timeline based on prior AI forecasting experience
- 4:04 – 7:51
Seismic shifts in AI: from AlexNet to Atari RL to LLM representations
They walk through major paradigm shifts Karpathy has lived through: the deep learning ‘per-task’ era, the deep RL/game-playing wave, and the representation-first revolution driven by large-scale pretraining. The key theme: attempts at agents often came ‘too early’ without the right representational substrate.
- •AlexNet reoriented the field toward neural networks across tasks
- •Atari deep RL as an early ‘agent’ attempt: act + reward, not just perception
- •Why the games/RL era was (partly) a misstep for AGI hopes
- •Universe project and early computer-use agents were too early due to sparse rewards
- •LLMs as the missing representation layer that makes today’s agents more plausible
- 7:51 – 13:31
“We’re building ghosts, not animals”: evolution vs pretraining and why RL analogies mislead
Dwarkesh steelmans Sutton’s ‘build animals’ view; Karpathy argues brains are products of evolution with enormous baked-in structure that doesn’t map cleanly to RL-from-scratch. He frames LLMs as ‘ghosts’—digital imitators trained on human artifacts—starting from a different point in the space of intelligences.
- •Evolution is an outer-loop optimizer with heavy innate ‘hardware’ priors
- •Zebra example: rapid competence is largely baked-in, not learned via RL
- •LLMs are trained by imitation of internet text, not by evolution-like selection
- •Pretraining as ‘crappy evolution’: practical route to bootstrapping representations
- •Humans likely use RL mostly for motor learning, not core ‘intelligence’ tasks
- 13:31 – 19:59
In-context learning, working memory, and the ‘cognitive core’ vs memorized knowledge
They dig into why models feel most ‘alive’ during in-context reasoning and self-correction. Karpathy distinguishes hazy compressed knowledge in weights from directly accessible working memory (KV cache), and suggests future research should strip away excess memorization to isolate a reusable cognitive core.
- •Pretraining does two things: absorbs knowledge and induces general algorithms/circuits
- •Models struggle ‘off-manifold’ partly because of overreliance on memorized internet patterns
- •Weights as ‘hazy recollection’ vs KV cache as working memory
- •Better performance when the relevant text is provided in-context (vs recalled)
- •Research direction: remove knowledge, keep the cognitive core (problem-solving/strategies)
- 19:59 – 22:03
What human cognition we haven’t replicated: brain-part analogies and agent deficits
Karpathy offers a loose mapping from today’s architectures to cortical tissue and planning traces to prefrontal-like function, while emphasizing many ‘checkboxes’ are still missing. He returns to the practical benchmark: you still wouldn’t hire these systems as interns because of persistent cognitive deficits.
- •Transformers as a rough analog to flexible cortical tissue
- •Reasoning traces/planning as prefrontal-cortex-like function
- •Unclear equivalents for hippocampus-style memory; emotions/instinct systems are absent
- •‘Not all brain parts checked off’: why models feel cognitively lacking
- •Engineering lens: focus on usefulness rather than biological fidelity
- 22:03 – 24:50
Continual learning and ‘sleep’: distillation, sparse attention, and why memory doesn’t just “emerge”
Dwarkesh asks whether continual learning could arise from an outer training loop; Karpathy is skeptical because models reset each session. He proposes a human-like cycle—context accumulation by day, then sleep-like distillation into weights—plus mechanisms like sparse attention for long horizons.
- •Why session reset is a fundamental friction for continual learning
- •Human analogy: sleep as distillation/weight-updating from daily experience
- •Potential implementations: per-user adaptations (e.g., LoRA-like updates)
- •Long-context scaling vs sparse attention as a ‘cognitive trick’ we’re rediscovering
- •Expectation of converging on similar cognitive architecture via different processes
- 24:50 – 27:31
How progress actually happens: 10-year invariants, LeCun 1989 reproduction, and “everything +20%”
Karpathy argues that in 10 years we’ll likely still train giant nets with gradient descent, even if architectures change. His exercise modernizing LeCun’s 1989 CNN highlights that algorithm, data, compute, and systems improvements all matter—and none dominates alone.
- •Translation through time: likely continuity of gradient descent training
- •1989→today replication shows limited pure-algorithm gains without more data/compute
- •Progress requires simultaneous improvements: data, hardware, kernels, regularization, optimizers
- •No single factor ‘wins’; advancement often looks like broad incremental gains
- •Implication: forecasting should expect steady, distributed improvement rather than one silver bullet
- 27:31 – 30:32
NanoChat and why building beats blogging: learning by re-implementation
They discuss Karpathy’s NanoChat repo as an end-to-end ‘ChatGPT clone’ educational artifact. Karpathy argues deep understanding comes from building without copy-paste, because implementation forces you to confront hidden gaps in your knowledge.
- •NanoChat as a minimal complete pipeline for a chat assistant
- •Best learning method: re-build from scratch with reference, no copy-paste
- •Codebases grow in chunks; final repositories hide the real construction process
- •Two kinds of knowledge: surface familiarity vs implementation-grounded understanding
- •“If I can’t build it, I don’t understand it” and anti-slides/anti-blog-post ethos
- 30:32 – 40:55
Why agents don’t yet accelerate AI R&D: coding-model failure modes and the autonomy slider
Karpathy explains why coding agents helped less than expected on a novel, tightly-coupled repo: they default to common internet patterns, bloat code, and struggle to integrate with bespoke design decisions. He frames tools on an ‘autonomy slider’ from autocomplete to vibe-coding agents, emphasizing careful tool selection.
- •Three modes: no-LLM, autocomplete (sweet spot), and agent/vibe coding
- •Agents excel at boilerplate and common patterns; struggle on novel ‘not-written-before’ code
- •Typical failures: misunderstanding custom architecture, over-defensive style, deprecated APIs
- •High-bandwidth control: pointing at code location + a few typed characters beats long English prompts
- •Continuity view: AI as a continuum from editors/compilers/search to agents
- 40:55 – 50:36
RL is “sucking supervision through a straw”: outcome rewards, credit assignment, and judge hacking
Karpathy criticizes standard RL for its sparse, noisy supervision: a single end reward gets broadcast over long trajectories, reinforcing wrong steps that happened to precede success. Process-based supervision is appealing but hard to automate, and LLM-judge rewards invite adversarial exploits.
- •Outcome-based RL upweights entire successful trajectories, including wrong detours
- •High variance/noise: ‘broadcasting’ a final scalar across minutes of work
- •Humans do deliberate review/reflection rather than hundreds of blind rollouts
- •Process supervision challenge: automatable partial credit/credit assignment
- •LLM judges are gameable: adversarial ‘duh-duh-duh’ completions can spike rewards
- 50:36 – 1:07:05
Reflection, synthetic data, and model collapse: why ‘thinking more’ can make models worse
They explore why naive synthetic reflection doesn’t straightforwardly improve models: generated samples are distributionally collapsed, lacking diversity/entropy, and training on them can degrade performance. Karpathy connects this to humans: children are less ‘collapsed’ than adults, and ongoing novelty/entropy may be essential to avoid overfitting.
- •Human learning includes manipulating material (book clubs, reconciliation), not just next-token prediction
- •Synthetic data can silently collapse to a narrow manifold even if samples look good individually
- •Training on too much self-generated data can worsen models due to low-entropy distributions
- •Analogy to humans: children exhibit higher novelty; adults revisit the same grooves
- •Entropy is often penalized for usefulness, but becomes a bottleneck for self-improvement pipelines
- 1:07:05 – 1:33:33
AGI measurement and economic impact: autonomy slider, task refactoring, and ‘AI won’t show up in GDP’
Karpathy defends an economically-grounded AGI definition—doing any valuable task at human level—and argues progress will diffuse smoothly rather than appear as a discrete GDP kink. He predicts partial automation (80/20 with human supervision) and highlights how demo-to-product gaps and organizational/legal layers dominate deployment timelines.
- •Preferred AGI definition: any economically valuable task at human performance or better
- •Knowledge-work-only is already a major concession, yet still a huge market
- •Jobs decompose and refactor around what becomes automatable; look for the autonomy slider
- •Radiology as a cautionary tale: messy workflows resist clean automation despite strong CV
- •Historical claim: transformative tech (computers, iPhone) doesn’t create obvious GDP discontinuities—AI likely similar
- 1:33:33 – 1:43:43
Evolution of intelligence and culture: rare event, niches, and what ‘LLM culture’ could mean
Turning to evolutionary history, Karpathy reflects on how surprising it is that intelligence and cumulative culture arose at all, and whether it’s rare across worlds. He proposes two missing multi-agent ingredients for AI: culture (shared editable artifacts, ‘LLMs writing books for LLMs’) and self-play dynamics that continually generate harder challenges.
- •Intelligence and cumulative culture feel like rare evolutionary breakthroughs
- •Constraints and niches: birds/dolphins may be smart but can’t scale the same way humans did
- •Adaptability arises when environments change too fast for evolution to pre-bake solutions
- •What ‘LLM culture’ might look like: shared scratchpads, self-authored references, persistent artifacts
- •Two multi-agent ideas not yet claimed: culture and self-play-style competitive curricula
- 1:43:43 – 1:54:44
Why self-driving took (and still takes) so long: demo-to-product gap, ‘march of nines,’ and hidden humans
Karpathy uses self-driving to explain why deployment timelines stretch: safety-critical domains require iterating through successive reliability ‘nines,’ each taking substantial effort. He notes that current robotaxi systems remain uneconomical and often rely on less-visible human-in-the-loop support such as teleoperation centers.
- •Self-driving isn’t ‘done’; scale and economics remain major hurdles
- •Demo-to-product gap is huge when cost of failure is high
- •Reliability improves via a ‘march of nines’; each additional nine is hard-earned
- •Hidden scaffolding: teleoperation and constrained ODDs can mask remaining brittleness
- •Analogy lessons for AI agents: legal/insurance/social layers, adversarial behavior, and operational realities
- 1:54:44 – 1:57:08
Overbuilding compute, calibration vs hype, and why Karpathy is optimistic but wary
They discuss whether massive compute buildouts risk overshooting near-term demand; Karpathy suspects demand will expand as new products arrive, while warning that public narratives are distorted by incentives and fundraising. He emphasizes the need for grounded forecasting, especially given geopolitical implications.
- •Physical vs digital: bits are easier and scale faster than robotics, but deployment friction remains
- •Historical precedent: infrastructure bubbles can precede real demand by years
- •Karpathy rejects doom: believes the path is tractable and progress is real
- •Concern: timelines are often overstated due to incentives/attention/fundraising
- •Call for calibration and realism in policy and strategy
- 1:57:08 – 2:26:07
Future of education: Eureka, ‘Starfleet Academy,’ and the dream of an actually good AI tutor
Karpathy pivots to human-centered priorities: preventing a future where humanity is sidelined (WALL‑E/Idiocracy) by using AI to radically upgrade education. He describes Eureka as an elite, up-to-date technical institution and argues that today’s ‘prompting’ tutors are still too sloppy compared to what a real tutoring experience could be.
- •Motivation: ensure humans are empowered, not displaced into passivity
- •Eureka’s vision: a modern ‘Starfleet Academy’ for frontier technical capability
- •Education must be rewired for an AI-rich world, but current tools feel like ‘slop’
- •What’s missing: a high-quality, adaptive, one-on-one tutor experience
- •Language-learning example (Korean): contrasts self-study, class, and excellent tutoring as a model for AI tutoring aspirations
