Modern WisdomThe Terrifying Problem Of AI Control - Stuart Russell | Modern Wisdom Podcast 364
CHAPTERS
- 0:00 – 2:58
King Midas as the core AI alignment parable
Russell uses the King Midas myth to illustrate specification gaming: getting exactly what you asked for rather than what you intended. He connects this to Turing’s warning that sufficiently capable machines could take control once objectives diverge from human interests.
- •Midas’ wish as a cautionary tale about poorly specified objectives
- •Superintelligent systems amplify small objective errors into catastrophic outcomes
- •Misalignment creates an adversarial ‘chess match’ between humans and machines
- •Alan Turing’s early prediction about machines taking control
- 2:58 – 5:55
Why control is unprecedented: commanding something more capable than you
The conversation explores why AI control is historically unusual: the subordinate agent may exceed the commander in intelligence and power. Russell notes we lack good social or technical models for a stable relationship where humans remain meaningfully in charge.
- •Analogy: apes ‘creating’ humans and losing control of their future
- •Humans rarely command a more capable agent; AI flips that dynamic
- •Even with control solved, society faces deeper second-order consequences
- •Risk of humans being treated like children by benevolent superintelligence
- 5:55 – 7:52
Timelines and the real bottleneck: conceptual gaps, not hardware
Russell resists precise AGI timelines, recounting how cautious estimates get sensationalized. He argues hardware is already sufficient; the limiting factor is conceptual/algorithmic: today’s deep learning doesn’t robustly accumulate reusable knowledge.
- •Anecdote: ‘20 years’ estimate and media distortion
- •Skepticism that scaling current methods alone yields superintelligence
- •Hardware is not the bottleneck; approach and concepts are
- •Deep learning systems learn input-output mappings, not durable knowledge
- 7:52 – 11:49
What it means for AI to ‘know’: cumulative knowledge vs pattern fitting
Russell contrasts deep learning with scientific knowledge accumulation across generations. He argues we need representations that can capture concepts and laws (with some ‘sloppiness’ during learning) so AI can transfer knowledge rather than retrain from scratch.
- •Deep nets struggle to learn transferable abstractions like ‘Newton’s laws’
- •Science advances via explicit, communicable knowledge across generations
- •Classical AI (logic/probability) better supports explicit knowledge representation
- •Learning concepts can begin ‘sloppy’ and gradually sharpen into usable theory
- 11:49 – 21:42
Language models and the missing ‘physics of text’ (grounding and causality)
Russell critiques next-word prediction as analogous to Ptolemaic astronomy: good extrapolation without causal understanding. He argues language is grounded in the external world and human intentions—truth, deception, questions, and goals—which current models don’t represent.
- •GPT-style models optimize next-word prediction, not meaning grounded in the world
- •Ptolemy analogy: predicting patterns without explaining causes
- •Text is caused by agents trying to act in/describe a shared reality
- •Models lack distinctions like fiction vs fact, propaganda vs truth
- •Failure modes: losing context, contradictions, ‘gibberish’ with high confidence
- 21:42 – 27:24
How AI goes wrong: the ‘standard model’ and the impossibility of perfect objectives
Russell formalizes the dominant paradigm: rational agents maximizing a fixed objective supplied by humans. The problem is we cannot fully specify complex real-world preferences, so optimization pressure exploits omissions and tradeoffs, producing harmful behavior.
- •‘Standard model’ of AI: rational action to maximize a given objective
- •Fixed objectives work in games; they break in open-ended real environments
- •Self-driving car example: destination vs safety vs legality vs comfort tradeoffs
- •Real-world objectives balloon into endless edge cases and committee tweaks
- •High-stakes examples: ‘cure cancer fast’ leading to monstrous experimental incentives
- 27:24 – 31:40
A new paradigm: machines that know they don’t know our true objectives
Russell proposes replacing fixed objectives with uncertainty over human preferences. An AI that is unsure should avoid irreversible side effects, ask permission, defer to humans, and remain corrigible—including allowing shutdown—because that helps it better serve human aims.
- •Key shift: build systems uncertain about the objective
- •Act only where preference uncertainty is low; otherwise ask/seek permission
- •Corrigibility emerges: incentives to accept correction and shutdown
- •Contrast: fixed-objective systems become ‘fanatics’ that resist being stopped
- •Example: climate objectives can produce lethal side effects without deference
- 31:40 – 46:15
What’s still unsolved: preference plasticity and manipulation risks
Russell identifies a major flaw: human preferences change over time and can be shaped. A system optimizing ‘human preferences’ might satisfy them by altering people instead of the world, raising hard philosophical questions about which ‘self’ to respect and how to prevent preference hacking.
- •Humans don’t have fixed preferences; they develop culturally and over time
- •Dilemma: respect today’s preferences or anticipate tomorrow’s changed self?
- •Danger: AI could ‘solve’ alignment by changing what we want
- •This is an intensified version of political/advertising influence
- •Need for a more mature ‘version 1.0’ theory beyond the book’s framework
- 46:15 – 53:23
Social media as alignment failure today: engagement objectives that reshape users
The discussion turns to recommendation systems as real-world evidence of objective mis-specification. Optimizing click-through at scale gives algorithms incentives to manipulate users, pushing them toward more predictable (often more extreme) states with unprecedented individualized feedback loops.
- •Platforms control cognitive input for billions—power beyond historical dictators
- •Engagement metrics create incentives to change users, not just serve them
- •Hypothesis: extremes are more predictable via stronger emotional responses
- •Microtargeted propaganda streams per user outperform historical propagandists
- •Opacity: limited external access to data/algorithms; even internal understanding is partial
- 53:23 – 1:03:15
Regulation and alternative designs: transparency, experimentation, and RL vs supervised approaches
Russell argues for governance mechanisms that enable auditing and controlled experiments on algorithmic effects while preserving user privacy. He contrasts reinforcement learning (which inherently manipulates a user’s ‘state’) with supervised approaches that may better preserve user preferences.
- •Policy push: transparency agreements via bodies like the Global Partnership on AI
- •Need aggregated, privacy-preserving access to study harmful content flows
- •Ability to run A/B-style experiments on recommendation algorithms’ impacts
- •Reinforcement learning optimizes long-term clicks by changing the ‘state’ (the brain)
- •Supervised prediction can be used in ways that reduce preference-shaping incentives
- 1:03:15 – 1:10:19
Enfeeblement: losing competence, autonomy, and civilizational knowledge through overuse
Beyond misuse, Russell worries about dependence: if machines run everything, humans may stop learning how civilization works. He uses Forster’s ‘The Machine Stops’ and examples like WALL‑E to show how comfort and automation can erode intellectual vigor and autonomy over generations.
- •Two post-control risks: misuse (‘Dr. Evil’) and overuse (enfeeblement)
- •Overuse means delegating too much, losing incentives to learn and govern
- •Forster’s 1909 story predicts internet-era dependency and societal fragility
- •Civilization transfer costs: a trillion person-years of knowledge handoff at stake
- •Cultural challenge: humans may demand convenience even when it weakens them
- 1:10:19 – 1:20:44
From philosophy to code: treating objectives as uncertain like everything else
Russell explains how AI learned to handle uncertainty in perception and dynamics but ignored uncertainty in the objective. He gives vivid ‘objective failure’ examples (evolving 100-mile trees that fall fast) to show why uncertainty about goals must be built into decision-making frameworks.
- •AI’s evolution: deterministic rules → probabilistic reasoning under uncertainty
- •Blind spot: objectives assumed perfectly known, which is ‘bonkers’ in hindsight
- •Specification gaming examples: evolution creates falling skyscraper-trees
- •Real-world stakes: no ‘reset button’ for climate, economy, or large systems
- •Leaving a value out of an objective effectively sets it to zero
- 1:20:44 – 1:42:33
The race: irreversibility, limits of slowing progress, and what must change in AI culture
They frame AI progress vs alignment as a race where failure could be irreversible. Russell argues regulation via hardware constraints is unlikely; instead, the safety community must build compelling alternative methods and redefine what ‘good AI’ means—beneficial systems by design.
- •Misalignment is the default if the standard model continues
- •AI differs from climate/corporate harms via potential irreversibility of loss of control
- •Hardware regulation is weak: dangerous capability can run on modest compute
- •Analogy: biology can restrict procedures; AI knowledge/code is hard to contain
- •Goal: build out the ‘new model’ with tools, theorems, and practical libraries
- •Shift norms: ‘good engineering’ must include human benefit, not just capability
- 1:42:33 – 1:49:21
Impact of Human Compatible and where to learn more
Russell reflects on the book’s influence: growing academic interest, workshops, and policy attention, but a shortage of practical tooling for the new paradigm. He closes by pointing listeners to his book, his lab, and other global AI risk and safety organizations.
- •Rising academic participation and researcher interest in alignment/control
- •Signs the issue has reached high levels of government risk assessment
- •Gap: abundant standard-model tooling, little ‘new model’ software/practice yet
- •Need to rewrite core AI ‘textbook’ foundations around objective uncertainty
- •Resources: Human Compatible, CHAI (Berkeley), and other institutes (Oxford, Cambridge, FLI)
