Modern WisdomShocking Ways AI Could End The World - Geoffrey Miller
CHAPTERS
- 0:00 – 0:25
Why super-fast, super-capable AI could outclass humans
Geoffrey opens with the core intuition behind AI existential risk: systems that aren’t just smarter than humans across many domains, but dramatically faster. The speed advantage changes the human-AI relationship from “tool use” to “being unable to keep up,” especially in high-stakes arenas.
- •AI risk is amplified by reaction speed, not just intelligence
- •General-purpose capability plus huge speed creates human helplessness
- •Analogy: speedster superheroes make everyone else feel frozen
- •Outclassing applies to many domains, not just one task
- 0:25 – 2:41
Why an evolutionary psychologist is focused on AI
Geoffrey explains his long-standing background in neural networks, genetic algorithms, and robotics before moving into evolutionary psychology. He describes how recent deep learning progress and effective altruism circles pulled him back toward AI risk work.
- •Early training in cognitive science and neural nets at Stanford
- •Work on autonomous robots and evolving neural networks
- •Renewed concern since ~2016 due to deep learning/LLMs
- •Influence of effective altruism and existential risk community
- 2:41 – 9:54
AI as the leading existential risk (and why this century feels uniquely dangerous)
The discussion frames AI within the broader x-risk landscape, referencing Toby Ord’s probability estimates and comparing AI to nukes and engineered pathogens. Geoffrey uses vivid metaphors to describe humanity entering an unusually high-risk era where mistakes could be terminal.
- •Ord’s ‘The Precipice’ and the one-in-six AI extinction estimate
- •Big three risks: AI, nuclear war, engineered bioweapons
- •“Key century” framing: unusually elevated risk window
- •Metaphors: narrow mountain path / Free Solo climb without ropes
- 9:54 – 13:06
From ‘tool’ to ‘agent’: the pivotal danger is outsourcing control
Chris challenges why smarter/faster AI isn’t just a better tool (like heavy machinery). Geoffrey argues the real danger begins when we give systems agency—decision-making power—because competition will incentivize autonomous, ultra-fast action loops.
- •Tool assistance is different from delegated agency
- •AI can nudge/manipulate even without full control
- •Strong incentives to automate decisions in finance and military
- •Speed pressures humans to remove ‘human-in-the-loop’ safeguards
- 13:06 – 15:47
How AI-risk thinking changed since ‘Superintelligence’: narrow AI can be dangerous now
They contrast older “fast takeoff/singularity” fears with newer concerns that near-term, narrower systems can destabilize society. Geoffrey gives examples like AI-assisted bioweapon design and deepfake-triggered crises that don’t require AGI.
- •Bostrom’s self-improvement ‘takeoff’ remains a concern
- •Shift toward recognizing severe risks from narrow systems
- •Bioweapon design and gain-of-function simulation as a near-term threat
- •Deepfakes could provoke panicked responses and even nuclear escalation
- 15:47 – 20:05
Why large language models blindsided experts: scale, hardware, and emergent capability
Geoffrey explains how exploding compute and parameter counts enabled unexpected “emergent” abilities. ChatGPT exceeded expectations by performing tasks it wasn’t explicitly designed for, forcing a reassessment of timelines and preparedness.
- •From thousands of parameters in the 1980s to trillions today
- •GPU/hardware advances made deep learning scale practical
- •Emergent capabilities surprised many researchers and the public
- •Concern: future jumps may blindside us even more
- 20:05 – 22:47
What AGI means, why ChatGPT isn’t there yet, and why copying capability is the multiplier
They define AGI in practical labor-market terms: the ability to perform most cognitive jobs at professional levels. Geoffrey emphasizes that copying and specializing instances of an AGI is what makes it transformative—and dangerous, because it scales both ‘good’ and ‘bad’ work.
- •AGI as broad, job-level human competence across domains
- •Copyability enables mass parallelization of expertise
- •Automation includes harmful roles: terrorism, spying, propaganda
- •AGI raises both unemployment and existential-risk concerns
- 22:47 – 25:55
Sam Altman, ‘good guys vs bad guys,’ and the logic of an AI arms race
Geoffrey offers a mixed view of Altman—admiring his intelligence but criticizing what he sees as cognitive dissonance about extinction risk. The conversation then tackles the ‘we must win first’ argument and whether an arms race toward catastrophe is rational.
- •Altman’s utopian vision vs claimed awareness of x-risk
- •If risk were truly taken seriously, Geoffrey argues you’d stop development
- •‘Good guys must win’ framing challenged as simplistic
- •Arms race described as a game of chicken into a wall
- 25:55 – 33:26
Governance vs grassroots: slowing AI through regulation, stigma, and public mobilization
Geoffrey contrasts slow, capture-prone policy approaches with a bottom-up strategy: stigmatizing reckless AI development similarly to tobacco, arms, or other socially restricted industries. A key hurdle is making risks vivid enough for the public to care.
- •Traditional AI governance is important but may be too slow
- •Risk of regulatory capture and low politician understanding
- •Proposal: moral stigmatization of reckless AI development
- •Need for imagination, fiction, and scenario-building to communicate risk
- 33:26 – 40:52
China’s incentives, censorship constraints, and how the US may be setting the arms-race pace
They discuss China’s more controlled public rollout and its emphasis on stability and censorship rather than techno-utopianism. Geoffrey argues US leadership may be forcing competitors into catch-up dynamics—and that US slowdown could reduce pressure globally.
- •China may restrict public models due to political control concerns
- •Different motivations: stability/social control vs ‘singularity’ visions
- •US firms appear far ahead, creating perceived hegemony threats
- •Claim: slowing down in the US could allow others to relax too
- 40:52 – 47:55
Alignment: what it is, why it’s hard, and whose values get encoded
Geoffrey defines alignment via a principal–agent analogy, then attacks the practical problem: humans disagree, and ‘alignment’ often defaults to the values of a narrow tech elite. Chris pushes the challenge further with moral uncertainty and conflicting preferences.
- •Alignment = ensuring AI goals match human values and intent
- •Genie/King Midas failures: literalism and unintended consequences
- •Whose values: end-user vs collective humanity vs elite norms
- •Value conflict, religion vs secularism, and moral uncertainty complicate coding ethics
- 47:55 – 51:13
Beyond ‘human values’: embodied values and non-human stakeholders
Geoffrey adds two less-discussed alignment layers: values embedded in biology (health, immune function, bodily needs) and the interests of other species impacted by world-scale AI. He argues current feedback-based alignment can’t capture these deeper or broader value sets.
- •‘Embodied values’ extend beyond what humans can articulate
- •Human-feedback training can’t encode bodily/biomedical interests fully
- •AGI as a major evolutionary transition affecting all life
- •Question: how to represent the interests of animals and ecosystems
- 51:13 – 55:43
The anti-doomer steelman, longevity as the strongest ‘pro-AI’ argument, and opportunity costs
Chris raises the opposing view that pausing AI carries massive opportunity costs. Geoffrey says most ‘AI will solve everything’ claims are weak, but admits longevity and anti-aging research is a serious temptation—then argues we could fund longevity directly without racing to AGI.
- •Best anti-doomer argument: stopping AI sacrifices potential benefits
- •Geoffrey is skeptical AI is required for climate/peace solutions
- •Longevity is the most compelling pro-AI case in his view
- •Alternative: invest directly in longevity research rather than justify it via AGI
- 55:43 – 1:02:17
Near-term harms: election manipulation, mass-personalized propaganda, and social-media warfare
Geoffrey predicts the most immediate shock will be political: AI-enabled microtargeting, deepfakes, and customized persuasion at scale. He extends this into a broader information environment where AI supercharges the ongoing battle to shape public worldview.
- •2024 elections as a proving ground for AI persuasion
- •Microtargeted propaganda tailored to individual voter psychology
- •Functional ‘theory of mind’ via modeling preferences from data
- •AI-accelerated culture war and ‘hegemony’/worldview competition
- 1:02:17 – 1:04:57
Friend-bots, pseudo-intimacy, and a likely cultural backlash
They explore AI companions that remember everything, optimize interactions, and outperform human attention and patience. Geoffrey argues this could erode dating, relationships, and birthrates—triggering moral, religious, or political backlash against AI.
- •AI companions could feel better than real relationships for many
- •Persistent memory and optimization make pseudo-intimacy compelling
- •Social consequences: isolation, reduced dating, falling birthrates
- •Backlash likely as harms become visible and culturally salient
- 1:04:57 – 1:20:05
Expert predictions, protest tactics, S-risk, and why ‘AI will save the world’ arguments fail
The conversation covers open letters, expert humility after being surprised by GPT, and the role of activism in shifting public attention. Geoffrey introduces S-risk (suffering risk) as potentially worse than extinction, criticizes techno-optimist essays, and ends with a call for public moral mobilization plus safer narrow-AI paths.
- •Open letters aim to raise attention, not directly stop labs
- •Experts were surprised by LLM capability; forecasts are uncertain
- •S-risk: futures of extreme suffering could be worse than extinction
- •Critique of Andreessen-style techno-optimism as uninformed on x-risk literature
- •Prescription: public personalization of risk and non-violent stigmatization; keep beneficial narrow AI while avoiding AGI race
