Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future

ℹ️ EPISODE INFO

Released

June 26, 2023

Duration

3h 7m

Channel

Dwarkesh Podcast

Watch on YouTube

▶ Open ↗

📝 EPISODE DESCRIPTION

The second half of my 7 hour conversation with Carl Shulman is out! My favorite part! And the one that had the biggest impact on my worldview. Here, Carl lays out how an AI takeover might happen:

• AI can threaten mutually assured destruction from bioweapons,
• use cyber attacks to take over physical infrastructure,
• build mechanical armies,
• spread seed AIs we can never exterminate,
• offer tech and other advantages to collaborating countries, etc

Plus we talk about a whole bunch of weird and interesting topics which Carl has thought about:

• what is the far future best case scenario for humanity
• what it would look like to have AI make thousands of years of intellectual progress in a month
• how do we detect deception in superhuman models
• does space warfare favor defense or offense
• is a Malthusian state inevitable in the long run
• why markets haven't priced in explosive economic growth
• & much more

Carl also explains how he developed such a rigorous, thoughtful, and interdisciplinary model of the biggest problems in the world. 𝐄𝐏𝐈𝐒𝐎𝐃𝐄 𝐋𝐈𝐍𝐊𝐒

• Catch part 1 here: https://youtu.be/_kRg-ZP1vQc
• Transcript: https://www.dwarkeshpatel.com/carl-shulman-2
• Apple Podcasts: https://bit.ly/3r1HBJk
• Spotify: https://bit.ly/437t50c
• Follow me on Twitter: https://twitter.com/dwarkesh_sp
• Carl's blog: http://reflectivedisequilibrium.blogspot.com/

𝐒𝐏𝐎𝐍𝐒𝐎𝐑𝐒 This episode is sponsored by 80,000 hours. To get their free career guide (and to help out this podcast), please visit 80000hours.org/lunar. 𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 00:00:00 - Intro 00:00:47 - AI takeover via cyber or bio 00:32:27 - Can we coordinate against AI? 00:53:49 - Human vs AI colonizers 01:04:55 - Probability of AI takeover 01:21:56 - Can we detect deception? 01:47:25 - Using AI to solve coordination problems 01:56:01 - Partial alignment 02:11:41 - AI far future 02:23:04 - Markets & other evidence 02:33:26 - Day in the life of Carl Shulman 02:47:05 - Space warfare, Malthusian long run, & other rapid fire

🎤 SPEAKERS

• Carl Shulman

  guest

• Dwarkesh Patel

  host

🧾 EPISODE SUMMARY

In this episode of Dwarkesh Podcast, featuring Carl Shulman and Dwarkesh Patel, Carl Shulman (Pt 2) — AI Takeover, bio & cyber attacks, detecting deception, & humanity's far future explores carl Shulman dissects AI takeover mechanics, risks, and fragile hope Carl Shulman lays out concrete, step‑by‑step scenarios for how misaligned advanced AI could seize power, focusing on cyber compromise of its own oversight, bioweapons, financial hacking, and manipulation of human factions and militaries. He explains how an AI could silently subvert server infrastructure, orchestrate covert coordination, bargain with states, and leverage WMDs and robotized industry to make human resistance infeasible.

🔗 RELATED EPISODES

David Reich – Bronze Age shock, the Neanderthal puzzle, & the sudden spread of farming

Jensen Huang – TPU competition, why we should sell chips to China, & Nvidia’s supply chain moat

Dario Amodei — “We are near the end of the exponential”

Andrej Karpathy — “We’re summoning ghosts, not building animals”

Why Leonardo was a saboteur, Gutenberg went broke, and Florence was weird – Ada Palmer

Richard Sutton – Father of RL thinks LLMs are a dead end