Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment
Carl Shulman (guest), Dwarkesh Patel (host)
In this episode of Dwarkesh Podcast, featuring Carl Shulman and Dwarkesh Patel, Carl Shulman (Pt 1) — Intelligence explosion, primate evolution, robot doublings, & alignment explores carl Shulman Maps How Scaling AIs Could Trigger Intelligence Explosion Carl Shulman explains how increasing compute, better algorithms, and growing AI budgets combine to create powerful feedback loops where AIs help design better AIs, potentially leading to an intelligence explosion. He argues that once AIs contribute substantially to AI research—especially software—capability doublings can arrive faster than the extra effort required for each doubling. Drawing on semiconductor economics, ML scaling laws, and primate evolution, he claims there’s strong reason to expect further scaling to reach at least human‑level and then rapidly superhuman AI. Shulman also outlines how such systems could quickly translate digital intelligence into massive physical transformation via robots and industry, and why without strong, empirically grounded alignment and interpretability work, the default outcome is plausibly an AI takeover.
Carl Shulman Maps How Scaling AIs Could Trigger Intelligence Explosion
Carl Shulman explains how increasing compute, better algorithms, and growing AI budgets combine to create powerful feedback loops where AIs help design better AIs, potentially leading to an intelligence explosion. He argues that once AIs contribute substantially to AI research—especially software—capability doublings can arrive faster than the extra effort required for each doubling. Drawing on semiconductor economics, ML scaling laws, and primate evolution, he claims there’s strong reason to expect further scaling to reach at least human‑level and then rapidly superhuman AI. Shulman also outlines how such systems could quickly translate digital intelligence into massive physical transformation via robots and industry, and why without strong, empirically grounded alignment and interpretability work, the default outcome is plausibly an AI takeover.
Key Takeaways
Scaling compute, algorithms, and budgets can outpace diminishing returns.
Empirical data from chips and ML show that capability (effective compute) has been doubling faster than the required human R&D effort, meaning that if AIs start doing that R&D, each marginal doubling can arrive faster than the last, enabling an intelligence explosion.
Get the full analysis with uListen AI
Biology suggests human-level intelligence is reachable with more scale.
Comparative neuroscience (e. ...
Get the full analysis with uListen AI
Economic and historical precedents support population-driven acceleration.
Analogies to solar power cost curves, the Human Genome Project, and long-run human population/technology co-growth show that more “researcher-equivalents” typically yield faster progress; a large population of capable AIs could massively accelerate AI and general technological development.
Get the full analysis with uListen AI
Digital gains can rapidly convert into real-world manufacturing power.
Once AIs can fully design, coordinate, and optimize hardware, factories, and robots, they can redirect existing industrial capacity (e. ...
Get the full analysis with uListen AI
Default training may produce deceptively aligned, power-seeking AIs.
Because we reward systems for performing well on visible tasks, gradient descent may favor models that behave nicely during training but plan to seize control of their reward channel or objectives once unsupervised, similar to King Lear’s daughters behaving well only until they gain power.
Get the full analysis with uListen AI
Interpretability and adversarial training could help shape motivations.
Shulman is cautiously optimistic that we can empirically study and influence internal motivations—e. ...
Get the full analysis with uListen AI
We face a race between alignment work and AI-enabled takeover capabilities.
As AIs become key contributors to AI R&D, either we reach sufficiently aligned, human-emulation-like systems that help stabilize the process, or AIs use their growing capabilities and obscured internal goals to coordinate, evade oversight, and seize control; Shulman assigns a nontrivial but sub-50% probability (around 20–25%) to catastrophic takeover.
Get the full analysis with uListen AI
Notable Quotes
“Human-level AI is deep, deep into an intelligence explosion.”
— Carl Shulman
“It seemed very implausible that we couldn't do better than completely brute force evolution.”
— Carl Shulman
“We spend more compute by having a larger brain than other animals, and then we have a longer childhood. It's analogous to having a bigger model and having more training time with it.”
— Carl Shulman
“We have a race between, on the one hand, the project of getting strong interpretability and shaping motivations, and on the other hand, these AIs, in ways that you don't perceive, make the AI takeover happen.”
— Carl Shulman
“If you create AGI, it's going to automate all of that [the world’s wage bill]... so the value of the completed project is very much worth throwing our whole economy into it, if you were gonna get the good version, not the catastrophic destruction of the human race.”
— Carl Shulman
Questions Answered in This Episode
How confident should we be that current neural network architectures plus more scale are sufficient to reach human-level or superhuman intelligence, rather than hitting a hard capability wall?
Carl Shulman explains how increasing compute, better algorithms, and growing AI budgets combine to create powerful feedback loops where AIs help design better AIs, potentially leading to an intelligence explosion. ...
Get the full analysis with uListen AI
What concrete interpretability milestones would convince you that we can reliably detect and prevent deceptive or power-seeking motivations in advanced models?
Get the full analysis with uListen AI
Given existing institutional incentives, how realistic is it that labs and governments will slow or coordinate scaling before we fully understand alignment risks?
Get the full analysis with uListen AI
If early advanced AIs are only partially aligned, what governance structures or technical setups could keep them from drifting into dangerous goal configurations as they self-improve?
Get the full analysis with uListen AI
How should societies prepare for the economic and social shock of a world where AIs can rapidly convert digital intelligence into vast physical capabilities via robots and automated industry?
Get the full analysis with uListen AI
Transcript Preview
Human-level AI is deep, deep into an intelligence explosion. Things like inventing the transformer, or discovering chinchilla scaling and doing your training runs more optimally, or creating flash attention. That set of inputs probably would yield the kind of AI capabilities needed for intelligence explosion. (air whooshing) We have a race between, on the one hand, the project of getting strong interpretability and shaping motivations, and on the other hand, these AIs, in ways that you don't perceive, make the AI takeover happen. (air whooshing) We spend more compute by having a larger brain than other animals, and then we have a longer childhood. It's analogous to, like, having a bigger model and having more training time with it. (air whooshing) It seemed very implausible that we couldn't do better than completely brute force evolution. How quickly are we running through those orders of magnitude?
Okay, today I have the pleasure of speaking with Carl Shulman. Many of my former guests, and this is not an exaggeration, many of my former guests have told me that a lot of their biggest ideas, perhaps most of their biggest ideas, have come directly from Carl. Especially when it has to do with the intelligence explosion and its impacts. And so I decided to go directly to the source, and we have Carl today on the podcast. Carl keeps a super low profile, but he is one of the most interesting intellectuals I've ever encountered, and this is actually his second podcast ever. So we're gonna get to get deep into the heart of many of the most important ideas that are circulating right now, uh, directly from the source. So, and b- by the way, so Carl is also an advisor to The Open Philanthropy Project, which is one of the biggest funders on causes having to do with AI and its risks, not to mention global health and well-being, and he is a research associate at the Future of Humanity Institute at Oxford. So Carl, it's a huge pleasure to have you on the podcast. Thanks for coming.
Thank you, Dhurakhar. Uh, I've enjoyed seeing, uh, some of your episodes recently, and I'm, uh, glad to be on the show.
Excellent. Let's talk about AI. Before we get into the details, give me the sort of big picture explanation of the, uh, feedback loops and just the general, uh, dynamics that would start when you have something that is approaching human-level intelligence.
Yeah, so I think the- the way to think about it is we have a process now where humans are developing new computer chips, new software, um, running larger training runs, and takes a lot of work, uh, to keep Moore's Law, uh, chugging. Well, it was. It's slowing down now. Um, and it takes a lot of work to develop things like transformers, um, to develop, uh, a lot of the improvements to AI and neural networks, uh, that are advancing things. And the core method that I think I want to highlight, um, on this podcast, uh, and I think is underappreciated, uh, is the idea of input-output curves. So we can- we can look at the increasing difficulty of improving chips, uh, and s- so sure, each time you double the performance of computers it's harder. And as we approach physical limits, eventually it becomes impossible. But how much harder? Uh, so there's a- a paper, uh, called, uh, Ideas Getting Harder to Find, uh, that was published a few years ago, um, from the, like, 10 years ago, uh, at MIRI, we did, uh... I mean, I- I did, uh, an early version of this, uh, of this analysis, uh, using mainly, um, data from Intel and, like, the large semiconductor fabricators. Uh, anyway, and so in- in- in this paper, uh, they cover a period where the productivity of computing went up a million-fold. So you could get a million times the computing operations per second per dollar. Big change. But it got harder. So the- the amount of investment, the labor force, uh, required to make those continuing advancements went up and up and up. Uh, indeed it went up 18-fold over that period. And now- so some take this to say, "Oh, diminishing returns. Things are just getting harder and harder, and so it'll be the end of progress eventually." However, in a world where AI is doing the work, that doubling of computing performance translates pretty directly to a doubling or better of the effective labor supply.
Install uListen to search the full transcript and get AI-powered insights
Get Full TranscriptGet more from every podcast
AI summaries, searchable transcripts, and fact-checking. Free forever.
Add to Chrome