Dwarkesh Podcast

Joseph Carlsmith - Utopia, AI, & Infinite Ethics

Joseph Carlsmith is a senior research analyst at Open Philanthropy and a doctoral student in philosophy at the University of Oxford. Learn about Joseph and read his work at: https://www.josephcarlsmith.com/ Podcast website: https://www.dwarkeshpatel.com/p/joseph-carlsmith Apple Podcasts: https://apple.co/3Qcboq0 Spotify: https://spoti.fi/3zrElYb Follow me: https://twitter.com/dwarkesh_sp Follow Joseph: https://twitter.com/jkcarlsmith Timestamps: 00:00:00 Preview 00:00:55 Introduction 00:03:42 How to define a better future? 00:10:08 Utopia 00:26:01 Robin Hanson’s EMs 00:28:24 Human Computational Capacity 00:35:04 FLOPS to emulate human cognition? 00:41:04 Infinite Ethics 01:01:40 SIA vs SSA 01:18:42 Futurism & Unreality 01:24:25 Blogging & Productivity 01:29:32 Book Recommendations 01:30:53 Conclusion

Joseph (Joe) CarlsmithguestDwarkesh Patelhost

Aug 2, 20221h 32mWatch on YouTube ↗

WHAT IT’S REALLY ABOUT

Joseph Carlsmith explores utopia, AI risk, and infinite ethics

Joseph Carlsmith, a philosopher and AI x‑risk researcher at Open Philanthropy, discusses how seriously we should take the possibility of a radically better future (utopia), and the risks posed by transformative AI. He argues that utopia should be seen as a profoundly better but still concrete, finite world, likely reachable only after a long process of civilizational wisdom and coordination, not a simple hedonistic tiling of the universe.
He explains his work estimating the computational power of the human brain as an input to AI timelines, stressing large uncertainties both in neuroscience and in how current machine learning might scale to human‑level intelligence. The conversation then turns to “infinite ethics” and anthropics: how to reason morally and probabilistically when the universe, or our influence within it, might be infinite.
Carlsmith defends taking low‑probability, high‑stakes possibilities (like infinite impact or insect suffering) seriously without letting them completely derail practical decision‑making, emphasizing epistemic humility, moral uncertainty, and the goal of preserving options so a wiser future civilization can handle the hardest questions. He also reflects on why much futurism feels abstract or unreal, and on his own writing practice and influences.

IDEAS WORTH REMEMBERING

5 ideas

Treat utopia as a real, finite possibility, not mystical perfection.

Carlsmith defines utopia as a ‘profoundly better’ but still resource‑constrained world, arguing we systematically underestimate how much better things could be and should relate to utopia as a concrete, achievable target rather than an abstract fantasy.

Plan for transformative AI under deep uncertainty about timelines and methods.

His work uses neuroscience‑based estimates of brain compute as an anchor for AI timelines, but he stresses large uncertainty—both about how much compute is needed and whether current deep learning paradigms will scale—implying we must prepare for both sooner‑than‑expected and later‑than‑expected breakthroughs.

Handle utopian ambition with caution, not denial.

Utopian thinking has historically fueled dangerous ideologies, but Carlsmith argues the solution is not to ignore the possibility of a much better world, but to monitor rigidity, fanaticism, and willingness to ‘break things’ in pursuit of moral visions.

Preserve options so a wiser future can address infinite‑ethics puzzles.

Because infinite worlds and acausal impacts can dominate expected‑value reasoning yet remain philosophically unsettled, he favors focusing now on survival, wisdom‑growth, and option‑preservation so future, more capable agents can make better‑informed choices.

Acknowledge moral uncertainty about animals and insects without paralysis.

He thinks it’s unreasonable to assign zero moral weight to creatures like ants, but instead of adopting extreme Jain‑like behavior, he advocates recognizing the trade‑offs, accepting residual risk, and taking responsibility for one’s chosen level of concern.

WORDS WORTH SAVING

5 quotes

“Utopia for me just means a kind of profoundly better future… something that we could do… a world that is radically better than the world we live in today.”

— Joseph Carlsmith

“The future is a big thing to try to model with this tiny mind, and so, of necessity, you need to use these extremely lossy abstractions.”

— Joseph Carlsmith

“I don’t sit around thinking that we sort of know what utopia is right now, and it’s hedonium… I really don’t assume that that’s what utopia is about at all.”

— Joseph Carlsmith

“There’s a middle ground between ‘I shall ignore this completely’ and ‘I shall be a Jain,’ which is recognizing that this is a real trade‑off, there’s uncertainty here, and taking responsibility for how you’re responding to that.”

— Joseph Carlsmith

“I think the right path forward is to survive long enough for our civilization to become much wiser… and then to use that position of wisdom and empowerment to act better with respect to these issues.”

— Joseph Carlsmith

Nature and importance of utopia as a radically better, yet finite futureAI timelines, brain compute estimates, and risks from transformative AIDangers and benefits of utopian thinking within effective altruism and longtermismInfinite ethics: moral reasoning under infinite worlds and infinite impactAnthropic reasoning: self‑indication vs self‑sampling assumptions and the doomsday argumentMoral uncertainty about non‑human animals and insect sufferingThe limits of futurism, abstraction vs concreteness, and Carlsmith’s writing process

High quality AI-generated summary created from speaker-labeled transcript.

Get more out of YouTube videos.

High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.