A Former AI Doomer Reveals a Dark Truth

1. 0:00 – 2:01

   A stark warning from a former AI doomer

   1. CNCal Newport0:00

      An argument that I've been making for a long time now is that if we really want to understand what's happening in AI, we have to understand the futurist ideologies that are heavily influencing the top frontier labs, Anthropic and OpenAI. The problem is, is these ideologies are confusing, and there's rumors swirling around them, and the excerpts we hear can sometimes be absurd, and we think, "That can't possibly be true." So I thought what would be useful is to bring in an insider to explain to us what is really happening in these labs and what do they truly believe. This is why I'm really happy to, to have on the show this week Emile Torres. Emile is a researcher and academic who focuses on eschatological and utopian movements. But what makes Emile really useful to this discussion is that they used to be involved in these ideologies. Emile comes from the world of futurist ideologies that believed that superintelligent AI would either kill us all or redeem humanity with a post-human digital God. But then Emile's moral compass led them out of that world, and now Emile has become, in my opinion, one of the most astute chroniclers, expositors, and critics of these ideologies. So Emile's gonna join me here in a second, and they're gonna walk us through what exactly are these different beliefs, how do they intersect, what is their role in the major frontier labs, and why should we be really worried about them. If you have any ambition to understand what's really happening in the world of AI, you need to listen to this conversation with Emile. It is going to, quite frankly, probably shock you, but this is a good shock. This message is one that we all need to hear. So let's get right into it. Emile, thank you for

2. 2:01 – 1:13:57

   Interview with Emile Torres

   1. CNCal Newport2:01

      joining me on the show.

   2. ETEmile Torres2:03

      Thanks for inviting me. Great to be here.

   3. CNCal Newport2:05

      Uh, I've been looking forward to this conversation for a long time. I think both your story and your research is being too overlooked in our current moment. I think it is very relevant to a lot of issues that people care about or are worried about or don't understand about the current AI situation. And so I'm excited to really get into both the meat of your work as well as the origin story, which I think is very important. But I, I thought we should start briefly with definitions just so that everyone is on the same page when we're talking about-

   4. ETEmile Torres2:36

      Yeah

   5. CNCal Newport2:36

      ... these various movements. So can you explain what the acronym TESCREAL means?

   6. ETEmile Torres2:43

      Yes, I can. So this was coined by myself during a collaboration with, uh, the computer scientist Timnit Gebru, uh, which we initiated in, I think, uh, late twenty twenty-two or so. And we were specifically interested in the underlying ideological drivers behind the current race to build ASI or artificial superintelligence. Um, so TESCREAL is, is a, a clunky, kind of unsightly term, but it does a lot of important explanatory work. So one way to think about the, the acronym is it's basically just a name for the dominant orthodox worldview within Silicon Valley that simultaneously highlights a certain intellectual history that is relevant to understanding that, that worldview. So it explains, uh, uh, what the worldview is, where it came from, what shaped it, how it evolved, uh, a-and so on. And so the, the acronym itself stands f- uh, it represents, uh, seven different ideologies. Um, and the, the, uh, letters appear in roughly the same order as the ideologies, uh, emerged chronologically over the past-

   7. CNCal Newport3:47

      Hmm

   8. ETEmile Torres3:47

      ... almost forty years. So the T stands for transhumanism, E for extropianism, then singularitarianism, cosmism, rationalism, effective altruism, and longtermism. A bunch of, like, really big, clunky, polysyllabic, uh, uh, terms. Um, but these ideologies are really fringe within American society, and I think e- you know, if you were to go and talk to the average, uh, person on the street, they would find them to be so bizarre that they'd consider, uh, the, the grand cosmic vision behind these ideologies as basically a non-starter. Nonetheless, they are massively influential within, uh, Silicon Valley. Like, basically everybody, you know, from Elon Musk to Mark Zuckerberg to Sam Altman to Dario Amodei are transhumanists. And many people in Silicon Valley, um, don't even, uh, call themselves transhumanists. Um, but that's for the same reason that I've never publicly stated I'm a round-Earther, [chuckles] right? It's just like what else would I be? Obviously, I'm that. So transhumanism really is the, the water that people in Silicon Valley swim in. It's the air that they breathe, and it's also the backbone of the, the whole TESCREAL, uh, worldview. And so once you understand transhumanism, then all of the other ideologies, some of which are just variants of transhumanism, are sort of easy to grasp.

   9. CNCal Newport5:02

      Well, I want to ask you a, a history question here because this is... some of this is new to me. The, the history of this that I best understand starts with the E. So the history I've been-

   10. ETEmile Torres5:11

       Yeah

   11. CNCal Newport5:11

       ... writing or, or talking about sort of starts with extropianism becoming a, a pretty big utopian futurist movement, I guess like the eighties and nineties.

   12. ETEmile Torres5:20

       Yeah.

   13. CNCal Newport5:20

       From there you get rationalism, um, sort of simultaneously effective altruism, which starts with a bunch of different goals, embraces-

   14. ETEmile Torres5:29

       Mm-hmm

   15. CNCal Newport5:29

       ... ideas from rationalism and longtermerism, and becomes more focused on a sort of, uh, more of a longtermerism, transhuman utopian, anti-AI type of, um, orientation. All right.

   16. ETEmile Torres5:42

       Yeah.

   17. CNCal Newport5:42

       So what am I-- what pieces am I missing in that really simplified history?

   18. ETEmile Torres5:47

       Um, no, that's, that's a pretty good overview. Um, so when, uh, Gebru and I are talking about transhumanism, we primarily mean modern transhumanism in its libertarian form. Um, that libertarian form, like the first organized, uh, t-trans-- libertarian transhumanist movement was the extropians, right? And so th-this-

   19. CNCal Newport6:06

       Okay

   20. ETEmile Torres6:06

       ... is a movement that Like you said, emerged in the late 1980s, early 1990s. Um, there was, you know, somewhat famously an extropian mailing list that brought together transhumanists from all around the world. It's where people like Eliezer Yudkowsky, one of the leading AI doomers out there, um, you know, was able to, uh, exchange ideas with people like Nick Bostrom, uh, who's another, uh, very important, uh, figure in the formation and development of the, the tescuro worldview. Um, so you're, you're absolutely right. And then, you know, the other ideologies, um, right after E in the acronym, like singularitarianism, this is basically just transhumanism, uh, libertarian transhumanism with a focus on the singularity. And at this point, I feel like most people listening probably know what the singularity is, right? That that wasn't the case like five years ago. Um, so, you know, a very quick, uh, definition. It's, um, it, it actually has several definitions, but the most common one these days is an intelligence explosion, right? Triggered by recursive self-improvement. So you get an AI system that's just, uh, clever enough or smart enough to develop the next generation of AI systems, and you get this positive feedback loop and suddenly through some kind of exponential or hyperbolic, uh, explosion of capabilities, you end up with, uh, a, a superintelligence. And then beyond that, you go out and colonize the entire universe, plunder the vast resources of the cosmos, and create this, you know, sprawling multi-galactic utopia a-among the stars. Um, and cosmism, the, the next letter, is, is just transhumanism that, um, emphasizes the aspect of this worldview that I, I just, uh, articulated, um, which is... It's basically the idea of like, well, why stop at just radically re-engineering the human organism, which is the central goal of, uh, transhumanism to create a new post-human species? Let's go out and create technologies that enable, enable us to re-engineer the entire universe. You know, whole, whole galaxies maybe intervening on the, you know, fundamental fabric of space time, uh, to manipulate the world in ways that, that we find, uh, desirable. So it's kind of just like transhumanism with this like broader cosmic, uh, vision. And, uh, y- and then rationalism, um, was founded by Yudkowsky, who was a transhumanist extropian, um, singularitarian who hired the leading, uh, modern cosmists named Ben Gr- namely Ben Goertzel, who worked at the, uh, Machine Intelligence Research Institute, which is originally the Singularity Institute founded by, uh, Yudkowsky. So rationalism like emerged directly out of, uh, this movement. And basically, the idea was, okay, if we're going to establish a cosmic post-human utopia at some point, we're gonna need to develop these advanced technologies. There's no other way around that, e-especially superintelligence. But to develop them in the right way such that these technologies do not destroy us, but rather enable us to, uh, establish this, you know, sprawling utopia, uh, heaven among the literal heavens, um, we're gonna need to optimize our rationality. So that is essentially the, the main thrust behind rationalism. And effective altruism, as you, you mentioned, I mean, it, it, uh, emerged basically from rationalist blogs like Overcoming Bias and, uh, Less Wrong. And the focus there is not so much optimizing our rationality, but rather optimizing our morality. And, and although the story that effective altruists like to tell is that initially they were focused primarily on global poverty and then animal welfare, and then this other cause area called longtermism, uh, came into the picture later on. Really, the effective altruists from the very origins of this movement were focused on longtermism, right? So longtermism is, is basically just cosmism, except it has a more explicit ethical foundation to it. And that ethical foundation, I know there are a lot of isms that I've mentioned so far, but that ethical foundation is utilitarianism, specifically total or totalist utilitarianism. And that just says that the... We have an obligation to increase the total amount of value in the universe, um, across cosmic space and time. How do we do that? Well, one way is if you have a fixed population, it's to increase the well-being, uh, or welfare of the entities that exist in that population. The other possibility is to just increase the population, right? So if you double the population and everything else is the same, you know, ceteris paribus, then you've doubled the amount of value. So from the, the longtermist view, uh, you know, cosmism basically-- [chuckles] like one way to think about it is it's just like, wouldn't it be really cool if we as transhumanists radically re-engineered ourselves, became post-human, you know, uploaded our minds to the cloud, went out and colonized the universe? Wouldn't that just be like a, a really awesome thing to do? Longtermism goes a step further and says, "Actually, no, we have a kind of ethical obligation to, uh, to do this." As, as Will MacAskill says in his 2022 book, uh, What We Owe the Future, the upshot is a moral case for space settlement, right? And so you could, you could maybe see why Elon Musk has called longtermism, quote, "a close match for my philosophy." Right?

   21. CNCal Newport11:08

       Yeah.

   22. ETEmile Torres11:08

       So that's, that's, uh, in a little bit more detail, just fleshing out some of the details, uh, of your, um, very good, uh, first pass history. [chuckles]

   23. CNCal Newport11:18

       So, so it sounds like the, the tricky part but both the simple part about understanding this bundle, the, the, use the term that you and Timnit often use, um, is there's these little differences which can be confusing to people, where the-

   24. ETEmile Torres11:31

       Mm-hmm

   25. CNCal Newport11:31

       ... the, you know, Yudkowsky and the rationalists would be-- they were dismissive of a Kurzweil style singularitarianism because it was too optimistic and he was more negative.

   26. ETEmile Torres11:41

       Mm-hmm.

   27. CNCal Newport11:41

       And the effective altruist are-- they're doing more like of these explicit utilitarian expected value computations where the ration- You know, so there's these little differences, and they don't-

   28. ETEmile Torres11:51

       Yeah

   29. CNCal Newport11:51

       ... they're not all the same group, and yet they're really knit together by s- fundamental ideas about transhumanist utopia, avoiding extinction, and technology being delivered. But what I could never figure out, and maybe you know, what were the cultural or technical forces that really made the, the Transhumanist foundation emerge in the '70s and '80s and give birth to this movement that ended up essentially taking over Silicon Valley?

   30. ETEmile Torres12:18

       May I comment briefly on the remarks you just made before I answer that-
3. 1:13:57 – 1:17:49

   Cal recaps the interview with Emile

   1. CNCal Newport1:13:57

      with Emile Torres. That was really useful. Look, I've been writing about these ideologies for a little while now. I had a big New York Times article. I've done a bunch of podcast interviews where I've given my best rendition. Even last week's Thursday episode of this show got into some of those details, but I think what I learned today that wasn't as clear to me before was just how extinctionist all these beliefs are. I think Emile is very good on this point, that the goal, the ultimate goal of all of these teskere bundle of ideologies is a, a post-human utopia where biological humans are gone, and what we're left with are these exalted digital angels, these sort of digital post-human versions of humanity. But what, as Emile points out, that means at some point, these, our tech overlords who are trying to bring this, this future to fruition are gonna have to kill all the other, the meat, biological people. Because most biological people in a world of 8 billion people don't wanna say, "I'll just disappear, and we can have a select few go off into a transhumanist future." They're like, "We like our world as we have it." So if you really pull the thread, this was new to me, and, uh, quite frankly, it's, it's, uh, disconcerting, but I think Emile's right about this. If you pull the thread on the teskere bundle of ideologies, you end up at some point killing billions of people on purpose because they're getting in the way of your dream of a digital utopia. All right? The other key point that came out of this that we both emphasized, neither Emile or myself think that this vision is actually possible, that you actually are gonna, if you have a good enough agent on your LLM or something, you're gonna end up with a super intelligent god. There's a lot of good computer science reasons to say that's kind of fanciful. But Emile and me both agree on this point: pursuing utopia can be devastating. That's what we've seen. The Nazis did not get the thousand-year Reich, but tens of millions of people died. The, the Soviet communists did not get their, their socialist utopia, but tens of millions of people died on the way. So we've learned this lesson well. The pursuit of utopias where you're saying the value of humans alive right now pale in comparison to the value of the utopia that I am pursuing, we have seen that exact configuration throughout history. No one has ever successfully got to their utopia. Every time it's been pursued with any sort of well-funded vigor, it's been a blight on humanity's history. So we really need to be woke to these ideologies and what's really happening here. I am a fan of AI. I think generative AI can produce any number of useful tools, and the whole thing is being sullied and hijacked and made into this unstable dynamite bundle of terribleness by this weird utopian, anti-humanist, pretty dark philosophy that's incredibly well-funded, and the two top frontier labs are completely driven by it. So I think this is a very important message. It needs to be out there. This has to be a part of the conversation we're having when we're trying to understand the AI moment, but also when we're trying to understand what to do next. So, uh, Emile, thank you for coming on the show. Hopefully, you enjoyed this. Uh, see you again next time. Hey, if you made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at calnewport.com/ideas, and when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today, calnewport.com/ideas. [outro music]

Episode duration: 1:17:49