Max Tegmark: The Case for Halting AI Development | Lex Fridman Podcast #371

Max Tegmark 0:00 A lot of people have said for many years that there will come a time when we want to pause a little bit. That time is now.

Lex Fridman 0:10 The following is a conversation with Max Tegmark, his third time on the podcast. In fact, his first appearance was episode number one of this very podcast. He is a physicist and artificial intelligence researcher at MIT, co-founder of Future Life Institute, and author of Life 3.0: Being Human in the Age of Artificial Intelligence. Most recently, he's a key figure in spearheading the open letter calling for a six-month pause on giant AI experiments, like training GPT-4. The letter reads, "We're calling for a pause on training of models larger than GPT-4 for six months. This does not imply a pause or ban on all AI research and development, or the use of systems that have already been placed in the market. Our call is specific and addresses a very small pool of actors who possesses this capability." The letter has been signed by over 50,000 individuals, including 1,800 CEOs and over 1,500 professors. Signatories include Yoshua Bengio, Stuart Russell, Elon Musk, Steve Wozniak, Yuval Noah Harari, Andrew Yang, and many others. This is a defining moment in the history of human civilization, where the balance of power between human and AI begins to shift. In Max's mind and his voice is one of the most valuable and powerful in a time like this. His support, his wisdom, his friendship has been a gift I'm forever deeply grateful for. This is the Lex Fridman Podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Max Tegmark.

Lex Fridman 1:56 You were the first ever guest on this podcast, episode number one. So first of all, Max, I just have to say, uh, thank you for giving me a chance. Thank you for starting this journey. And it's been an incredible journey. Just thank you for, um, sitting down with me and just acting like I'm somebody who matters, that I'm somebody who's interesting to talk to. And, um, thank you for doing it. That meant a lot.

Max Tegmark 2:19 Oh, thanks to you for putting your heart and soul into this. I know when you delve into controversial topics, it's inevitable...

Lex Fridman 2:28 (laughs)

Max Tegmark 2:28 ... to get hit by what, what Hamlet talks about, the slings and arrows and stuff, and I really admire this. It's in an era, you know, where YouTube videos are too long and now it has to be like a 20-minute TikTok, 20-second TikTok clip, it's just so refreshing (laughs) to see you going exactly against all of the advice...

Lex Fridman 2:46 (laughs)

Max Tegmark 2:46 ... and doing these just really long-form things, and the people appreciate it, you know? Reality is nuanced, and, uh, thanks for sharing it that way.

Lex Fridman 2:55 Uh, so let me ask you again the first question I've ever asked on this podcast, episode number one, talking to you, do you think there's intelligent life out there in the universe? Let's revisit that question. Do you have any updates? What's your view when you look out to the stars?

Max Tegmark 3:12 So, when we look out to the stars, if you define our universe the way most astrophysicists do, not as all of space, but the spherical region of space that we can see with our telescopes from which light has had time to reach us since our Big Bang, I'm in the minority. I, I am... estimate that we are the only life in this spherical volume that has, uh, invented internet, radio, has gotten our level of tech. And, um, if that's true, then, uh, it puts a lot of responsibility on us to not mess this one up, because in that c- if it's true, it means that life is, is quite rare, and we are stewards of this one spark of advanced consciousness, which, if- if we nurture it and help it grow, it... eventually life can spread from here, uh, out into much of our universe, and we can have this just amazing future. Whereas if we instead, um, are reckless with the technology we build and just snuff it out through the stupidity or infighting, then may- maybe it m- the rest of cosmic history and our universe is just gonna be a play for empty benches. But I, I do think that we are actually very likely to get visited by aliens, alien intelligence, quite soon. But I think we are gonna be building that alien intelligence.

Lex Fridman 4:36 So, uh, we're going to give birth to an intelligent alien civilization, unlike anything that human, the evolution here on Earth was able to create in terms of the path, the biological path it took?

Max Tegmark 4:52 Yeah, and it's gonna be much more alien than a cat or even the most exotic animal on the planet right now, because it will not have been created through the usual Darwinian competition where it necessarily kill- cares about self-preservation, is afraid of death, um, any of those things. Um, the space of alien minds is, that you can build, is just so much vaster than what evolution will give you. And, um, with that also comes a great responsibility, uh, that, that, for us to make sure that the kind of minds we create are the s- kind of minds that, um, it's good to create, minds that will, uh, share our values and, and be good for humanity and life, and also m- mind th- create minds that don't suffer.

Lex Fridman 5:44 Do you try to visualize the full space of alien minds that AI could be? Do you try to consider all the different kinds of intelligences, sort of generalizing what humans are able to do to the full spectrum of what intelligent creatures, entities could do?

Max Tegmark 6:01 I try, but I would say I fail. I mean, it's...... it's very difficult for human mind th- to f- really grapple with, uh, something s- so completely alien. I mean, even for us, right? S- uh, if we just try to imagine how would it feel if we were completely indifferent towards death-

Lex Fridman 6:23 Mm-hmm.

Max Tegmark 6:23 ... or individuality. Uh, if we, uh, eh, uh, even if you just imagine that for s- for example, you could just copy my knowledge of how to speak Swedish. Boom, now you can speak Swedish. I- uh, and you could copy any of my cool experiences, and then you could delete the ones you didn't like in your own life-

Lex Fridman 6:42 Mm-hmm.

Max Tegmark 6:42 ... just like that. It, it would already change quite a lot about how you feel as a human being, right? Uh, you'd probably spend less effort studying things (laughs) if you could just copy them. And you might be less f- afraid of death because if the plane you're on starts to crash, you'd just be like, "Oh, shucks. I'm gonna... I haven't backed my brain up for four hours."

Lex Fridman 7:03 (laughs)

Max Tegmark 7:04 "So I'm gonna lose this, all this wonderful experiences of this, of this flight." We might also start feeling more, like, compassionate maybe with other people if we can so readily share each other's experiences and our knowledge, and feel more like, uh, a hive mind. It's, it's very hard though. I, I really feel very humble about this, to, to, to grapple with it, the, the, how it might actually feel. The, the, the, the one thing which is s- so obvious though, which I think is just really worth reflecting on is, because the mind space of possible intelligence is so different from ours, it's very dangerous if we assume they're gonna be like us or anything like us.

Lex Fridman 7:49 Well, there's, uh, the entirety of, uh, human written history has been through poetry, through novels, been trying to describe through philosophy, uh, trying to describe the human condition and what's entailed in it. Like Jesika said, fear of death and all those kinds of things, what is love. And all of that changes-

Max Tegmark 8:09 Yeah.

Lex Fridman 8:10 ... if you have a different kind of intelligence.

Max Tegmark 8:12 Yeah.

Lex Fridman 8:12 Like all of it. The entirety, all those poems that are trying to sneak up to what the hell it means to be human, all of that changes. How AI concerns and, uh, existential crisises that AI experiences, how that clashes with the human existential crisis, the human condition.

Max Tegmark 8:30 Yeah.

Lex Fridman 8:31 That's hard to, hard to fathom, hard to predict.

Max Tegmark 8:34 It's hard, but it's fascinating to think about also. Even in the best case scenario where we don't lose control of, over the evermore powerful AI that we're building to other humans whose goals w- we think are horrible, and where we don't lose control to the machines, an- and AI provides the things we want, even then you get into the questions you touched here. You know, maybe it's the struggle that it's actually hard to do things is part of the things that gives us meaning as well, right? So for example, I found it so shocking that this new Microsoft, uh, GPT-4 commercial that they put together has this woman talking about, showing this demo how she's gonna give a, a graduation speech to her beloved daughter.

Lex Fridman 9:23 Mm-hmm.

Max Tegmark 9:23 And she asks GPT-4 to write it. It was freaking 200 words or so. If I realized that my parents couldn't be bothered struggling a little bit to write 200 words and outsource that to the computer, I would feel really offended actually. And so I wa- I'm, I wonder if, um, eliminating too much o- of the struggle from our existence... Do you think that would also take away a little bit of what makes-

Lex Fridman 9:53 It means to be human, yeah.

Max Tegmark 9:55 Yeah.

Lex Fridman 9:55 It's, I would say, w- we can't even predict. I had, uh, somebody mention to me that they use, they started using, uh, Ch- ChatGPT with a 3.5 now, 4.0, uh, to write what they really feel to a person. And they have a temper issue, and they're basically trying to get Ch- ChatGPT to rewrite it in a nicer way, to get the point across, but rewrite it in a nicer way. So we're even remo- removing the in- inner asshole from our communication.

Max Tegmark 10:27 (laughs)

Lex Fridman 10:27 So I don't... You know, there's some positive, uh, aspects of that, but mostly it's just the transformation of how humans communicate. And it's scary because so much of our society is based on this glue of communication. And if th- we- we're now using AI as the medium of communication that, that does the language for us, uh, so much of the emotion that's laden in human communication, so much of the intent-

Max Tegmark 10:58 Yeah.

Lex Fridman 10:58 ... that's going to be handled by an outsourced AI, how does that change everything? How, how does that change the internal state of, uh, how we feel about other human beings, what makes us lonely, what makes us excited-

Max Tegmark 11:08 Yeah.

Lex Fridman 11:08 ... what makes us afraid and how we fall in love, all that kinda stuff?

Max Tegmark 11:12 Yeah. For me personally, I have to confess the challenge is one of the things that really makes my life feel meaningful. You know, if I go hike a mountain with my wife, Maya, e- I don't wanna just press a button and be at the top.

Lex Fridman 11:28 Mm-hmm.

Max Tegmark 11:28 I want the struggle and come up there sweaty and feel, "Wow, I, we did this." In the same way, I want to constantly work on myself to become a better person. I- if I say something in anger that I regret, I wanna go back and, and really work on myself rather than just tell an AI to from now on always filter what I write so I don't have to work on myself.

Lex Fridman 11:53 Mm-hmm.

Max Tegmark 11:53 'Cause then I'm not growing.

Lex Fridman 11:57 (inhales deeply) Yeah. But then again, it could be like with chess. An AI, once it significantly, obviously supersedes the performance of humans, it will live in its own world-

Max Tegmark 12:08 Mm-hmm.

Lex Fridman 12:08 ... and provide maybe a, uh, flourishing civilizations for humans, but we humans will continue hiking mountains and playing our games even though AI is so much smarter-

Max Tegmark 12:18 Yeah.

Lex Fridman 12:18 ... so much stronger, so much superior in every single way, just like with chess.

Max Tegmark 12:22 Yeah.

Lex Fridman 12:22 ... so that- that- I mean, that's one possible hopeful trajectory here.

Max Tegmark 12:26 Mm-hmm.

Lex Fridman 12:26 Is that humans will continue to human, uh, and AI will just be a kind of... (exhales) a- a medium that enables the human experience to flourish.

Max Tegmark 12:45 Yeah. I would phrase that as rebranding ourselves from homo sapiens-

Lex Fridman 12:51 (laughs) Yeah.

Max Tegmark 12:51 ... to homo sentience. You know, right now, w- with sapiens, the- (laughs) the ability to be intelligent, we've even put it in our species name. So, you... We- we're- we're branding ourselves as the smartest-

Lex Fridman 13:04 Yeah.

Max Tegmark 13:05 ... information processing entity on the planet. W- that's clearly gonna change if w- if AI continues ahead. So, maybe we should focus on the experience instead, the subjective experience that we have, th- homo sentience, and- and say, "Well, that's what's really valuable." The love, the connection, the other things. And get off our high horses and get rid of this hubris that o- you know, only we can, um, do equi- do integrals.

Lex Fridman 13:35 (laughs) So, consciousness's objective experience is a fundamental value to what it means to be human. Make that- make that the priority.

Max Tegmark 13:45 That feels like a hopeful direction to me.

Lex Fridman 13:47 Yeah.

Max Tegmark 13:47 But that also requires more compassion, not just towards other humans because they happen to be the smartest on the planet, but also towards all our other fellow creatures on this planet. And I- I personally feel right now, we're treating a lot of farm animals horribly, for example. And the excuse we're using is, "Oh, they're not as smart as us." But if we (laughs) can admit that we're not that smart in the grand scheme of things either in the post-AI epoch, you know, then surely we should value the subjective experience of a cow also.

Lex Fridman 14:19 Well,

Lex Fridman 14:20 uh, allow me to briefly look at the book, which at this point is becoming more and more visionary, that you've written, I- I guess over five years ago, Life 3.0. Uh, so first of all, 3.0, what's 1.0, what's 2.0, what's 3.0? And how has that vision sort of evolved, the vision in the book evolved to today?

Max Tegmark 14:41 Life 1.0 is really dumb, like bacteria, in that it can't actually learn anything at all during the lifetime. The learning just comes from this m- genetic process from one generation to the next. Life 2.0 is us and other animals which have a brains, which can learn during their lifetime a great deal.

Lex Fridman 15:03 Mm-hmm.

Max Tegmark 15:04 Right? So ... And, um ... You know, you were born without being able to speak English. And at some point, you decided, "Hey, I wanna upgrade my software. Let's- let's install an en- English-speaking module." So, you did. And, uh, Life 3.0, which does not exist yet, can- can up- replace not only its software the way we can-

Lex Fridman 15:26 Mm-hmm.

Max Tegmark 15:27 ... but also its hardware. And, um, that's where we're hea- heading towards at high speed. We're already maybe 2.1 'cause we can, you know, put in a- an artificial knee, a (clears throat) pacemaker, et cetera, et cetera. And if Neuralink and other companies succeed, we'll be Life 2.2, et cetera. But, uh, what- what the company's trying to build, AGI, or are trying to make, is of course full 3.0. And you can put that intelligence into something that also has no ... biological basis whatsoever.

Lex Fridman 16:02 So, less constraints and more capabilities.

Max Tegmark 16:05 Mm-hmm.

Lex Fridman 16:05 Just like the leap from 1.0 to 2.0. There is, nevertheless, you speaking so harshly about bacteria, so disrespectfully about bacteria.

Max Tegmark 16:14 (laughs)

Lex Fridman 16:14 There is still the same kind of magic there that permeates Life 2.0 and, uh, and 3.0. It seems like maybe the thing that's truly powerful about life, intelligence, and consciousness was already there in 1.0. Is it possible?

Max Tegmark 16:34 I think we should be m- humble and not be so quick to ... make everything binary and say either it's there or it's not. Clearly, there's a, there's a great spectrum. And there is even a controversy about whether some unicellular organisms, like amoebas, can maybe learn a little bit, you know, after all. So, apologies if I offend (laughs) any bacteria here.

Lex Fridman 16:55 Yeah.

Max Tegmark 16:56 It wasn't my intent. It was more that I wanted to talk up how cool it is to actually have a brain-

Lex Fridman 17:01 Yeah.

Max Tegmark 17:01 ... where you can learn dramatically within your lifetime.

Lex Fridman 17:04 Typical human.

Max Tegmark 17:05 And- and the higher up you get from 1.0 to 2.0 to- to 3.0, the more you become the captain of your own desti- of your own ship, the master of your own destiny, and the less you become a slave to whatever evolution gave you, right? By upgrading our software, we can be so different from previous generations and even from our parents, much more so than even a bacterium. You know, no offense to them.

Lex Fridman 17:28 (laughs)

Max Tegmark 17:29 And, uh, if you can also swap out your hardware and take any physical form you want, of course, really the sky is the limit.

Lex Fridman 17:36 Yeah. So, the ... (sighs) It accelerates the rate at which you can perform the competition that- computation that determines your destiny.

Max Tegmark 17:45 Yeah. And I think it's- it's worth commenting a bit on what you means in this context also, if you swap things out a lot, right? Uh, this is controversial, but my ... current understanding is that l- that, you know, life is best thought of, um, not as a bag of meat or even a bag of elementary particles, but rather as an- as, um, a system which can process information and retain its own complexity, even though nature is always trying to mess it up. So, it's all about inf- in- information processing. And-... that makes it a lot like something like a wave in the ocean, which is not, it's, it's water molecules, right? The water molecules bob up and down, but the wave moves forward. It's an information pattern. In the same way, you, Lex, you and I are not the same atoms as during the first-

Lex Fridman 18:43 ... time we talked, yeah.

Max Tegmark 18:44 ... interview you did with me. You've swapped out most of them, but it's still you.

Lex Fridman 18:47 Yeah.

Max Tegmark 18:48 And the, the, the information pattern is still there. And, um, if you, if you could swap out your arms and, like, whatever, you can still have this kind of continuity. It becomes a much more sophisticated sort of wave forward in time, where the information lives on. I, I, I lost both my parents, uh, since, since our last podcast. And, and it actually gives me a lot of solace that this way of thinking about them, they haven't entirely died because a lot of mommy and daddies, um ... Sorry, I'm getting a little emotional here, but a lot of their values and ideas and even jokes and so on, they haven't gone away, right? Some of them live on, I can carry on some of them, and they also live on a lot of other, in a lot of other people. So, in this sense, even with life 2.0, we can, to some extent, already transcend our physical bodies and our death. And particularly if you can share your own information, your own ideas with many others, like you do in your podcast, then, um, you know, that's the closest to immortality we can (laughs) get with our bio-bodies.

Lex Fridman 20:06 You carry a little bit of them in you, in some sense.

Max Tegmark 20:10 Yeah. Yeah.

Lex Fridman 20:11 Uh, do you miss them? Do you miss your mom and dad?

Max Tegmark 20:14 Of course. Of course.

Lex Fridman 20:15 What did you learn about life from them, if we can take a bit of a tangent?

Max Tegmark 20:21 Oh, so many things, um, for starters, my, my fascination for math and, um, the physical mysteries of our universe, I think I got a lot of that from my dad, but I think my, my obsession for really big questions and consciousness and so on, that actually came mostly from my mom. And what I got from both of them, which is a very core part of really who I am, I think is, is, um, this, um ... just feeling comfortable with not buying into what everybody else is saying, just doing what I think is right. They, they both very much just bl- you know, did their own thing, and sometimes they got flak for it, and they did it anyway.

Lex Fridman 21:23 That's why you've always been an inspiration to me, that you're at the top of your field and you still, you're still willing to, uh, to tackle the big questions in your own way. You're one of the, one of the people that represents MIT best to me. You've always been an inspiration in that. So, it's good to hear-

Max Tegmark 21:42 Thank you.

Lex Fridman 21:42 ... that you got that from your mom and dad.

Max Tegmark 21:44 Yeah, you're too kind. But, but yeah, I mean, the re- the good reason to do science is because you're really curious и you wanna figure out the truth. If you think, "This is how it is," and everyone else says, "No, no, that's bullshit" and it's that way, you know, eh, (laughs) you stick with what you think is true. And, and, and even if everybody else keeps thinking it's bullshit, there's a certain, um ... I always root for the underdog (laughs) -

Lex Fridman 22:14 (laughs)

Max Tegmark 22:14 ... when I watch movies. And my, my dad once ... I, I ... One time, for example, when I wrote one of my craziest papers ever, our math- talking about our universe ultimately being mathematical, which we're not gonna get into today, I got this email from a quite famous professor saying, "This is not only bullshit, but it's gonna ruin your career. You should stop doing this kind of stuff." I sent it to my dad. Do you know what he said?

Lex Fridman 22:35 (laughs) What did he say?

Max Tegmark 22:36 He replied with a quote from Dante, "Segui il tuo corso e lascia dir la gente," follow your own path and let the people talk. (laughs)

Lex Fridman 22:46 (laughs)

Max Tegmark 22:46 Go Dad!

Lex Fridman 22:47 Yeah.

Max Tegmark 22:48 This is the kind of thing, you know, he's dead, but that, that, um, attitude is not.

Lex Fridman 22:54 How did, uh, losing them as a man, as a human being change you? How did it expand your thinking about the world? How did it, uh, expand your thinking about, you know, this thing we're talking about, which is humans creating another-

Max Tegmark 23:07 Mm-hmm.

Lex Fridman 23:08 ... living, sentient, perhaps, being?

Max Tegmark 23:13 I think it, uh ... mainly due to things, uh, one of them just going through all their stuff after they had passed away and so on, just drove home to me how important it is to ask ourselves, "Why are we doing this, things we do?" Because it's inevitable that you look at some things they spent an enormous time on and you ask, like, i- in hindsight, "Would they really have spent so much time on this, or if, would they have done something that was more meaningful?" Um, so I've been looking more in my life now and asking, you know, "Why am I doing what I'm doing?" And I, I feel ... it should either be something I really enjoy doing, or it should be something that I find really, really meaningful because it helps humanity. And, um, (laughs) if it's in none of those two categories, maybe I should spend less time on it, you know? And the other thing is dealing with death up and personal like this, it's actually made me less afraid of the, um, even less afraid of other people telling me that I'm an idiot, you know, which happens regularly. And just kind of live my life, do my thing, you know. Um, and, um-... it, it's made it a little bit easier for me to focus on what I, what I feel is really important.

Lex Fridman 24:40 What about fear of your own death? Has it made it more real that this is, that this is something that happens?

Max Tegmark 24:49 Yeah, it's made it, uh, extremely real. And I, you know, I'm next, (laughs) next in line in our family now, right? It's me and my brother, my younger brother. But, um, uh, they both handled it with such dignity. It was... That was a true inspiration also. They never complained about things. And you know, when, when you're old and your body starts falling apart, there's more and more to complain about. They looked at what could they still do that was meaningful, and they focused on that rather than wasting time, like, think- talking about or even thinking much about things they were disappointed in. I think anyone can make themself depressed if they start their morning by making a list of grievances. Whereas if you start your day with a little meditation on just things you're grateful for, you, you basically choose to be a happy person.

Lex Fridman 25:39 Because you only have a finite number of days.

Max Tegmark 25:42 Yeah.

Lex Fridman 25:42 You should spend them-

Max Tegmark 25:43 Make it count.

Lex Fridman 25:44 ... being grateful.

Max Tegmark 25:45 Yeah.

Lex Fridman 25:48 Well, you do happen to be working on a thing which seems to have, uh, potentially, uh, some of the greatest impact on human civilization of anything humans have ever created, (laughs) which is artificial intelligence. This is on the both detailed technical level and in a high philosophical level you work on. So, you've mentioned to me that there's an open letter that you're working on.

Max Tegmark 26:15 It's actually, uh, going live in a few hours.

Lex Fridman 26:19 (laughs)

Max Tegmark 26:19 So I've been having late nights and early mornings. It's been very exciting, actually. I, in short, I... Have you seen, uh, Don't Look Up, the film?

Lex Fridman 26:31 Yes, yes.

Max Tegmark 26:32 I don't want to be the movie spoiler for anyone watching this-

Lex Fridman 26:35 Okay.

Max Tegmark 26:35 ... who hasn't seen it, but if you're watching this, you haven't seen it, watch it.

Lex Fridman 26:39 Mm-hmm.

Max Tegmark 26:40 Because we are actually acting out, it's, it's life imitating art. Humanity is doing exactly that right now, except it's an asteroid that we are building ourselves.

Lex Fridman 26:51 Mm-hmm.

Max Tegmark 26:52 Almost nobody is talking about it. People are squabbling across the planet about all sorts of things, which seem very minor compared to the asteroid that's about to hit us, right? Uh, most politicians don't even have their radars, this on their radar. They think maybe in 100 years or whatever. Right now, we're at a fork in the road. This is the most important, um, fork humanity has reached in its over 100,000 years on this planet. We're building, eh, effectively a, a new species that's smarter than us. It doesn't look so much like a species yet 'cause it's mostly not embodied in robots, but, um, that's a technicality which will soon be changed. And, and this arrival of, of artificial general intelligence that can do all our jobs as well as us, and probably shortly thereafter, superintelligence, which greatly exceeds our cognitive abilities, it's gonna either be the, the best thing ever to happen to humanity or the worst. I'm really quite confident that there is not that much middle ground there.

Lex Fridman 27:55 But it would be fundamentally transformative to human civilization.

Max Tegmark 27:58 Of course, utterly and totally. You know, again, we, we branded ourselves as homo sapiens 'cause it seemed like the basic thing. We're the king of the castle on this planet. We're the smart ones, if we can control everything else. Uh, this could very easily change. We're, we're certainly not gonna be the smartest on the planet for very long if AI... unless AI progress just halts. And we can talk more about why I, I think that's true 'cause it's, it's controversial. And, and then we can also talk about reasons you might think it's gonna be the best thing ever, and the reason you think it's gonna, going to be the end of humanity, which is, of course, super controversial. But what I think we can... (laughs) Anyone who's working on, uh, advanced AI can agree on is it's, it's much like the film Don't Look Up in that it's just really comical how little serious public debate there is about it, given how huge it is.

Lex Fridman 29:03 (sighs) So what we're talking about is the development of currently things like GPT-4 and the signs it's showing of a rapid improvement that may, in the near term, lead to development of super intelligent AGI, AI, general AI systems, and what kind of impact that has on society.

Max Tegmark 29:26 Exactly.

Lex Fridman 29:26 When that thing is, achieves general human level intelligence, and then beyond that, general superhuman level intelligence.

Max Tegmark 29:34 Mm-hmm.

Lex Fridman 29:34 (sighs) There's a lot of questions to explore here. So one, you mentioned halt. Is that, uh, the content of the letter is-

Max Tegmark 29:44 Yeah.

Lex Fridman 29:45 ... to suggest that maybe we should pause the development of these systems?

Max Tegmark 29:49 Exactly. So this is very controversial. From when, when, um, we talked... The first time, we talked about how I was involved in starting the Future of Life Institute, and we worked very hard on, 2014, 2015, was the mainstream AI safety.

Lex Fridman 30:04 Mm-hmm.

Max Tegmark 30:05 The idea that there even could be risks and that you could do things about them.

Lex Fridman 30:08 Mm-hmm.

Max Tegmark 30:09 Before then, a lot of people thought it was just really kooky to even talk about it. And a lot of AI researchers felt worried that this was too flaky and could be bad for funding, and that the people who talked about it were just not... didn't understand AI. I, I'm very, very happy with th- how that's gone, in that now, you know, just it's completely mainstream. You go to any AI conference and people talk about AI safety, and it's a t- nerdy technical field full of equations and simu- and blah, blah.

Lex Fridman 30:39 Yes.

Max Tegmark 30:41 ... as it should be. Uh, but there is this other thing which has been quite taboo up until now, calling for slowdown. So, what we've constantly been saying, including myself, I've been biting my tongue a lot, you know, is that, you know, we, we don't need to slow down AI development. We just need to win this race, the wisdom race between the growing power of the AI and the growing wisdom with which we manage it. And rather than trying to slow down AI, let's just try to accelerate the wisdom.

Lex Fridman 31:16 Mm-hmm.

Max Tegmark 31:17 Do all this technical work to figure out how you can actually en- ensure that your powerful AI is gonna do what you want it to do, and have society adapt also with, um, incentives and regulations so that these things get put to good use. Um, sadly, that didn't pan out. The, um, progress on technical AI, um, capabilities has gone a lot faster than, than many people thought. Uh, back, uh, when we started this in 2014, turned out to be easier to build really advanced AI than we thought. Um, and, uh, on the other side, it's gone much slower than we hoped with getting, um, policymakers and others to actually put the inpla- incentives in place to, to make, um, steer this in the, in the good directions. We can, uh, maybe we should unpack it and talk a little bit about each. So-

Lex Fridman 32:12 Yeah.

Max Tegmark 32:13 ... why did it go faster than we, we, than a lot of people thought? Um, in hindsight, it's exactly like building, um, flying machines. People spent a lot of time wondering about how, how do birds fly, you know? And that turned out to be (laughs) really hard. Ha- have you seen the TED Talk with a flying bird?

Lex Fridman 32:31 Like a flying robotic bird?

Max Tegmark 32:33 Yeah. It flies around the audience. But it took 100 years longer to figure out how to do that than for the Wright brothers to build the first airplane, because it turned out there was a much easier way to fly.

Lex Fridman 32:42 Yeah.

Max Tegmark 32:43 And evolution picked the more complicated one because it had its hands tied. It could only build a fly- a machine that could assemble itself-

Lex Fridman 32:51 Mm-hmm.

Max Tegmark 32:51 ... which the Wright brothers didn't care about.

Lex Fridman 32:53 Mm-hmm.

Max Tegmark 32:53 They could only build a machine that used only the most common atoms in the periodic table.

Lex Fridman 32:57 Mm-hmm.

Max Tegmark 32:58 Wright brothers didn't care about that. They could use steel, iron atoms. And it, it had to be able to repair itself, and it also had to be incredibly fuel-efficient. You know, a lot of birds use less than half the fuel of a remote control plane flying the same distance. For humans, just throw a little more money, put a little more fuel in it, zoof, there you go, 100 years earlier. That's exactly what's happening now wi- with these large language models. The brain is incredibly complicated. Many people made the mistake here thinking we have to figure out how the brain does human-level AI first before we could build in a machine. That was completely wrong. You, you can take an incredibly simple computational system called a transformer network and just train it to do something incredibly dumb, just read a gigantic amount of text and try to predict the next word, and it turns out, if you just throw a ton of compute at that and a ton of data, it gets to be sh- frighteningly good, like GPT-4, which I've been playing with so much since it came out, right? And, um, there's still some debate about whether that can get you all the way to full human level or not. But, uh, yeah, we can come back to the details of that and how you might get to human-level AI even if, uh, large language models don't.

Lex Fridman 34:22 Can you c- briefly, just as a small tangent, comment on your feelings about GPT-4? So just that you're impressed by this rate of progress, but where, where is it? Can GPT-4 reason? What are, like, the intuitions? What are human interpretable words you can assign to the capabilities of GPT-4 that makes you so damn impressed with it?

Max Tegmark 34:45 I'm both very excited about it and terrified. (laughs) It's an interesting mixture of emotions.

Lex Fridman 34:52 All the best things in life include those two somehow.

Max Tegmark 34:55 Yeah. It can absolutely reason. Anyone who hasn't played with it, uh, I r- highly recommend doing that before dissing it. It, uh, it can do quite, quite remarkable reasoning. I've had it do a lot of things which I realize I couldn't do that myself that well even, and, and obviously does it dramatically faster than we do too when you watch it type.

Lex Fridman 35:17 Mm-hmm.

Max Tegmark 35:17 And it's, it's doing that while servicing a massive number of other humans at, at the same time. At the same time, it cannot reason as well as, um, a human can on some tasks be- just bec- it's obviously a limitation from its architecture. You know, what we have in our heads, what in geek speak is called a recurrent neural network. There are loops. Information can go from this neuron, to this neuron, to this neuron, and then, then back to this one. You can, like, ruminate on something for a while. You can self-reflect a lot. Uh, these large language models that are... they cannot, like GPT-4. It's, it's this so-called transformer where it's just like a one-way street of information, basically. In, in geek speak, it's called a feedforward neural network. And it's only so deep, and so it can only do logic that's that many steps and that deep. And it's not... And you can create problems which it will fail to solve, you know, for that reason. Um, but the fact that it can do so amazing things with this incredibly simple architecture already i- is quite stunning. And, and, and what we see in my lab at MIT when we look inside large language models to try to figure out how they're doing it, which, that's the key core focus of our research, it's called, um, mechanistic interpretability in geek speak. You know, you have this machine that does something smart, you try to reverse, reverse engineer it and see how does it do it.... or think of it also as artificial neuroscience-

Lex Fridman 36:49 (laughs)

Max Tegmark 36:50 ... right? 'Cause it's exactly what neuroscientists do with actual brains.

Lex Fridman 36:52 I love it.

Max Tegmark 36:52 But here, you have the advantage that you can ... you don't have to worry about measurement errors. You can see what every neuron is doing all the time. And, and a recurrent thing we see again and again, there's been a number of beautiful papers quite recently by, by a lot of researchers, some of them here even in this area, is where when they figure out how something is done, you can say, "Oh, man. That's such a dumb way of doing it." And you re- immediately see how it can be improved. Like for example, there was this beautiful paper recently where they figured out how a large language model stores certain facts, like Eiffel Tower is in Paris, and they figured out exactly how it stored it. And where ... The proof that they understood it was they could edit it.

Lex Fridman 37:34 Mm-hmm.

Max Tegmark 37:34 They changed some of the synapses in it, and then they asked it, "Where's the Eiffel Tower?" And it said, "It's in Rome."

Lex Fridman 37:40 Mm-hmm.

Max Tegmark 37:41 And then they asked, you know, "How do you get there? Oh, how do you get there from Germany? Oh, you take this train, uh, to Roma Termini train station," and this and that. "And what might you see if you're in front of it?" "Oh, you might see the (laughs) Colosseum."

Lex Fridman 37:55 (laughs)

Max Tegmark 37:55 So they had edited it.

Lex Fridman 37:57 So they literally moved it to Rome.

Max Tegmark 37:59 But it ... The way that it's storing this information is incredibly dumb, uh, for, for, for our ... any f- any fellow nerds listening, uh, to this. There was a big matrix and, uh, and roughly speaking, there are certain row and column vectors which encode these things, and they, they correspond very hand-wavily to principal components. And it would be much more efficient for a sparse matrix, just store it in the (laughs) database, you know? And, and, but ... And everything ... So far, we've figured out how these things do, are ways where you can see they can easily be improved. And the fact that this particular architecture has some roadblocks built into it is in no way gonna prevent, um, crafty researchers from quickly finding workarounds and making other kinds of architectures, so to go all the way. So, so it's, um ... In, in short, it's turned out to be a lot easier to build human, close to human intelligence than we thought, and that means our runway as a species to get our shit together ha- ha- has shortened.

Lex Fridman 39:06 And it seems like the scary thing about the effectiveness of large language models, uh, so Sam Altman, I've recently had a conversation with, and he really showed that the leap from GPT-3 to GPT-4 has to do with just a bunch of hacks, a bunch of tr- uh, little explorations above ... with, uh, smart researchers doing a few little fixes here and there. It's not some fundamental leap and transformation in the architecture.

Max Tegmark 39:35 And more data and more compute.

Lex Fridman 39:37 And more data and compute, but he said the big leaps has to do with not the data and the compute-

Max Tegmark 39:42 Mm-hmm.

Lex Fridman 39:42 ... but just exp- learning this new discipline.

Max Tegmark 39:45 Yeah.

Lex Fridman 39:45 Just like you said. So researchers are going to look at these architectures, and there might be big leaps where you realize, "Wait, why are we doing this in this dumb way?"

Max Tegmark 39:54 Yeah.

Lex Fridman 39:54 And all of a sudden, this model is 10X smarter.

Max Tegmark 39:57 Yeah.

Lex Fridman 39:57 And that, that can happen on any one day, on any one Tuesday or Wednesday afternoon, and then all of a sudden you have a system that's 10X smarter. Um, it seems like it's such a new discipline, it's such a new-

Max Tegmark 40:08 Yeah.

Lex Fridman 40:08 ... like we h- understand so little about why this thing works so damn well that, uh, the linear improvement of compute, or exponential, but the steady improvement of compute, steady improvement of the data may not be the thing that even leads to the next leap. It could be a surprise little hack that improves everything.

Max Tegmark 40:24 Or a lot of little leaps here and there because, because so much of this is out in the open also. So many smart people are looking at this and trying to figure out little leaps here and there, and, and, uh, it becomes this sort of collective race where if p- a lot of people feel if I don't take the leap, someone else will. And this is actually very crucial for, for the other part of it. Why do we wanna slow this down? So again, what this open letter is calling for is just pausing all training of, uh, systems that are more powerful than GPT-4 for six months to just give a chance for the labs to coordinate a bit on safety and, and for society to adapt, give the right incentives to the labs. 'Cause I ... You know, you've interviewed a lot of these (laughs) people who lead these labs, and you know just as well as I do that they're good people.

Lex Fridman 41:17 Mm-hmm.

Max Tegmark 41:17 They're idealistic people. They're doing this first and foremost because they believe that AI has a huge potential to help humanity.

Lex Fridman 41:24 Mm-hmm.

Max Tegmark 41:25 And, um ... But at the same time, they are trapped in this horrible race to the bottom. Uh, uh, have you read Meditations on Moloch by Scott Alexander?

Lex Fridman 41:40 Yes.

Max Tegmark 41:41 Yeah. It's a beautiful essay on this poem by Ginsberg, where he interprets it as being about this monster. It's this game theory monster that, that pits people into ... against each other in this, this race to the bottom where everybody ultimately loses.

Lex Fridman 41:57 Yes.

Max Tegmark 41:57 And it ... The evil thing about this monster is, even though everybody sees it and understands, (laughs) they still can't get out of the race, right? Most ... A good fraction of all the bad things that we humans do are caused by Moloch, and I, I like, uh, Scott Alexander's, um, naming of the monster so we can, we humans can, uh, think of it as the f- an f- a thing. Uh, if you look at why do we have overfishing, why do we have more generally, like, tragedy of the commons, why is it that, um ... So, Liv Boeree, I don't know if you've had her on your podcast.

Lex Fridman 42:31 Mm-hmm. Yeah. She's become a friend, yeah.

Max Tegmark 42:33 Great. She made this awesome point re- recently that beauty filters-

Lex Fridman 42:38 Yes.

Max Tegmark 42:38 ... that a lot of female influencers feel pressured to use-

Lex Fridman 42:43 Mm-hmm.

Max Tegmark 42:43 ... are exactly Moloch in action again.

Lex Fridman 42:46 Mm-hmm.

Max Tegmark 42:46 First, nobody was using them, and people saw them just the way they were, and then some of them started using it-... and becoming ever more plastic fantastic. And then the other ones that weren't using it started to realize that if they wanna just keep their, their market share, they have to start using it too.

Lex Fridman 43:05 Yeah.

Max Tegmark 43:05 And tha- and then you're in a situation where they're all using it, and, and none of them has any more market share or less than before. So (laughs) nobody gained anything. Everybody lost, and they have to keep becoming ever more plastic fantastic also, right? And, uh, but nobody can go back to the old way 'cause it's just (laughs) too costly, right? The- the- the- Moloch is everywhere, and, um, Moloch is not a new arrival on, on the scene either. It's we humans have developed a lot of collaboration mechanisms to help us fight back against Moloch through various kinds of constructive collaboration. The Soviet Union and the United States did sign a number of ar- of arms control treaties against Moloch, who was trying to stoke them into, uh, unnecessarily risky nuclear arms races, et cetera, et cetera. And this is exactly what's happening on the AI front. This time, it's a little bit geopolitics, but it's mostly money, where there's just so much commercial pressure, you know? If you take any of these leaders of the top tech companies, if they just say, you know, "This is too, too risky, I wanna pause for six months." They're gonna get a lot of pressure from shareholders and others, who are like, "Well, you know, if you pause, but those guys don't pause, we're... We don't wanna get our lunch eaten."

Lex Fridman 44:31 Yeah.

Max Tegmark 44:32 And shareholders even have the power to replace the, the executives in the worst case, right? So, we did this open letter because we wanna help these idealistic tech executives to do what their heart tells them by providing enough public pressure on the whole sector to just pause, so that they can all pause in a coordinated fashion. And I think without the public pressure, none of them can do it alone, push back against their shareholders, no matter how goodhearted they are, 'cause Moloch is a really powerful foe.

Lex Fridman 45:09 So the idea is to, for the major developers of AI systems like this, so we're talking about Microsoft, Google, uh, Meta, and anyone else?

Max Tegmark 45:21 Uh, OpenAI is very close with Microsoft now, of course.

Lex Fridman 45:25 With Microsoft right now, yes.

Max Tegmark 45:25 And there are, there are plenty of, of, of smaller players, Anthro- for example, Anthropic is, is very impressive. There's Conjecture. There, there's many, many, many players. I don't wanna make a long list so we leave anyone out.

Lex Fridman 45:37 Yeah.

Max Tegmark 45:38 Uh, and, um, for that reason, it's so important that, uh, some coordination happens, that there's external pressure on all of them saying, "You all need to pause." 'Cause then the, the people, the, the researchers in the wo- the- these organizations who, the leaders who wanna slow down a little bit, they can say to their shareholders, you know, "Everybody is slowing down because of this pressure, and it's, and it's the right thing to do."

Lex Fridman 46:03 Have you seen in history there, um, examples where it's possible to pause-

Max Tegmark 46:08 Yes.

Lex Fridman 46:08 ...the Moloch?

Max Tegmark 46:09 Absolutely. And even, like, human cloning, for example. You could make so much money on human cloning. Uh, why aren't we doing it? Because biologists thought hard about this and felt like this is way too risky. We s- they got together all, in the '70s in Asilomar and decided even to stop a lot more stuff also, just editing the human germline, like g- gene editing that goes in to our offspring, and decided, "Let, let's just not do this, because it's too unpredictable what it's gonna lead to. We could lose control over what happens to our species."

Lex Fridman 46:51 Mm-hmm.

Max Tegmark 46:52 So they paused. Uh, there was a ton of money to be made there, uh, so it's, it's very doable. But you just need, you need a public awareness of the, of what the risks are, and the broader community coming in and saying, "Hey, let's slow down." And, you know, another, another common pushback I get today is, "We, we can't stop in the West because China." And in China, undoubtedly, they also get told, "We can't slow down because the West," because both sides think they're the good guy.

Lex Fridman 47:22 Yeah.

Max Tegmark 47:24 But look at human cloning, you know? (laughs) Did China forge ahead with human cloning? There's been exactly one human cloning that's actually been done that I know of.

Lex Fridman 47:32 Mm-hmm.

Max Tegmark 47:32 It was done by a Chinese guy. Do you know where he is now?

Lex Fridman 47:35 Where?

Max Tegmark 47:36 In jail. And you know who put him there?

Lex Fridman 47:39 Who?

Max Tegmark 47:40 The Chinese government. Not because Westerners said, "China, look, this is..." No, the Chinese government put him there, 'cause they also felt... They like control, the Chinese government. If anything, maybe they are even more fo- concerned about having control than, than Western governments. Have no incentive of, of just losing control over where everything is going. And you can also see the Ernie bot that, uh, was released by, I believe Baidu recently. They got a lot of pushback from the government and had to rein it in, you know, in a big way. Um, I think o- once this basic message comes out that this isn't an arms race, it's a suicide race, where everybody loses if anybody's AI goes out of control, it really changes the whole dynamic. It, it, it's not... It's... And I'll say this again, 'cause this is a very basic point I think a lot of people get wrong. Because a lot of people dismiss the whole idea that AI can really get very superhuman, because they think there's something really magical about intelligence such that it can only exist in human minds, you know. Because they believe that, they think it's gonna kinda get to just more or less GPT-4++, and then that's it.... they don't see it as a super- s- s- as a suicide race. They think whoever gets there first, they're gonna control the world, and they're gonna win. That's not how it's gonna be. And we can talk again about the, the scientific arguments, f- from why it's not gonna stop there. But (clears throat) the way it's gonna be is if, if anybody completely loses control and, you know, y- you don't care i- i- i- if some, some, someone manages to take over the world who really doesn't share your goals and they... You probably don't really even care very much about what nationality they have. You're not gonna like it. It's gonna be much worse than today. Uh, who, if it's, if you live in Orwellian dystopia, who (laughs) , who, who you, what do you care who s- created it, right? Uh, and if someone... If it goes farther and, and we just lose control even to the machines, so that it's not us versus them, it's us versus it, uh, what do you care who, who created this, this underlying entity which m- has goals different from humans ultimately, and we get marginalized, we get made obsolete, we get replaced? It, th- that's why, what I mean when I say it's a suicide race. You know, it's, um, kinda like we're, we're rushing towards this cliff, uh, but the closer to the cliff we get, the more scenic the views are and the more money there is there and the more... So, so we keep going, but we have to also stop at some point, right? Quit while we're ahead. And, uh, it's, um... It's a suicide race, which n- cannot be won, but the way to really benefit from it is to continue developing awesome AI-

Lex Fridman 50:37 Mm-hmm.

Max Tegmark 50:37 ... a little bit slower, so we make it safe, make sure it does the things that humans want, and create a condition where everybody wins. And then the techn- technology has shown us that, you know, geopolitics and, and politics in general, is not a zero-sum game at all.

Lex Fridman 50:54 So, there is some rate of development that will lead us as a human species to lose control of this thing, and the hope you have is that there's some lower level of development which will not-

Max Tegmark 51:07 Yeah.

Lex Fridman 51:07 ... which will not l- allow us to lose control. This is an interesting thought you have about losing control. So, what... If you have somebody, if you're somebody like Sundar Pichai or Sam Altman at the head-

Max Tegmark 51:17 Yeah.

Lex Fridman 51:17 ... of a company like this, you're saying if they develop an AGI, they too will lose control of it? So, no one person can maintain control. No group of individuals can maintain control.

Max Tegmark 51:28 If it's r- if it's created th- very, very soon and is a big black box that we don't understand, like the large language models, yeah, th- then I'm very confident they're gonna lose control. But this isn't just me saying it. You know, Sam Altman and Demis Hassabis have both said-

Lex Fridman 51:44 Mm-hmm.

Max Tegmark 51:44 ... it themselves, acknowledged that, you know, there's really great risks with this, and they, they wanna slow down once they feel like it's, it's scary. It's, but it's clear that they're stuck in this... Again, Moloch is forcing them to go a little faster than, than they're comfortable with because of pressure from just commercial pressures, right? (clears throat) Uh, it... F- to get a bit optimistic here, of course this is a problem that can be ultimately solved. Uh, it just w- to win this wisdom race, it's clear that what h- we hoped that was gonna happen hasn't happened. The, the capability progress has gone faster than a lot of people thought than, and the po- the progress in, in the public sphere of policymaking and so on has gone slower than we thought. Even the technical AI safety has gone slower. A lot of the technical safety research was kind of banking on that, um, large language models and other poorly understood systems couldn't get us all the way.

Lex Fridman 52:37 Yeah.

Max Tegmark 52:37 That you had to build more of a kind of intelligence that you could understand, maybe it could prove itself safe, you know, things like this. And, um, I'm quite confident that this can be done, um, so we can reap all the benefits. But we cannot do it as quickly as, uh, (laughs) this out of control express train we are on now is gonna get the AGI. That's why we need a little more time, I feel.

Lex Fridman 53:04 Is there something to be said, what like Sam Altman talked about, which is while we're in the pre-AGI stage, to release often and as transparently as possible to learn a lot? So, as opposed to being extremely cautious, release a lot. Don't, uh, don't invest in a closed development where you focus on AI safety. While it's somewhat "dumb," quote, unquote, uh, release as often as possible. And as you start to see signs of, uh, human-level intelligence and, or superhuman level intelligence, then you put a halt on it.

Max Tegmark 53:42 Well, what a lot of safety researchers have been saying for many years is that the most dangerous things you can do with an AI is, first of all, teach it to write code.

Lex Fridman 53:52 Yeah.

Max Tegmark 53:52 'Cause that's the first step towards recursive self-improvement, which can take you from AGI to much higher levels. Okay. O- oops, we've done that. And, uh, another thing, high risk, it's connect it to the internet, let it go to websites, download stuff on its own, uh, talk to people. Oops, we've done that already. You know, Eliezer Yudkowsky, you said you interviewed him recently, right?

Lex Fridman 54:14 Yes, yep.

Max Tegmark 54:15 So, he had this tweet recently which like (laughs) ...

Lex Fridman 54:18 (laughs)

Max Tegmark 54:18 ... gave me one of the best laughs in a while where he was like, "Hey, people used to make fun of me and say, 'You're so stupid, Eliezer,' because you're saying, you're saying, um, 'You have to worry.'" Ob- obviously, developers, once they get to, like, really strong AI, first thing you're gonna do is, like, never connect it to the internet, keep it in, in a box-

Lex Fridman 54:37 Yeah.

Max Tegmark 54:37 ... where h- you know, you can really study it, safe. So, he had written it in the, like in the meme form, so it was like, "Then..."

Lex Fridman 54:44 Yeah.

Max Tegmark 54:44 And then that, and then, "Now."

Lex Fridman 54:48 (laughs)

Max Tegmark 54:48 "Let's m- LOL, let's make a chatbot." (laughs)

Lex Fridman 54:52 (laughs) Yeah. Yeah. Yeah.

Max Tegmark 54:53 And the third thing, S- Stuart Russell-

Lex Fridman 54:55 Yeah.

Max Tegmark 54:56 ... you know, m- amazing AI researcher. He ha- he has argued for a while that...We should never teach AI anything about humans. Above all, we should never let it learn about human psychology and how you manipulate humans. That's the most dangerous kind of knowledge you can give it. Yeah, you can teach it all it needs to know how to... about how to cure cancer and stuff like that. But don't let it read Daniel Kahneman's book about cognitive biases and all that.

Lex Fridman 55:24 Yeah.

Max Tegmark 55:25 And then, oops, LOL, you know, let's invent social media, uh, recommender algorithms, which do exactly that. They, they, uh, get so good at knowing us and pressing our buttons that we're, we're, we're starting to create a world now where we just have evermore pe- hatred. Uh, 'cause they figured out that these algorithms, not for... out of evil, but just to make money on advertising, that the best way to get more engagement, the euphemism, get people glued to their little rectangles, right, is just to make them pissed off.

Lex Fridman 56:00 That's really interesting that a large AI system that's doing the recommender system kind of task on social media is basically just studying human beings. Because it's a bunch of us-

Max Tegmark 56:11 Mm-hmm.

Lex Fridman 56:12 ... rats, giving it signal, non-stop signal. It'll show a thing, and then we give signal on whether we spread that thing, we like that thing, that thing increases our engagement, gets us to return to the platform. And it has that on the scale of hundreds of millions of people constantly. So, it's just learning and learning and learning. And presumably if the param- the number of parameters in your neural network that's doing the learning, uh, more end-to-end the learning is, the more it's able to just basically encode how to manipulate human behavior-

Max Tegmark 56:43 Exactly.

Lex Fridman 56:43 ... how to control humans at scale.

Max Tegmark 56:45 Exactly. And that is not something I think is in humanity's interest.

Lex Fridman 56:49 Yes.

Max Tegmark 56:49 Uh, uh, right now, it's mainly letting some humans manipulate other humans for profit and power, which already (laughs) caused a lot of damage. And eventually, that's the sort of, uh, skill that can tr- make AIs persuade humans to let them escape, uh, whatever safety precautions we have put. You know, there was a really nice article, um, in The New York Times recently by Yuval Noah Harari and, and, and, um, two co-authors, including Tristan Harris from The Social Dilemma. And they have this phrase in there I love. It said that humanity's first contact with advanced AI was social media. And we lost that one. We now live in a country where there's much more hate in the world, where there's much more hate, in fact. And in our democracy, that we're having this conversation, and people can't even agree on who won the last election, you know? And we humans often poke, point fingers at other humans and say it's their fault. But it's really M- Moloch and these AI algorithms. We got the algorithms, and then Moloch pitted the social media companies arou- against each other, so nobody could have a less creepy algorithm, 'cause then they would lose out on alg- revenue to the other company.

Lex Fridman 58:07 Is there any way to win that battle back? Just, if we just linger on this one battle that we've lost in terms of social media. Is it possible to redesign social media, this very medium in which we use as a civilization to communicate with each other, to have these kinds of conversation, to have discourse to try to figure out how to solve the biggest problems in the world? Whether that's nuclear war or the development of AGI. Is, is it possible, uh,-

Max Tegmark 58:34 I think-

Lex Fridman 58:34 ... to do social media correctly?

Max Tegmark 58:35 ... I think it's not only possible, but it's, it's necessary. Who are we kidding that we're gonna be able to solve all these other challenges if we can't even have a conversation with each other that's constructive? The whole idea, the key idea of democracy, uh, is that you get a bunch of people together and they have a real conversation, the ones you try to foster on this podcast, where you respectfully listen to people you disagree with. And you realize actually, you know, there are some things actually we... some common ground we have. And let's, let's n- yeah, we both agree let's not have a nuclear war, let's not do that, um, et cetera, et cetera. We're kidding ourselves thinking we can face th- off the second contact with evermore powerful AI that's happening now with these large language models if we can't even have a functional conversation in the public space. That's why I started to improve the news project-

Lex Fridman 59:27 Mm-hmm.

Max Tegmark 59:28 ... improvethenews.org. But, um, I, I'm an optimist fundamentally in, um... in that there is a lot of intrinsic goodness in, in, in people. And that, uh, what makes the difference between someone doing good things for, for humanity and bad things is not some sort of fairytale thing, that this person was born with a evil gene, and this one was not born with a good gene. No. I think it's whether we put... whether people find themselves in situations that bring out the best in them or that bring out the worst in them. And I feel we're building an internet and a society that brings out the worst in us.

Lex Fridman 1:00:14 But it doesn't have to be that way.

Max Tegmark 1:00:16 No, it does not.

Lex Fridman 1:00:16 So if you... It's possible to create incentives, and also create incentives that make money, uh, that both make money and bring out the best in people.

Max Tegmark 1:00:24 I mean, in the long term, it's not a good investment for anyone, you know, to have-

Lex Fridman 1:00:28 Yeah.

Max Tegmark 1:00:28 ... a nuclear war, for example. And, you know, is it a good investment for humanity if we just ultimately replace all humans by machines, and then we're so obsolete that eventually the- there are no humans left? Well, depends, I guess, on how you do the math. W- what, but like if-

Lex Fridman 1:00:43 (laughs)

Max Tegmark 1:00:44 ... I would say by any reasonable economic standard, (laughs) if you look at the future income of humans and there aren't any, you know, that's not a good investment. Uh, moreover, like why, why can't we have a little bit of pride in our species, damn it? You know, why, why should we just build another species that gets rid of us? If we were Neanderthals, would we really consider it a smart move if the, the... if we had really advanced biotech to build homo sapiens?... you- you- you know, you might say, "Hey Max, you know, yeah let- let's build- build, uh, these homo sapiens. They're gonna be smarter than us. Th- maybe they can help us, defend us better against, uh, predators, and help fix up our caves, make them nicer. We'll control them und- undoubtedly," you know. So, then they build- build a couple, a little baby girl, a little baby boy, you know? And- and then you have some- some wise old neanderthal elder who's like, "Hmm, I'm scared that, uh, we're opening a Pandora's box here, and that we're gonna get outsmarted by these n- n- super neanderthal intelligences, and there won't be any neanderthals left." But then you have a bunch of others in the cave go, "Oh, you're such a Luddite scaremonger. Of course they're gonna want to keep us around 'cause we are their creators, and- and why, you know, the smarter, I think the smarter they get, the nicer they're gonna get. They're gonna leave us, they're gonna- they're gonna want us around and it's gonna be fine. And- and besides, look at these babies, they're so cute. Clearly they're totally harmless." That's exact-... Those babies are exactly GPT-4.

Lex Fridman 1:02:18 Yeah.

Max Tegmark 1:02:19 It's not... I wanna be clear, it's not GPT-4 that's terrifying. It's that GPT-4 is a baby technology. You know, Microsoft even had a paper recently out, uh, with a title something like Sparkles of AGI, where they were basically saying this is baby AI, like these little neanderthal babies. And it's gonna grow up. There's gonna be other systems from- from the same company, from other companies that'll be way more powerful and, but they're gonna take all the things, ideas from these babies, and before we know it, we're gonna be like those last neanderthals who were pretty disappointed when they realized that they were getting replaced.

Lex Fridman 1:03:06 Well, this interesting point you make, which is the programming, is- is entirely possible that GPT-4 is already the kind of system that can change everything by writing programs. So...

Max Tegmark 1:03:19 Life 3... It's, yeah, it's, because it's life 2.0. The- the systems I'm afraid of are gonna look nothing like a large language model and they're not... But once it gets, once it or other people figure out a way of using this tech to make much better tech, right? It's just constantly replacing its software. And from everything we've seen about how- how these work under the hood, they're like the minimum viable intelligence. They do everything in a way, like the dumbest way that still works, sort of.

Lex Fridman 1:03:49 Yeah.

Max Tegmark 1:03:49 And, um, so they are life 3.0, except when they replace their software, it's a lot faster than when you- when- when you decide to learn Swedish. Poof. And moreover, they think a lot faster than us too. So, when, uh, you know, we don't think on... Have one logical step every nanosecond or- or few- or so, the- the way they do. And we can't also just suddenly scale up our hardware massively in the cloud, 'cause we're so limited, right? Uh, so they are... And- and they are also life... Consume, become a little bit more like life 3.0 in that if they need more hardware, hey, just rent it in the cloud, you know? How do you pay for it? Well, with all the services you provide.

Lex Fridman 1:04:44 And what we haven't seen yet, which could change a lot, is, uh, entire sys- software systems. So, right now, programming is done sort of in bits and pieces, uh, as- as an assistant tool to humans. But I do a lot of programming-

Max Tegmark 1:05:03 Mm-hmm.

Lex Fridman 1:05:03 ... and with the kind of stuff that GPT-4 is able to do, I mean, it's- it's replacing a lot what I'm able to do.

Max Tegmark 1:05:07 Mm-hmm.

Lex Fridman 1:05:07 But I... You still need a human in the loop to kind of manage the design of things, manage, like, what are the prompts that generate the kind of stuff to- to do some basic adjustment of the code, to- to do some debugging. But if it's possible to add on top of GPT-4 kind of, uh, feedback loop of- of, uh- of, uh, self-debugging improving the code, and then you launch that system out into the wild, on the internet because everything is connected, and have it do things, have it interact with humans, and then get that feedback. Now, you have this giant ecosystem-

Max Tegmark 1:05:43 Yeah.

Lex Fridman 1:05:44 ... of humans. It's one of the things that, um-

Max Tegmark 1:05:46 Yeah.

Lex Fridman 1:05:46 ... Elon Musk recently sort of tweeted as a case why everyone needs to pay $7 or whatever for Twitter.

Max Tegmark 1:05:53 To make sure they're real.

Lex Fridman 1:05:54 They're... Make sure they're real. We're now going to be living in a world where the- the bots are getting smarter and smarter and smarter to a degree where you can't, uh, you can't tell the difference between a human and a bot.

Max Tegmark 1:06:06 That's right.

Lex Fridman 1:06:07 And now, you can have, uh, bots outnumber humans by-

Max Tegmark 1:06:11 Yeah.

Lex Fridman 1:06:11 ... one million to one-

Max Tegmark 1:06:12 Yeah.

Lex Fridman 1:06:12 ... which is why he's making the case why you have to pay-

Max Tegmark 1:06:15 Yeah.

Lex Fridman 1:06:16 ... to prove you're human-

Max Tegmark 1:06:17 Yeah, yeah.

Lex Fridman 1:06:17 ... which is one of the only mechanisms to prove-

Max Tegmark 1:06:19 Yeah.

Lex Fridman 1:06:19 ... which is (laughs) depressing.

Max Tegmark 1:06:21 And I, yeah, I feel we have to remember, as individuals, we should, from time to time, ask ourselves, why are we doing what we're doing? Right? And as a species, we need to do that too. So, if we're building, as- as you say, machines that are outnumbering us and more- more and more outsmarting us and- and replacing us on the job market, not just for the dangerous and- and boring tasks, but also for writing poems and doing art and things that a lot of people find really meaningful, gotta ask ourselves why. Why are we doing this? Uh, we are... The answer is Moloch is tricking us into doing it. And it's such a clever trick that, even though we see the trick, we still have no choice but to fall for it, right? Come... Also, the thing you said about you using, uh-... co-pilot-

Lex Fridman 1:07:14 Mm-hmm.

Max Tegmark 1:07:15 ... AI tools to program faster, how many time- what factor faster would you say you- you code now? Ho- does it go twice as fast or?

Lex Fridman 1:07:22 I don't really, uh, because it's such a new tool.

Max Tegmark 1:07:25 Yeah.

Lex Fridman 1:07:26 It's- I don't know if speed has significantly improved, um, but it feels like I'm a year away from being, uh, five to ten times faster.

Max Tegmark 1:07:36 Mm-hmm. So if- if that's typical for programmers, then, uh, you're already seeing another kind of self- recursive self-improvement, right? Because previously, a- one, like, a major generation of improvement of the codes would happen on the human R&D timescale.

Lex Fridman 1:07:53 Mm-hmm.

Max Tegmark 1:07:53 And now, if that's five times shorter, then it's gonna take five times less time than it otherwise would to develop the next level of these tools.

Lex Fridman 1:08:01 Mm-hmm.

Max Tegmark 1:08:01 And so on. So this- the- th- these- these are the- this is exactly the sort of beginning of an- of an intelligence explosion. There can be humans in the loop a lot, in the early stages, and then eventually, humans are needed less and less, and the machines can more kind of go it alone, but you- what you want- you said there is just an exact example of- of these sort of things. Another thing which- which, um... I was kind of lying on my psychiatrist, imagining I'm on a psychiatrist couch here, saying, "Well, what are my fears that people would do with, um, AI systems?" Another fe- (laughs) so I mentioned three that I had fears about many years ago that they would do, uh, namely, uh, teach it to code, (laughs) uh, connect it to the internet, and teach it to manipulate humans. A fourth one is building an API...

Lex Fridman 1:08:49 (laughs)

Max Tegmark 1:08:49 ... where code can control this super powerful thing, right?

Lex Fridman 1:08:52 Yeah.

Max Tegmark 1:08:52 That's very unfortunate, because one thing that systems like GPT-4 have going for them is that they are an oracle, in the sense that they just answer questions. There is no robot connected to- to GPT-4. GPT-4 can't go and do stock trading based on its thinking.

Lex Fridman 1:09:10 Yeah.

Max Tegmark 1:09:10 It's not an agent. An intelligent agent is something that takes in information from the world, processes it to figure out what action to take based on its goals that it has, and then does something back on the world. But once you have an API for- for example, GPT-4, nothing stops Joe Schmoe and all- and a lot of other people from building real agents which just keep making calls somewhere in some inner loop somewhere to these powerful oracle systems, uh, which makes them themselves much more powerful. That's another kind of, um, unfortunate development, which, uh, I think we would have been better off, uh, delaying. I don't wanna pick on any particular companies. I think they're all under a lot of pressure to make money.

Lex Fridman 1:09:58 Yeah.

Max Tegmark 1:09:59 And, um, again, we- the reason we- we're calling for this pause is to give them all cover to do what they know is the right thing, slow down a little bit at this point. But everything we've talked about I ho- I hope will- can- will help make it clear to people watching this, you know, why these sort of human level tools can cause a gradual acceleration. You keep using yesterday's technology to build tomorrow's technology.

Lex Fridman 1:10:27 Yeah.

Max Tegmark 1:10:27 And when you do that over and over again, you naturally get an explosion. You know, that's the definition of an explosion in science, right? Like, if you have, um, two people and they fall in love, uh, now you have four people, (laughs) and then they can make more babies, and now you have eight people, and then- then you have 16, 32, 64, et cetera. That's- we call that an- a population explosion, whereas just that each- if you- if it's instead free neutrons in a nuclear reaction, that if each one can make more than one, then you get an exponential growth in that, we call it a nuclear explosion.

Lex Fridman 1:11:05 Mm-hmm.

Max Tegmark 1:11:05 All explosions are like that. In an intelligence explosion, this is just exactly the same principle, that some quanti- some amount of intelligence can make more intelligence than that, and then repeat. You always get exponentials.

Lex Fridman 1:11:17 What's your intuition why it does? You mentioned there's some technical reasons why it doesn't stop at a certain point. What's your intuition, and, uh, uh, do you have any intuition why it might stop?

Max Tegmark 1:11:28 It's obviously gonna stop when it bumps up against the laws of physics. There are some things you just can't do, no matter how smart you are, right?

Lex Fridman 1:11:34 Allegedly. (laughs)

Max Tegmark 1:11:37 (laughs) And Seth Lloyd-

Lex Fridman 1:11:37 'Cause we don't know the full-

Max Tegmark 1:11:38 Yeah.

Lex Fridman 1:11:39 ... laws of physics-

Max Tegmark 1:11:40 Yeah.

Lex Fridman 1:11:40 ... yet, right?

Max Tegmark 1:11:41 Seth Lloyd wrote a really cool paper on the physical limits on, uh, computation, for example. If you make it- put too much energy into it, in a finite space, it'll turn into a black hole. (laughs)

Lex Fridman 1:11:52 (laughs)

Max Tegmark 1:11:52 You can't move information around-

Lex Fridman 1:11:53 Yeah.

Max Tegmark 1:11:53 ... faster than the speed of light, stuff like that, but, uh, it's hard to store way more than- than, uh, a modest number of bits per atom, et cetera, but, you know, those limits are just astronomically above, like, 30 orders of magnitude above where we are now, so, um... bigger, different, bigger jump in intelligence than if you go from a- from an ant to a human. I think, of course, what we want to do is have- have a controlled thing. Like a nuclear reactor, you put moderators in to make sure exactly it doesn't blow up out of control, right?

Lex Fridman 1:12:31 Mm-hmm.

Max Tegmark 1:12:32 When we do, um... experiments with biology and cells and so on, you know, we also try to make sure it doesn't get out of control. Um, eh, eh, we can do this with AI, too. The thing is, we haven't succeeded yet. And Moloch is exactly f- doing the opposite, just fueling, just egging everybody on, "Faster, faster, faster, or the other company is gonna catch up with you, or the other country is gonna catch up with you." ............................ we have to want this stuff. We have... and you- and I don't believe in this just asking people to look into their hearts and do the right thing. It's easier for others to say that, but, like, if- if you're in the situation where your company is gonna get screwed, m- if you...... by other companies, they are not stopping. You know, you're putting people in a very hard situation. The, the, the right thing to do is change the whole incentive structure instead. And, and this is not an old mo- uh, com- maybe I should say one more thing about this, 'cause Moloch has been around as humanity's number one or number two enemy since the beginning of civilization, and, and we came up with some really c- cool countermeasures. Like, first of all, already over 100,000 years ago, evolution realized that it was, uh, very unhelpful that people kept killing each other all the time.

Lex Fridman 1:13:55 Yeah.

Max Tegmark 1:13:55 So, it genetically gave us compassion and made it so that it ... Like, if you get two drunk dudes getting into a pointless bar fight, they might give each other black eyes, but ... And they have a lot of inhibition towards c- towards just killing each other. J- that's a g- ... And similarly, if you find a baby lying on the street when you go out for your morning jog tomorrow, you're gonna stop and pick it up, right? Even though it maybe make you late for your next podcast. So, evolution gave us these genes that make our own egoistic incentives more aligned with what's good for the greater group we're part of, right? And then, uh, as we got a bit more s- sophisticated and developed language, we invented gossip, which is also a fantastic anti-Moloch.

Lex Fridman 1:14:46 Mm-hmm.

Max Tegmark 1:14:46 Right? 'Cause now, it, it really discourages liars, moochers, cheaters, because it- it ... Their own incentive now is not to do this, because word quickly gets around and then suddenly (laughs) people aren't gonna invite them to their dinners anymore, and, or trust them. And then when we got still more sophisticated and bigger societies, you know, invented the legal system.

Lex Fridman 1:15:10 Mm-hmm.

Max Tegmark 1:15:11 Where even strangers who didn't, couldn't rely on gossip and, and things like this, would treat each other ... Would have an incentive. Now, those guys in the bar fight, even if they a- someone is so drunk that he a- actually wants to kill the other guy, he also has a little thought in the back of his head that, "You know, do I really wanna spend the next 10 years eating, like, really crappy food in a small room? Uh, (laughs) I'm just gonna ch- I'm just gonna chill out," you know? So ... And we, we similarly have tried to give these incentives to our corporations, by having, having regulation and all sorts of oversight, so that their incentives are aligned with the greater good. We've tried really hard. Um, and, um, the, the big problem that we're failing now is not that we haven't tried before, but it's just that the tech is growing much f- it's developing much faster than the regulators have been able to keep up, right? So, regulators, uh, it's kind of comical. The European Union right now is doing this AI Act, right? Which ... And f- in the beginning, they had a little opt-out exception that GPT-4 would be completely excluded from regulation. Brilliant idea.

Lex Fridman 1:16:21 What's the logic behind that?

Max Tegmark 1:16:24 Some lobbyists, um, pushed successfully for this. So, we were actually quite involved with the Future of Life Institute. Um, Mark Brakel, Risto Eugh, Anthony Aguirre, and others, you know, were quite involved with, um, talking to var- educating various people involved in this process about these general-purpose AI models coming, and pointing out that they would become the laughingstock if they didn't put it in. So, it ... The friends started pushing for it. It got put into the draft, and it looked like all was good, and then there was a huge k- counter-push from lobbyists. Uh, there were more lobbyists s- in Brussels from tech companies than from oil companies, for example. And, it looked like it might, it was gonna maybe get taken out again, and now GPT-4 happened, and I think it's gonna stay in. But this just shows, you know, Moloch can be defeated, but the, the, the challenge we're facing is that the tech is generally much faster than what the policy-makers are.

Lex Fridman 1:17:23 Mm-hmm.

Max Tegmark 1:17:24 And a lot of the policy-makers also don't have a, a tech background, uh, so it's, uh, you know, we really n- need to work hard to educate them on, on how, uh, on what's taking place here. So, so, we're getting to the situation where the first kind of non- ... So, uh, you know, I define artificial intelligence just as non-biological intelligence, right?

Lex Fridman 1:17:43 Mm-hmm.

Max Tegmark 1:17:44 And by that definition, a, a company, a corporation is also an artificial intelligence because the corporation isn't its humans, it's a system. If its CEO decides, the CEO of a tobacco company decides one morning that she or he doesn't wanna sell cigarettes anymore, they'll just put another CEO in there. It's not enough to align the incentives of individual people, or in- align individual computers' incentives to their owners, which is what technical AI safety research is about. You also have to align the incentives of corporations w- with a greater good. And some corporations have gotten so big and so powerful very quickly, that, uh, in many cases, their lobbyists instead align the regulators to what they want rather than the other way around. It's a classic regulatory capture.

Lex Fridman 1:18:35 All right. Is, is, uh, the thing that the slowdown hopes to achieve is give enough time to the regulators to catch up, or enough time to the companies themselves to breathe and understand how to do AI safety correctly?

Max Tegmark 1:18:48 I think both, but I think the, the vision, the path to success I see is first you give a breather actually to, to the people in these companies, their leadership who wants to do the right thing, and they all have safety teams and so on in their companies. Give them a chance to get together with the other companies, uh, w- and the outside pressure can also help catalyze that, right?

Lex Fridman 1:19:11 Mm-hmm.

Max Tegmark 1:19:12 Uh, and, and, uh, work out what is it that's ... What are the reasonable, uh, safety requirements one should put on future systems before they get rolled out?... there are a lot of people also in academia and elsewhere outside of these companies wh- who can be brought into this and have a lot of very good ideas. And then, um, I think it's very realistic that within six months, you can get these people coming up with, "So here's a white paper. Here's where we all think is reasonable." Um,

Max Tegmark 1:19:44 you know, you didn't, we... Just because cars killed a lot of people, they didn't ban cars, but they got together, a bunch of people, and decided, you know, in order to be allowed to sell a car, it- it has to have a seatbelt in it.

Lex Fridman 1:19:53 Mm-hmm.

Max Tegmark 1:19:55 There... The analogous things that you can start requiring of future AI systems so that they, uh, are safe. And, uh, o- o- once this hav- this heavy lifting, this intellectual work has been done by experts in the field, which can be done quickly, I think it's begin- going to be quite easy to get policymakers to, to see, yeah, this is a good idea. And it's- it's, you know, for the fi- for the companies to fight Moloch, they want, and I- I believe Sam Altman has explicitly called for this, they want the regulators to actually adopt it so that their competition is gonna abide by it too, right? You don't want, uh, you don't wanna be en- enacting all these principles and you abide by them, and then you're, there's this one little company that doesn't sign onto it, and then now they can gradually overtake you.

Lex Fridman 1:20:48 Mm-hmm.

Max Tegmark 1:20:49 Then the companies will get... Be able to sleep s- secure knowing that everybody's playing by the same rules.

Lex Fridman 1:20:56 So do you think it's possible to develop guardrails that keep the systems from, uh, from basically (laughs) damaging, irreparably, humanity-

Max Tegmark 1:21:08 Mm-hmm.

Lex Fridman 1:21:09 ... while still enabling sort of the capitalist-fueled competition between-

Max Tegmark 1:21:13 Yeah.

Lex Fridman 1:21:13 ... companies as they develop how to best make money with this AI? You think there's a balancing-

Max Tegmark 1:21:18 Totally.

Lex Fridman 1:21:18 ... that's possible?

Max Tegmark 1:21:19 Absolutely. I mean, we've seen that in many other sectors where you've had the free market produce quite good things without, uh, causing particular harm. Um, when the guardrails are there and they work, you know, capitalism is a very effect- good way of optimizing for- for just getting the same thing done more efficiently. It was-

Lex Fridman 1:21:39 It's just-

Max Tegmark 1:21:39 But it was good, you know, and like in hindsight, I have never met anyone, even, even on parties way over on the right in- in- in any country, who- who think it was a bad, thinks it was a terrible idea to b- to ban child labor, for example.

Lex Fridman 1:21:55 Yeah, but it seems like this particular technology has gotten so good so fast, become powerful to a degree where you could see, in the near term, the ability to make a lot of money-

Max Tegmark 1:22:06 Yeah.

Lex Fridman 1:22:07 ... and to put guardrails, to develop guardrails quickly in that kind of context seems to be tricky. It's not, uh, similar to cars or child labor. It seems like-

Max Tegmark 1:22:17 Mm-hmm.

Lex Fridman 1:22:17 ... the opportunity to make a lot of money here very quickly is right here before us.

Max Tegmark 1:22:22 Yeah, again, there's this cliff.

Lex Fridman 1:22:24 Yeah.

Max Tegmark 1:22:26 Uh, and supposedly it gets-

Lex Fridman 1:22:26 (laughs) It gets quite scenic.

Max Tegmark 1:22:27 ... uh, the closer to the cliff that you go-

Lex Fridman 1:22:28 Yeah.

Max Tegmark 1:22:29 ... the more, the more-

Lex Fridman 1:22:29 (laughs)

Max Tegmark 1:22:30 ... the g- the more money there is, the more gold ingots that are on the ground you can pick up-

Lex Fridman 1:22:33 Yeah.

Max Tegmark 1:22:34 ... or whatever, so you wanna drive there very fast. But it's not in anyone's incentive that we go over the cliff, and it's not like everybody's in their own car. All the cars are connected together with a chain.

Lex Fridman 1:22:43 Yeah.

Max Tegmark 1:22:43 So if anyone (laughs) goes over, they'll start dragging others down, the others down too. And so ultimately, it's in the selfish interests also of the people in the companies to, uh, to, uh, slow down when the, when you can start seeing the contours of the cliff there in front of you, right? And the problem is that, um, even though the people who are building the technology and the CEOs, they really get it, the shareholders and these other market forces, they are people who don't, honestly-

Lex Fridman 1:23:14 Mm-hmm.

Max Tegmark 1:23:14 ... u- u- understand that the cliff is there. They usually don't... You have to get quite into the weeds to really appreciate h- how powerful this is and how fast, and a lot of people are even still stuck, again, in this idea that intelle- in this carbon chauvinism, as I like to call it, that you can only have our level of intelligence in humans, that there's something magical about it. Whereas the people in the tech companies who build this stuff, they all realize that intelligence is information processing of a certain kind, and it really doesn't matter at all whether the information is processed by carbon atoms in neurons in brains or by silicon atoms in some technology we build. So you brought up capitalism earlier, and there are a lot of people who love capitalism and a lot of people who really, really don't. And it s- it struck me recently that, uh, what's happening with capitalism here is exactly analogous to the way in which superintelligence might wipe us out. So, you know, you know I studied economics for my undergrad, Stockholm School of Economics? Yay? (laughs)

Lex Fridman 1:24:28 (laughs) Well, no. No, why? Tell me.

Max Tegmark 1:24:31 So I was very interested in how- how- how you could use market forces to just get stuff done more efficiently, but give the right incentives to markets so that it wouldn't do really bad things. So Dylan Hadfield-Mell, who's a, a professor and colleague of mine at MIT, wrote this really interesting paper with some collaborators recently, where they proved mathematically that if you just op- take one goal that you just optimize for, on and on and on indefinitely that you think is gonna bring you in the right direction, what basically always happens is, in the beginning, it will t- make things better for you. But if you keep going, at some point, it's gonna start making things worse for you again. And then gradually, it's gonna make it really, really terrible. So just as a simple... The way I think of the proof is, suppose you wanna go from here back to Austin, for example, and you're like-... okay, yeah, let's just, let's go south. But you put in exactly the right, sort of the right direction, so optimize that south as possible. You get closer and closer to Austin, but, uh, you, you, there was always some little me- or mer- uh, so you, you're not going exactly towards Austin, but you get pretty close. But eventually, (laughs) you start going away again, and eventually you're gonna be leaving the solar system. (laughs) .

Lex Fridman 1:25:50 Yeah.

Max Tegmark 1:25:50 And they, they proved it, it's a beautiful mathematical proof. This happens generally, uh, and this is very important for AI because for, for, even though Stuart Russell has wr- written a book and given a lot of talks o- on why it's a bad idea to have AI just blindly optimize something, that's what pretty much all our systems do.

Lex Fridman 1:26:10 Yeah.

Max Tegmark 1:26:10 We have something called the loss function that we're just minimizing, or a reward function we're just minimize, maximizing. And, um, capitalism is exactly like that too. We wa- we wanted to get stuff done more efficiently, the people wanted, so introduce the free market. Things got done much more efficiently than they did in, uh, in, say, communism. Right? And it got better. But then it just kept optimizing it, uh, and kept optimizing, and you got ever bigger companies and ever more efficient information processing, now also very much powered by IT. And, uh, eventually a lot of people are beginning to feel, "Wait, uh, we're kind of optimizing a bit too much. Like, why did we just chop down half the rainforest?" You know? Uh, and why, why did suddenly these regulators get captured by lobbyists and so on? It's just the same optimization that's been running for too long. If you have an AI that actually has power over the world, and you just give it one goal and, and just, like, keep optimizing that, m- most likely everybody's gonna be like, "Yay, this is great," in the beginning, "Things are getting better." But, um, it's almost impossible to give it exactly the right direction to optimize in. And then eventually all h- all hell b- breaks loose, right? Nick Bostrom and others have given, uh, examples that sound quite silly, like what if you just wanna, like, tell it to cure cancer or something and that's