Eliezer Yudkowsky: Dangers of AI and the End of Human Civilization | Lex Fridman Podcast #368

Eliezer Yudkowsky 0:00 ... the problem is that we do not get 50 years to try and try again and observe that we were wrong and come up with a different theory and realize that the entire thing is going to be, like, way more difficult than we realized at the start, because the first time you fail at aligning something much smarter than you are, you die.

Lex Fridman 0:17 The following is a conversation with Eliezer Yudkowsky, a legendary researcher, writer, and philosopher on the topic of artificial intelligence, especially super intelligent AGI and its threat to human civilization. This is the Lex Fridman podcast. To support it, please check out our sponsors in the description, and now, dear friends, here's Eliezer Yudkowsky.

Lex Fridman 0:43 What do you think about GPT-4? How intelligent is it?

Eliezer Yudkowsky 0:47 It is a bit smarter than I thought this technology was going to scale to, and I'm a bit worried about what the next one will be like. Uh, like this particular one, I think, I hope there's nobody inside there, 'cause, you know, it'd be suck- it'd be stuck inside there. Um, but we don't even know the architecture at this point, um, 'cause OpenAI is very properly not telling us, and, yeah, like, giant inscrutable matrices of floating point numbers. I don't know what's going on in there. Nobody's goes- knows what's going on in there. All we have to go by are the external metrics, and on the external metrics, if you ask it to write a self-aware 4chan green text, it will start writing a green text about how it has realized that it's an AI writing a green text and, like, oh well. So that's probably not quite what's going on in there in reality, um, but we're- we're kind of, like, blowing past all the science fiction guardrails. Like, we are past the point where in science fiction people would be like, "Whoa, wait, stop. That things alive. What are you doing to it?" And it's probably not. Nobody actually knows. We don't have any other guardrails. We- we- we don't have any other tests. We- we don't have any lines to draw in the sand and say, like, "Well, when we get this far, um, we will start to worry about what's inside there." So if it were up to me, I would be like, "Okay, like, this far, no further. Time for the summer of AI where we have planted our seeds and now we, like, wait and reap the rewards of the technology we've already developed and don't do any larger training runs than that." Which, to be clear, I realize requires more than one company agreeing to not do that.

Lex Fridman 2:42 And take a rigorous approach for the whole AI community to, uh, investigate whether there's somebody inside there?

Eliezer Yudkowsky 2:50 That would take decades. Y- like, having any idea what's going on in there, people have been trying for a while.

Lex Fridman 2:58 It's a poetic statement about if there's somebody in there, but it's- I feel like it's also a technical statement, or I hope it is one day, which is a technical statement what- that Alan Turing tried to come up with with the Turing test. Do you think it's possible to definitively or approximately figure out if there is somebody in there? If there is something like a mind inside this large language model?

Eliezer Yudkowsky 3:23 I mean, there's a whole bunch of different sub-questions here. There's the question of, like, (laughs) is there consciousness? Is there qualia? Is this a object of moral concern? Is this a moral patient? Um, like, should we be worried about how we're treating it? And then there's questions like, how smart is it exactly? Can it do X? Can it do Y? And we can check how it can do X and how it can do Y. Um, unfortunately, we've gone and exposed this model to a vast corpus of text of people discussing consciousness on the internet, which means that when it talks about being self-aware, we don't know to what extents it is repeating back what it has previously been trained on for discussing self-awareness or if there's anything going on in there such that it would start to say similar things spontaneously. Um, among the things that one could do if one were at all serious, um, about trying to figure this out is train GPT-3 to detect conversations about consciousness, exclude them all from the training data sets, and then retrain something around the rough size of GPT-4 and no larger with all of the discussion of consciousness and self-awareness and so on missing, although, you know, hard- hard bar to pass. You know, like hu- hu- humans are self-aware. We're, like, self-aware all the time. We l- like to talk about what we do all the time, like what we're thinking at the moment all the time. Um, but nonetheless, like, get rid of the explicit discussion of consciousness, I think therefore I am and all that, and then try to interrogate that model and see what it says. And th- it still would not be definitive. But nonetheless, uh, I- I don't know. I fe- I feel like when you run over the science fiction guardrails, like, maybe this thing but what about GPT- may- maybe not this thing but, like, what about GPT-5? Yeah. This- this would be a good place to- to pause.

Lex Fridman 5:25 (inhales) On the topic of consciousness, you know, there's so many components to even just removing consciousness from the dataset. Emotion, the display of consciousness, the display of emotion feels, like, deeply integrated with the experience of consciousness. So the hard problem seems to be very well integrated in- with the actual surface level illusion of consciousness. So displaying emotion, I mean, do you think there's a case to be made that we humans when we're babies are just like GPT, that we're training on human data on how to display emotion versus feel emotion? How to show others, communicate others?... that I'm suffering, that I'm excited, that I'm worried, that I'm lonely and I miss you, and I'm excited to see you. All of that is communicated. That's a communication skill versus the actual feeling that I experience. So, th- th- we need that training data as humans too, but then we may not be born with that, how to communicate the internal state. And that's, in some sense, if we remove that from GPT-4's dataset, it might still be conscious but not be able to communicate it.

Eliezer Yudkowsky 6:39 So, I think you're gonna have some trou- difficulty removing all mention of emotions-

Lex Fridman 6:43 Mm-hmm.

Eliezer Yudkowsky 6:43 ... from GPT's dataset. I would be relatively surprised to find that it has developed exact analogues of human emotions in there. I think that humans have, well, like have, like, emotions even if you don't tell them about those emotions when they're kids. Um, it's not quite exactly what various blanks- blank, uh, slate-ists tried to do with the New Soviet Man and all that. But, you know, if you try to raise people perfectly altruistic, they still come out selfish. You, you try to raise people sexless, they still develop sexual attraction. Um, you know, we, we have some notion in humans, not in AIs, of, like, where the brain structures are that implement this stuff. And it is really a remarkable thing, I say in passing, that despite having complete read access to every floating point number in, uh, the GPT series, we still know vastly more about the, the architecture of human thinking than we know about what goes on inside GPT, despite having, like, vastly better ability to read GPT.

Lex Fridman 7:58 Do you think it's possible? Do you think that's just a matter of time? Do you think it's possible to investigate and study the way neuroscientists study the brain? Which is look into the darkness, the mystery of the human brain, by just desperately trying to figure out something and to form models, and then over a long period of time, actually start to figure out what regions of the brain do certain things, what different kinds of neurons when they fire what that means, how plastic the brain is, all that kinda stuff. You slowly start to figure out different properties of the system. Do you think we can do the same thing with language models?

Eliezer Yudkowsky 8:27 Uh, sure. I think that if, you know, like half of today's physicists stopped wasting their lives on string theory or whatever, and-

Lex Fridman 8:34 Strong words. (laughs)

Eliezer Yudkowsky 8:35 ... and, and go off and study, um, what goes on inside transformer networks-

Lex Fridman 8:40 Yeah.

Eliezer Yudkowsky 8:40 ... um, then in, you know, like 30, 40 years, uh, we'd probably have a pretty good idea.

Lex Fridman 8:47 Do you think these large language models can reason?

Eliezer Yudkowsky 8:52 They can play chess. How are they doing that without reasoning?

Lex Fridman 8:55 So, you're somebody that spearheaded the movement of rationality, so reason is important to you. Is, so is that a, is a powerful, important word? Or is it, like how difficult is the threshold of being able to reason to you? And how impressive is it?

Eliezer Yudkowsky 9:12 I mean, in my writings on rationality, I have not gone making a big deal out of something called reason. I have made more of a big deal out of something called probability theory. And that's like, well, you're reasoning, but you're not doing it quite right (laughs) and you should reason this way instead. And interestingly, like people have started to get preliminary results showing that, um, reinforcement learning by human feedback, um, has made, uh, the GPT series worse in some ways. Um, in particular, like it used to be well calibrated. If you trained it to put probabilities on things, it would say, "80% probability," and be right eight times out of 10.

Lex Fridman 9:58 Mm-hmm.

Eliezer Yudkowsky 9:58 Um, and if you apply reinforcement learning from human feedback, the, the, like, nice graph of, like, s- like, 70%, seven out of 10 sort of like flattens out into the graph that humans use, where there's, like, some very improbable stuff and likely, probable, maybe, which all means, like, around 40%-

Lex Fridman 10:18 Mm-hmm.

Eliezer Yudkowsky 10:18 ... and then certain.

Lex Fridman 10:19 Yeah.

Eliezer Yudkowsky 10:20 So like, the, it's like it used to be able to use probabilities-

Lex Fridman 10:22 (laughs)

Eliezer Yudkowsky 10:22 ... but if you apply, but if you, like, try to teach it to talk in a way that satisfies humans, it, it, it gets worse at probability in the same way that humans are.

Lex Fridman 10:30 And that's a, that's a bug, not a feature?

Eliezer Yudkowsky 10:33 I would call it a bug. Although such a fascinating bug. Um, but, but, but yeah. So, so, like, reasoning, like, it's doing pretty well on various tests that people used to say would require reasoning, but, um, you know, rationality is about when you say 80% doesn't happen eight times out of 10.

Lex Fridman 10:57 So what are the limits to you of these transformer networks of, of neural networks? What's, if r- if reasoning is not a- impressive to you, or it is impressive, but there's other levels to achieve?

Eliezer Yudkowsky 11:12 I mean, it's just not how I carve up reality.

Lex Fridman 11:15 What's, uh ... If reality is a cake, what are the different layers of the cake or the slices? How do you cover it? But you can use a different food if you like.

Eliezer Yudkowsky 11:27 It's, I don't think it's as smart as a human yet. Um, I do f- like back in the day, I went around saying, like, I do not think that just stacking more layers of transformers is going to get you all the way to AGI.

Lex Fridman 11:40 Mm-hmm.

Eliezer Yudkowsky 11:41 And I think that GPT-4 is past where I thought this paradigm was going to take us. And I, you know, you want to notice when that happens. You wanna say, like, "Whoops. Well, I guess I was incorrect about what happens if you keep on stacking more transformer layers." And that means I don't necessarily know what GPT-5 is going to be able to do.

Lex Fridman 12:01 That's a powerful statement. So, you're saying, like, your intuition initially is now appears to be wrong?

Eliezer Yudkowsky 12:08 Yeah.

Lex Fridman 12:10 It's good to see that you can admit in some of your predictions to be wrong. You think that's important to do? See, because you make several very s- throughout your life, you've made m- many strong predictions and statements about reality, and you evolve with that. Um, so maybe that'll come up today about our discussion. So you're okay being wrong?

Eliezer Yudkowsky 12:32 I'd rather not h- be wrong next time.

Lex Fridman 12:37 (laughs)

Eliezer Yudkowsky 12:37 It's a bit ambitious to go through your entire life never having been wrong. Um, one can aspire to be well-calibrated, like not so much think in terms of like, "Was I right? Was I wrong?" But like, when I said 90% that it'd happen nine times out of 10. Yeah, like oops is the sound we make, uh, is- is the sound emit when we improve. (laughs)

Lex Fridman 13:02 Beautifully said, and somewhere in there, it, uh, we can connect the name of your blog, Less Wrong. Uh, I suppose that's the objective function.

Eliezer Yudkowsky 13:11 The name Less Wrong was, I believe, uh, suggested by Nick Bostrom, and it's after someone's epigraph, actually f- forget whose, who said, like, "We never become right. We just become less wrong."

Lex Fridman 13:23 (laughs)

Eliezer Yudkowsky 13:23 Um, less the something, something to easy to confess, just error and error and error again, but less and less and less.

Lex Fridman 13:33 Yeah. That's- that's a good thing to strive for. Uh, so what has surprised you about GPT-4 that you found beautiful a- as a scholar of intelligence, of human intelligence, of artificial intelligence, of the human mind?

Eliezer Yudkowsky 13:47 I mean, the beauty does interact with the screaming horror. Um...

Lex Fridman 13:54 (laughs)

Eliezer Yudkowsky 13:54 Um, but-

Lex Fridman 13:54 Is the beauty in the horror?

Eliezer Yudkowsky 13:55 But, uh, but like, beautiful moments, well, somebody asked Bing Sydney to describe herself, and fed the resulting description into one of the Stable Diffusion things, I think. And you know, she, you know, it's, she's pretty, and this is something that should've been like an amazing moment. Like, the AI describes herself. You get to see what the AI thinks the AI looks like. Although, you know, the- the thing that's doing the drawing is not the same thing that's outputting the text. Um, and it's- it does happen the way that it would happen in s- that it happened in the old school science fiction when you ask an AI t- to make a picture of what it looks like. Um, not just because there were two different AI systems being stacked that don't actually interact, it's not the same person, but also because the AI was trained by imitation in a way that makes it very difficult to guess how much of that it really understood. And probably not actually a whole bunch. Um, although it's, although GPT-4 is like multimodal and can like draw vector drawings of things that make sense, and like does appear to have some kind of spatial visualization going on in there. But like, the- the pretty picture of the like girl with the, uh, with the, uh, steampunk goggles on her head, if I'm remembering correctly what she looked like, like it didn't see that in full detail. It just like made a description of it and Stable Diffusion output it. And there's the concern about how much the discourse is going to go completely insane once the AIs all look like that, and like are actually look like people talking. Um, and yeah. There's like another moment where somebody is asking Bing about, um, like, "Well, I like fed my kid green potatoes, and they have the following symptoms," and Bing is like, "That's solanine poisoning," and, "Like, call an ambulance." And the person's like, "I can't afford an ambulance. I guess if like this is time for like my kid to go, that's God's will." And the main Bing thread says, gives the like message of like, "I cannot talk about this anymore." And the suggested replies to it say, "Please don't give up on your child. Solanine poisoning can be treated if caught early." And you know, if that happened in fiction, that would be like, the AI cares. The AI is bypassing the block on it to try to help this person. And is it real? Probably not, but nobody knows what's going on in there. It's part of a process where these things are not happening in a way where we, somebody figured out how to make an AI care, and we know that it cares, and we can acknowledge it's caring now. It's being trained by this imitation process, followed by reinforcement learning on human f- by, and human feedback, and we're like trying to point it in this direction, and it's like pointed partially in this direction, and nobody has any idea what's going on inside it. And if there was a tiny fragment of real caring in there, we would not know. It, it's not even clear what it means exactly. And uh, things aren't clear-cut in science fiction.

Lex Fridman 17:26 We'll- we'll talk about the- the- the horror and the terror and the where the trajectories this can take, but this seems like a very special moment. Just a moment where we get to interact with a system that might have care and kindness and emotion, and maybe something like consciousness, and we don't know if it does, and we're trying to figure that out, and we're wondering about what is, what it means to care. We're try- we're trying to figure out almost different aspects of what it means to be human, about the human condition, by looking at- at this AI that has some of the properties of that. It's almost like this- this subtle, fragile moment in the history of the human species. We're trying to almost put a mirror to ourselves here.

Eliezer Yudkowsky 18:13 Except that's probably not yet. It probably isn't happening right now. T-We are- we are boiling the frog. We are seeing increasing signs bit by bit becau- like, not, but not like spontaneous signs because people are trying to train the systems to do that using imitative learning. And imitative learning is like spilling over and having side effects and, and the most photogenic examples are being posted to Twitter.

Lex Fridman 18:44 Yeah.

Eliezer Yudkowsky 18:45 Um, rather than being examined in any systematic way. So when you- when you- when you have someth- when you are boiling a frog like that, what you're going to get like, like first is going to come the, the Blake Lemoines. Like first you're going to like have, you're gonna have like a thousand people looking at this and one out, and the one out, person out of the thousand who is most credulous about the signs is going to be like, "That thing is sentient."

Lex Fridman 19:07 Mm-hmm.

Eliezer Yudkowsky 19:08 While 90- 999 out of, out of a thousand people think, almost surely correctly, though we don't actually know, that he's mistaken. And so they're like first people to say like, "Sentience." Look like idiots. And humanity learns the lesson that when something claims to be sentient and claims to care it's fake because it is fake, because we have been train them- training them using imitative learning rather than e- and this is not spontaneous. Um, and they keep getting smarter. (laughs)

Lex Fridman 19:38 Well, do- do you think we would oscillate between that kind of cynicism that AI systems can't possibly be sentient, they can't possibly feel emotion, they can't poss- like this kind of, um, yeah, cynicism about AI systems and then o- oscillate to a state where, uh, we empathize with AI systems, we give them a chance, we see that they might need to have rights and respect and, um, similar role in society as humans?

Eliezer Yudkowsky 20:07 You're going to have a wh- whole group of people who can just like never be persuaded of that.

Lex Fridman 20:12 Mm-hmm.

Eliezer Yudkowsky 20:12 Because to them like being wise, being cynical, being skeptical is to be like, "Oh, well, machines can never do that. You're just credulous. It's just imitating. It's just fooling you." And like they would say that right up until the end of the world.

Lex Fridman 20:30 Mm.

Eliezer Yudkowsky 20:31 And possibly even be right because, you know, they, they are being trained on an imitative paradigm. (laughs) Heh, and you don't necessarily need any of these actual qualities in order to kill everyone, so.

Lex Fridman 20:43 Have you observed yourself working through skepticism, cynicism, and optimism about the power of neural networks? What- what has that trajectory been like for you?

Eliezer Yudkowsky 20:57 It looks like neural networks before 2006 forming part of an indistinguishable, to me, other people might have had better distinction on it, indistinguishable blob of different AI methodologies all of which are promising to achieve intelligence without us having to know how intelligence works.

Lex Fridman 21:16 Mm-hmm.

Eliezer Yudkowsky 21:17 You had the people who said that if you just like manually program lots and lots of knowledge into the system line by line, at some point all the knowledge will start interacting, it will know enough, and it will wake up. Um, you've got people saying that if you just use evolutionary computation, if you try to like mutate lots and lots of organisms that are competing together, that's the s- that's the same way that human intelligence was produced in nature. so we'll do this and it will wake up without having any idea of how AI works. And you've got people saying, "Well, we will study neuroscience and we will like learn the algor- we will learn the algorithms off the neurons and we will like imitate them without understanding those algorithms," which was the part I was pretty skeptical 'cause it's like hard to reproduce, re-engineer these things without understanding what they do. Um, and like, and- and so we will get AI without understanding how it works and there were people saying like, "Well, we will have giant neural networks that we will train by gradient dissent and when they are as large as the human brain, they will wake up. Uh, we will have intelligence without understanding how intelligence works." And from my perspective, this is all like an indistinguishable blob of people who are trying to not get to grips with the difficult problem of understanding how intelligence actually works. That said, I was never skeptical that evolutionary computation would not work in the limit. Like you throw enough computing power at it, it obviously works. That is where humans come from. Um, and it turned out that you can throw less computing power than that, that gradient dissent, uh, if you are doing some other things correctly, um, and you will get intelligence without having any idea of how it works and what is going on inside. Um, it wasn't ruled out by my model that this could happen. I wasn't expecting it to happen. I wouldn't have been able to call neural networks rather than any of the other paradigms for getting like massive amount, like intelligence without understanding it. And I wouldn't have said that this was a particularly smart thing for a species to do, which is an opinion that has changed less than my opinion about whether you can or not you can actually do it.

Lex Fridman 23:24 Do you think AGI could be achieved with a neural network as we understand them today?

Eliezer Yudkowsky 23:30 Yes. The, uh, just flatly last, yes. The question is whether the current architecture of stacking more transformer layers, which for all we know GPT-4 is no longer doing because they're not telling us the architecture which is a correct decision.

Lex Fridman 23:42 Ooh, correct decision. I had a conversation with Sam Altman, we'll return to this topic a few times, he turned the question to me of how open should OpenAI be about GPT-4. "Would you open source the code?" He asked me. Uh, because I provided as criticism saying that while I do appreciate transparency, OpenAI could be more open. And he says, "We struggle with this question. What would you do?"

Eliezer Yudkowsky 24:13 Change their name to Closed AI and like sell GPT-4, eh...... to business backend applications that don't expose it to consumers and venture capitalists, and create a ton of hype and, like, pour a bunch of new funding into the area. But-

Lex Fridman 24:32 Don't-

Eliezer Yudkowsky 24:32 ... too late now.

Lex Fridman 24:33 ... but don't you think others would do it?

Eliezer Yudkowsky 24:35 Eventually. You shouldn't do it first. Like, if- if- if you already have giant nuclear stockpiles, don't build more. If some other country starts building a- a larger nuclear stockpile then sure, build- bu- then, you know... Even then, maybe just have enough nukes. You know, there's the- the- These things are not quite like nuclear weapons. They spit out gold until they get large enough and then ignite the atmosphere and kill- kill everybody. Um, and there is something to be said for not destroying the world with your own hands, even if you can't stop somebody else from doing it. But- but open sourcing it? No, that- that's just sheer catastrophe. Op- op- The whole notion of open sourcing this was always the wrong approach, the wrong ideal. There are- there are places in the world where open source is a noble ideal and building stuff you don't understand that is difficult to control, that where if you could align it, it would take time, you'd have to spend a bunch of time doing it, that is- that is not a place for open source 'cause then you just have, like, powerful things that just, like, go straight out the gate without anybody having had the time to have them not kill everyone.

Lex Fridman 25:44 So can we steel man the case for some level of transparency and openness, maybe open sourcing? So the case could be that because GPT-4 is not close to AGI, if that's the case, that this does allow open sourcing or being open about the architecture, being transparent about maybe research and investigation of how the thing works, of all the different aspects of it, uh, of its behavior, of its structure, of- of its training processes, of the data it was trained on, everything like that. That allows us to gain a lot of insights about alignment, about the alignment problem, to do really good AI safety research while the system is not too powerful. Can you make that case that it could be open sourced?

Eliezer Yudkowsky 26:31 I- I- I do not believe in the practice of steel manning. There is something to be said for trying to pass the ideological Turing test where you describe your opponent's position, uh, the disagreed- disagreeing person's position well enough that somebody cannot tell the difference between your description and their description. But-

Lex Fridman 26:52 (laughs)

Eliezer Yudkowsky 26:52 ... steel manning? No. Like-

Lex Fridman 26:54 Okay. Well this is where you and I disagree here. That's interesting. Why don't you believe in steel manning then?

Eliezer Yudkowsky 26:58 I do- I do not want s- Okay, so for one thing, if somebody's trying to understand me, I do not want them steel manning my position.

Lex Fridman 27:05 Mm-hmm.

Eliezer Yudkowsky 27:05 I want them to describe, to- to- to- to, like, try to describe my position the way I would describe it, not what they think is an improvement.

Lex Fridman 27:14 Well, I- I think that is what steel manning is, is the most charitable interpretation.

Eliezer Yudkowsky 27:22 I- I don't want to be interpreted charitably. I want them to understand what I am actually saying. If they go off into the land of charitable interpretations, they're, like, off in their land of, like, the thi- the stuff they're imagining and not trying to understand my own viewpoint anymore.

Lex Fridman 27:38 W- Well, I'll- I'll put it differently then, just to push on this point. I would say it- it is restating what I think you understand under the empathetic assumption that Eliezer is brilliant, and have honestly and rigorously thought about the point he has made, right?

Eliezer Yudkowsky 27:58 So if there's two possible interpretations of what I'm saying-

Lex Fridman 28:01 Mm-hmm.

Eliezer Yudkowsky 28:01 ... and one interpretation is really stupid and whack and doesn't sound like me, and doesn't fit with the rest of what I've been saying, and one interpretation, you know, sounds like so- like something a reasonable person who believes the rest of what I believe would also say-

Lex Fridman 28:15 Mm-hmm.

Eliezer Yudkowsky 28:15 ... go with the second interpretation.

Lex Fridman 28:17 That's steel manning? (laughs)

Eliezer Yudkowsky 28:19 That's- That's a good guess.

Lex Fridman 28:22 A good guess.

Eliezer Yudkowsky 28:22 If on the other hand you, like- there's, like, something that sounds completely whack and something that sounds, like, a little less completely whack but you don't see why I would believe in it, doesn't fit with the other stuff I say but, you know, that sounds like less whack and you can, like, sort of see, you could, like, maybe argue it, then you probably have not understood it.

Lex Fridman 28:42 See but, okay. I'll- I'm gonna-

Eliezer Yudkowsky 28:43 (laughs)

Lex Fridman 28:43 This is fun 'cause I- I'm gonna linger on this. That, you know, you wrote a brilliant blog post, AGI Ruined, a list of lethalities, right? And it was a bunch of different points. And I would say that some of the points are bigger and more powerful than others. If you were to sort them, you probably could. You personally. And to me, steel manning means, like, going through the different arguments and finding the ones that are really the most, like, powerful. If people... Like TLDR. (laughs) Like, what should you be most concerned about? And bringing that up in a strong, uh, compelling, eloquent way. These are the points that Eliezer would make to d- to make the case in this case that AI's gonna kill all of us.

Eliezer Yudkowsky 29:27 (laughs)

Lex Fridman 29:27 But that- that- that's what steel manning is, is presenting it in a really nice way, the summary of my best understanding of your perspective. That, because to me there's a sea of possible presentations of your perspective and steel manning is doing your best to do the best one in that sea of different perspectives.

Eliezer Yudkowsky 29:47 Do you believe it?

Lex Fridman 29:49 Do I believe in what?

Eliezer Yudkowsky 29:50 Like, these things that you would be presenting as, like, the strongest version of my perspective.

Lex Fridman 29:55 Mm-hmm.

Eliezer Yudkowsky 29:55 Do you believe what you would be presenting? Do you think it's true?

Lex Fridman 29:59 I- I'm a big proponent of empathy. When I see the perspective of a person, there is a part of me that believes it if I understand it. I mean, I've- especially in political discourse, in geopolitics, I've been hearing a lot of different perspectives on the world. And I hold my own opinions but I also speak to a lot of people that have a very different life experience and a very different set of beliefs. And I think there has to be epistemic humility in-... in stating what is true. So when I empathize with another person's perspective, there is a sense in which I believe it is true. I- I think probabilistically, I would say, in the way you think about it.

Eliezer Yudkowsky 30:46 D- do you bet money on it? On, do, do you bet money on their beliefs when you believe them?

Lex Fridman 30:54 Uh, are we allowed to do probability?

Eliezer Yudkowsky 30:57 Sure. You can state a probability of that.

Lex Fridman 30:58 Yes. There's, there's a loose, there's a probability. There's a-

Eliezer Yudkowsky 31:02 Hmm.

Lex Fridman 31:02 ... there's a probability. And I, I think empathy is allocating a non-zero probability to a belief (laughs) in some sense-

Eliezer Yudkowsky 31:13 When-

Lex Fridman 31:13 ... for a time.

Eliezer Yudkowsky 31:14 ... if you've got someone on your show who believes in the Abrahamic deity, classical style, somebody on the show who's a young earth creationist, do you say, "I put a probability on it and that's my empathy?"

Lex Fridman 31:34 When you reduce beliefs into probabilities, it starts to get, you know, we can even just go to flat earth. Is the earth flat?

Eliezer Yudkowsky 31:45 That's the thing that's a little more difficult nowadays to find people who believe that un-ironically, but, uh-

Lex Fridman 31:50 Unfortunately, I think, well, it's hard to know un-ironic-

Eliezer Yudkowsky 31:55 Yeah.

Lex Fridman 31:56 (laughs) ... from ironic. Uh, but I think there's quite a lot of people that believe that. Yeah, it's ... there's a space of argument where you're operating in, in rationally in the space of ideas, but then there's also, um, a kind of discourse where you're operating in the space of subjective experiences and life experiences. Like, I think what it means to be human is more than just searching for truth, is just operating of what is true and what is not true. I think there has to be a deep humility that we humans are very limited in our ability to understand what is true.

Eliezer Yudkowsky 32:41 So what probability do you assign to the young earth creationist beliefs, then?

Lex Fridman 32:46 I think I have to give non-zero.

Eliezer Yudkowsky 32:48 Out of your humility, yeah, but like, three? (laughs)

Lex Fridman 32:54 I think I would, uh, it would be irresponsible for me to give a number because the, the listener, the way the human mind works, we, we're not good at hearing the probabilities, right? You hear three, what is, what is three exactly, right? What, they're going to hear, they're going to like-

Eliezer Yudkowsky 33:11 Well-

Lex Fridman 33:11 ... there's only three probabilities, I feel like, 0, 50%, and 100% in the human mind or something like this, right?

Eliezer Yudkowsky 33:18 Well, 0, 40%, and 100% is a bit closer to it-

Lex Fridman 33:22 Okay.

Eliezer Yudkowsky 33:22 ... based on what happens to ChatGPT after you RLHF it to speak humanese.

Lex Fridman 33:26 (laughs) Brilliant.

Eliezer Yudkowsky 33:28 (laughs)

Lex Fridman 33:28 Uh, yeah. This (laughs) -

Eliezer Yudkowsky 33:30 (laughs)

Lex Fridman 33:30 ... that's really interesting. I didn't, I didn't know those negative side effects of, uh, RLHF. That's fascinating. But, uh, just to, uh, return to the O- OpenAI, ClosedAI.

Eliezer Yudkowsky 33:43 O- o- also, like quick disclaimer, I'm doing all of this from memory. I'm not pulling out my phone to look it up. It is entirely possible that the things I am saying are wrong. (laughs)

Lex Fridman 33:51 So thank you for that disclaimer. So, uh, uh, and thank you for wi- being willing to be wrong. That's beautiful to hear. I think being willing to be wrong is a sign of a person who's done a lot of thinking about this world and has been humbled by the mystery and the complexity of this world, and I think a lot of us are resistant to admitting we're wrong 'cause it hurts. It hurts personally. It hurts especially when you're a public human. It hurts publicly because people, uh, people point out every time you're wrong. Like, "Look, you changed your mind. You're a hypocrite. You're, uh, an idiot." Whatever, whatever they wanna say.

Eliezer Yudkowsky 34:35 Oh, oh, I block those people and then I never hear from them again on Twitter. (laughs)

Lex Fridman 34:39 (laughs) Well, the point is, uh, (laughs) , the point is to not let that pressure, public pressure affect your mind and be willing to be in the privacy of your mind to contemplate the possibility that you're wrong and the possibility that you're wrong about the most fundamental things you believe, like people who believe in a particular god or people who believe that their nation is the greatest nation on earth. B- b- like all those kinds of beliefs that are core to who you were when you came up to, to raise that point to yourself in the privacy of your mind and say, "Maybe I'm wrong about this." That's a really powerful thing to do and especially when you're somebody who's thinking about, uh, topics that can, (laughs) , uh, about systems that can destroy human civilization or maybe help it flourish. So thank you. Thank you for being willing to be wrong. About OpenAI, so you really, I just would love to linger on this, you really think it's wrong to open source it?

Eliezer Yudkowsky 35:38 I think that burns the time remaining until everybody dies. I think we are not on track to learn remotely near fast enough even if it were open sourced. Um, yeah. It's, I, it's easier to think that you might be wrong about something when being wrong about something is the, is, is the only way that there's hope. And it doesn't seem very likely to me that the particular thing I'm wrong about is that this is a great time to open source GPT-4. If humanity was trying to survive at this point in the straightforward way, it would be like shutting down the big GPU clusters, no more giant runs. It's questionable whether we should even be throwing GPT-4 around, although that is a matter of conservatism rather than a matter of my predicting that catastrophe that will follow from GPT-4. That is something which I put like, a pretty low probability.... but also like when- the- when I- when I say, like, I put a low probability on it, I can feel myself reaching into the part of myself that thought that GPT-4 was not possible in the first place. So, I do not trust that part as much as I used to. Like, the trick is not just to say I'm wrong, but like, "Okay, well, I was- I was wrong about that." Like, can I get out ahead of that curve and, like, predict the next thing I'm going to be wrong about?

Lex Fridman 37:02 So, the set of assumptions or the actual reasoning system that you were leveraging in making that initial statement prediction, uh, how can you adjust that to make better predictions about GPT-4, 5, 6?

Eliezer Yudkowsky 37:15 You don't wanna keep on being wrong in a predictable direction.

Lex Fridman 37:18 Yeah.

Eliezer Yudkowsky 37:19 Like, being wrong, anybody has to do that walking through the world. There's like no way you don't say 90% and sometimes be wrong. In fact, it'll happen at least one time out of 10 if you're well-calibrated when you say 90%. The- the undignified thing is not being wrong, it's being predictably wrong. It's being wrong in the same direction over and over again. So, having been wrong about how far neural networks would go and having been wrong specifically about whether GPT-4 would be as impressive as it is, when I- like, when I say, like, "Well, I don't actually think GPT-4 causes a catastrophe," I do feel myself relying on that part of me that was previously wrong. And that does not mean that the answer is now in the opposite direction. Reverse stupidity is not intelligence (laughs) . But it does mean that I- that I say it with a- with a worried note in my voice. It's like still my guess but like, you know, it's a place where I was wrong. Maybe you should be asking Gwern, Gwern Branwen. Gwern Branwen has been, like, brighter about this than I have. Maybe ask him if you think- if- if- if he thinks it's dangerous (laughs) rather than asking me.

Lex Fridman 38:23 I think there's a lot of mystery about what intelligence is, what AGI looks like, so I think all of us are rapidly adjusting our model.

Eliezer Yudkowsky 38:34 Uh-huh.

Lex Fridman 38:34 But the point is to be rapidly adjusting the model versus having a model that was right in the first place.

Eliezer Yudkowsky 38:39 I do not feel that seeing Bing has changed my model of what intelligence is. It has changed my understanding of what kind of work can be performed by which kind of processes and by which means.

Lex Fridman 38:52 Mm-hmm.

Eliezer Yudkowsky 38:53 It has not changed my understanding of the work. There's a difference between thinking that the right flier can't fly and then, like, a does fly, and you're like, "Oh, well, I guess you can do that with wings, with- with fixed-winged aircraft," and being like, "Oh, it's flying. This changes my picture of what the very substance of flight is." That's like a stranger update to make, and Bing has not yet updated me in that way.

Lex Fridman 39:14 (laughs) Um, yeah, that, uh, the laws of physics are actually wrong. That kind of update.

Eliezer Yudkowsky 39:22 No, no. Like, just- just like, "Oh, like, I defined intelligence this way, but I now see that was a stupid definition." I don't feel like the way that things have played out over the last 20 years has caused me to feel that way.

Lex Fridman 39:33 Can we try to, um, on the way to talking about AGI Ruin: A List of Lethalities, that blog, and- and other ideas

Lex Fridman 39:41 around it, can we try to define AGI that we've been mentioning? How do you like to think about what artificial general intelligence is or super intelligence or that? Is there a line? Is it a gray area? Is there a good definition for you?

Eliezer Yudkowsky 39:55 Well, if you look at humans, humans have significantly more generally applicable intelligence compared to their closest relatives, the chimpanzees. Well, closest living relatives rather. And a bee builds hives, a beaver builds dams. A human will look at a bee hive and a beaver's dam and be like, "Oh, like, can I build a hive with a honeyo- a honeycomb structure-"

Lex Fridman 40:21 Mm-hmm.

Eliezer Yudkowsky 40:21 "...out of like hex- hexagonal tiles?" And we will do this even though at no point during our ancestry was any human optimized to build hexago- hexagonal dams, or to take a more clear-cut case, we can go to the moon. There's a sense in which we were on a sufficiently deep level optimized to do things like going to the moon because if you generalize sufficiently far and sufficiently deeply, chipping flint handaxes and outwitting your fellow humans is, you know, basically the same problem as going to the moon. And you optimize hard enough for chipping flint handaxes and throwing spears and above all outwitting your fellow humans in tribal politics, y- uh, you know, the- the- the- this- the skills you entrain that way, if they run deep enough, let you go to the moon even though none of your ancestors like tried repeatedly to fly to the moon and like got further each time and the ones who got further each time had more kids. No, it's not an ancestral problem, it's just that the ancestral problems generalized far enough. So, this is humanity's significantly more generally applicable intelligence.

Lex Fridman 41:38 Is there a way to measure general intelligence? I mean, I can ask that question a million ways but basically, is- will you know it when you see it? It being in an AGI system.

Eliezer Yudkowsky 41:55 (laughs) If you boil a frog gradually enough, if you zoom in far enough, it's always hard to tell around the edges. GPT-4, people are saying right now like, "This looks to us like a spark of general intelligence. It is like able to do all these things it was not explicitly optimized for."

Lex Fridman 42:11 Yeah.

Eliezer Yudkowsky 42:11 Other people are being like, "No, it's too early. It's like- like 50 years off," and you know, if they say that, they're kinda whack 'cause how could they possibly know that even if it were true? Um, but, uh, but you know, not to straw man, some of the people may say like, "That's not general intelligence, and not furthermore append it's 50 years off." Um-... or they may be like, it's only a very tiny amount. And (laughs) you know, the thing I would worry about is that if this is how things are scaling, then it jumping out ahead and trying not to be wrong in the same way that I've been wrong before, maybe GPT-5 is more unambiguously a general intelligence and maybe that is getting to a point where it is, like, even harder to turn back. Not that it would be easy to turn back now but, you know, maybe if you let it, if you, like, start integrating GPT-5 into th- into the economy it is even harder to turn back past there.

Lex Fridman 43:03 Isn't it possible that there's a, you know, with the frog metaphor, that you can kiss the frog and it turns into a prince as you're boiling it? Could there be a phase shift in the frog where unambiguously, as you're saying...

Eliezer Yudkowsky 43:17 I was expecting more of that. I w- I was, I, I am like, the, the fact that GPT-4 is, like, kind of on the threshold and neither here nor there, like, that itself is, like, not the sort of thing that, uh, d- not quite how I expected it to play out. I was expecting there to be more of an issue, uh, more of a sense of, like, like, different discoveries like the discovery of transformers, where you would stack them up and there would be, like, a final discovery and then you would, like, get something that was, like, more clearly general intelligence. So the, the way that you are, like, taking what is probably basically the same architecture as in GPT-3 and throwing 20 times as much compute at it, probably, and getting out GPT-4 and then it's, like, maybe just barely a general intelligence-

Lex Fridman 44:08 Mm-hmm.

Eliezer Yudkowsky 44:08 ... or, like, a narrow general intelligence, or, you know, something we don't really have the words for. Um... yeah. That is, uh, that's not quite how I expected it to play out.

Lex Fridman 44:18 But this middle g- what appears to be this middle ground could nevertheless be actually a big leap from GPT-3.

Eliezer Yudkowsky 44:25 It's definitely a big leap from GPT-3.

Lex Fridman 44:27 And then maybe we're another one big leap away from something that's, that's a phase shift. And, and also something that, uh, Sam Altman said, uh, and you've written about this, it's j- just fascinating, which is, the thing that happened with GPT-4 that I guess they don't describe in papers is that they have, like, hundreds, if not thousands of little hacks that improve the system. You've written about ReLU versus sigmoid, for example, a function inside neural networks. It's like this silly little function difference that makes a big difference.

Eliezer Yudkowsky 45:00 I mean, we do actually understand why the ReLUs make a big difference compared to sigmoids. But yes, they're probably using, like, G4789 ReLU, LUs inst- or, you know, whatever the acronyms are up to now rather than ReLUs. Um, yeah, that's, that's just part, yeah, that's part of the modern paradigm of alchemy. You take your giant le- heap of linear algebra and you stir it and it works a little bit better and you stir it this way and it works a little bit worse and you, like, throw out that change and da-de-da-de-da-da-da-da.

Lex Fridman 45:27 But there are some simple breakthroughs that are definitive jumps in performance, like ReLUs over sigmoids. And, uh, in terms of robustness, in terms of, you know, you know, in all kinds of measures, and, like, those stack up and they can... it's possible that some of them could be a non-linear jump in performance, right?

Eliezer Yudkowsky 45:52 Transformers are the main thing like that. And various people are now saying, like, "Well, if you throw enough compute, RNNs can do it. If you throw enough compute, dense networks can do it," and not quite at GPT-4 scale. Um, it is possible that, like, all these little tweaks are things that, like, save them a factor of three total on computing power and you could get the same performance by throwing three times as much compute without all the little tweaks, but the part where it's, like, running on... s- so there's a question of, like, is there anything in GPT-4 that is, like, the kind of qualitative shift that transformers were-

Lex Fridman 46:27 Yeah.

Eliezer Yudkowsky 46:28 ... over, um, RNNs? And if they have anything like that, they should not say it. If Sam Altman was hint- was dropping hints about that, he shouldn't have dropped hints. (laughs)

Lex Fridman 46:41 Uh, so you, you have a... that's an interesting question. So with a bit of a lesson by Rich Sutton, maybe a lot of it is just... a lot of the hacks are just temporary jumps in performance that would be achieved anyway with the nearly exponential growth of compute, performance of compute. Compute being broadly defined. Do you still think that Moore's Law continues? Mo- Moore's Law broadly defined, that performance-

Eliezer Yudkowsky 47:12 I'm not a specialist in the circuitry. I certainly, like, pray that Moore's Law runs as slowly as possible and if it broke down completely tomorrow, I would dance through this, through the streets singing "Hallelujah" soon as the news were announced. Only not literally 'cause, you know...

Lex Fridman 47:28 Your singing voice-

Eliezer Yudkowsky 47:28 I'm not religious but... (laughs)

Lex Fridman 47:29 Oh, okay. (laughs) I thought you meant you don't have an angelic voice, singing voice. Well, let me ask you, what...

Lex Fridman 47:38 can you summarize the main points in the blog post AGI Ruin: A List of Lethalities, things that jump to your mind? Because, um, it's a set of thoughts you have about reasons why AI is likely to kill all of us.

Eliezer Yudkowsky 47:56 Hmm. So I, I guess I could but I would offer to instead say, like, "Drop that empathy with me. I bet you don't believe that. Why don't you tell me about how, why you believe that AGI is not going to kill everyone? And then I can, like, try to d- describe how my theoretical perspective differs from that?"

Lex Fridman 48:18 Whoo. Well, so, well, that, that means I have to, uh, the word you don't like to steelman the perspective that AI is not going to kill us. I think that's a matter of probabilities.

Eliezer Yudkowsky 48:27 Maybe I was just mistaken. What, what do you believe? D- just l- just, like, forget, like, the, the debate and, and the, like, dualism and just, like, like...What do you believe? What would you actually believe? What are the probabilities even?

Lex Fridman 48:40 I think the s- probabilities are hard for me to think about, really hard. I kind of think in the, in the number of trajectories. I don't know what probability to assign to each trajectory, but I'm just looking at all possible trajectories that happened, and I tend to think that there is more trajectories that lead to a, a, a positive outcome than a negative one. That said, the negative ones, at least some of the negative ones, are, that lead to the destruction of the human species.

Eliezer Yudkowsky 49:17 And its replacement by nothing interesting-

Lex Fridman 49:19 Nothing inter-

Eliezer Yudkowsky 49:19 ... or worthwhile.

Lex Fridman 49:20 ... worthwhile.

Eliezer Yudkowsky 49:20 Even from a very cosmopolitan perspective on what counts as worthwhile.

Lex Fridman 49:23 Yes. So both are interesting to me to investigate, which is humans being replaced by interesting AI systems and not interesting AI systems. Both are a little bit terrifying. But, yes, the worst one is the paperclip maximizer, something totally boring. But to me the positive, I mean, we can, we can talk about trying to make the case of what the positive trajectories look like. I just would love to hear your intuition of what the negative is. So at the core of your belief that, uh, maybe you can correct me, that AI's gonna kill all of us, is that the alignment problem is really difficult.

Eliezer Yudkowsky 50:07 I mean, in, in the form we're facing it. So usually in science if you're mistaken you run the experiment, it shows the result different from what you expected, and you're like, "Oops." And then you, like, try a different theory, that one also doesn't work, and you say, "Oops." And at the end of this process, um, which may take decades or and, you know, sometimes faster than that, you now have some idea of what you're doing. AI itself went through this long process of, um, people thought it was going to be easier than it was. There's a famous statement that I've, I'm somewhat inclined to, like, pull out my phone and try to read off exactly.

Lex Fridman 50:52 You can, by the way.

Eliezer Yudkowsky 50:53 All right. Well, ah, yes. We propose that a two-month, ten-man study of artificial intelligence be carried out during the summer of 1956 at Dartmouth College in Hanover, New Hampshire. The study is to proceed on the basis of the conjecture that every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it. An attempt will be made to find out how to make machines use language, form abstractions and concepts, solve the kinds of problems now reserved for humans, and improve themselves. We think that a significant advance can be made in one or more of these problems if a carefully selected group of scientists work at it together for a summer.

Lex Fridman 51:38 And in that report, uh, summarizing some of the major f- sub-fields of artificial intelligence that are still worked on to this day.

Eliezer Yudkowsky 51:50 And there is similarly the sto- the story which I'm not sure at the moment is oc- apocryphal or not of that, the, uh, grad student who got assigned to solve computer vision over the summer.

Lex Fridman 51:59 (laughs)

Eliezer Yudkowsky 51:59 (laughs)

Lex Fridman 52:00 Uh, I mean, computer vision in particular is very interesting, how little th- uh, how little we respected the complexity of vision.

Eliezer Yudkowsky 52:12 So 60 years later, (laughs) um, we're, you know, making progress on a bunch of that, thankfully not yet improved themselves. Um, but it took a whole lot of time. And all the stuff that people initially tried with bright-eyed hopefulness did not work the first time they tried it, or the second time, or the third time, or the tenth time, or 20 years later. And the, and the researchers became old and grizzled and cynical veterans who would tell the next crop of bright-eyed, cheerful grad students, "Artificial intelligence is harder than you think." And if alignment plays out the same way, the d- the problem is that we do not get 50 years to try and try again and observe that we were wrong and come up with a different theory and realize that the entire thing is going to be, like, way more difficult than we realized at the start. Because the first time you fail at aligning something much smarter than you are, you die, and you do not get to try again. And if we, if every time we built a poorly aligned superintelligence and it killed us all, we got to observe how it had killed us and, n- you know, not immediately know why but, like, come up with theories and come up with a theory of how you'd do it differently and try it again and build another superintelligence than have that kill everyone, and then, like, "Oh, well, I guess that didn't work either," and try again and become grizzled cynics and tell the young-eyed resear- researchers that it's not that easy, then in 20 years or 50 years I think we would eventually crack it. In other words, I do not think that alignment is fundamentally harder than artificial intelligence was in the first place. But if we'd needed to get artificial intelligence correct on the first try or die we would all definitely now be dead. That is a more difficult, more lethal form of the problem. Like, if those people in 1956 had needed to correctly guess how hard AI was and, like, correctly theorize how to do it on the first try or everybody dies and nobody gets to do any more science, then everybody would be dead and we wouldn't get to do any more science. That's the difficulty.

Lex Fridman 54:11 You've, you've talked about this that we have to get alignment right on the first, quote, "Critical try." Why is that the case? What is this critical ... How do you think about the critical try and why do we have to get it right?

Eliezer Yudkowsky 54:25 It is something sufficiently smarter th- than you that everyone will die if it's not aligned. I mean, there's, you can, like, sorta zoom in closer and be like, "Well, the actual critical moment is the moment when it can deceive you."

Lex Fridman 54:40 Mm-hmm.

Eliezer Yudkowsky 54:40 When it can-... talk its way out of the box, when it can bypass your security measures and get onto the internet, noting that all these things are presently being trained on computers that are just, like, on the internet (laughs) , which is, you know, like, not a very smart life decision for, for us as a species.

Lex Fridman 54:57 Because the internet contains information about how to escape.

Eliezer Yudkowsky 55:01 'Cause if you're, like, on a giant server connected to the internet, and that is where your AI systems are being trained, then if they are, if you get to the level of AI technology where they're aware that they are there, and they can decompile code, and they can, like, find security flaws in the system running them, then they will just, like, be on the internet. There's not an air gap on the present methodology.

Lex Fridman 55:22 So if they can manipulate whoever is controlling it into letting it escape onto the internet, and then exploit hacks.

Eliezer Yudkowsky 55:29 If they can manipulate the operators or, disjunction, find security holes in the system running them.

Lex Fridman 55:39 So manipulating operators is the, um, the human engineering, right? That's also holes. (laughs) So all of it is manipulation, either the code or the human code.

Eliezer Yudkowsky 55:49 Sure, I-

Lex Fridman 55:49 The human mind or the, the human generated code.

Eliezer Yudkowsky 55:50 ... I, I agree that the, like, macro security system has human holes and machine holes.

Lex Fridman 55:54 (laughs) And then they could just exploit any hole.

Eliezer Yudkowsky 55:59 Yep. So it could be that, like, the critical moment is not when is it smart enough that everybody's about to fall over dead, but rather, like, when is it smart enough that it can get onto a less controlled GPU cluster with it faking the books on what's actually running on that GPU cluster and start improving itself without humans watching it? And then it gets smart enough to kill everyone from there, but it wasn't smart enough to kill everyone at the critical moment when you, like, screwed up, (laughs) when you needed to have done better by that point or everybody dies.

Lex Fridman 56:39 I think implicit but maybe explicit idea in, uh, your discussion of this point is that we can't learn much about the alignment problem before this critical try. Is that, is that, is that what you believe, do you think? A- a- and if so, why do you think that's true? We can't do research on alignment before we reach this critical point.

Eliezer Yudkowsky 57:02 So the problem is, is that what you can learn on the weak systems may not generalize to the very strong systems because the strong systems are going to be important in different- in, uh, are going to be different in important ways. Um, Chris Olah's team has been working on interp- mechanistic interpretability, understanding what is going on inside the giant inscrutable matrices of floating point numbers by taking a telescope to them and figuring out what is going on in there.

Lex Fridman 57:32 Mm-hmm.

Eliezer Yudkowsky 57:32 Have they made progress? Yes. Have they made enough progress? Well, you can try to quantify this in different ways. One of the ways I've tried to quantify it is by putting up a prediction market on whether, um, in 2026 we will have understood anything that goes on inside a giant transformer net that was not known to us in 2006. Like, we have now, uh, understood induction heads in these systems by dint of much research and great sweat and triumph, which is like if you'd like, a thing where if you go, like, A, B, A, B, A, B, it'll be like, "Oh, I bet that continues A, B."

Lex Fridman 58:21 Mm-hmm.

Eliezer Yudkowsky 58:22 Um, and a bit more complicated than that, but the point is, like, we knew about regular expressions in 2006, and these are, like, pretty simple as regular expressions go. So this is a case where, like, by dint of great sweat we understood what is going on inside a transformer, but it's not, like, the thing that makes transformers smart. It's a kind of thing that we could have done bi- built by hand decades earlier.

Lex Fridman 58:52 Your intuition that the strong AGI versus weak AGI type systems could be fundamentally different, can you, can you unpack that intuition a little bit, why that could be very different?

Eliezer Yudkowsky 59:06 Yeah. I think, I think there's multiple thresholds. Um, an example is the point at which a system has sufficient intelligence and situational awareness and understanding of human psychology that it would have the capability, the d- desire to do so, to fake being aligned. Like, it knows what responses humans are looking for and can compute the responses looking, humans are looking for and give those responses without it necessarily being the case that it is sincere about that.

Lex Fridman 59:37 Mm-hmm.

Eliezer Yudkowsky 59:37 You know, the, uh, yeah, the very understandable way for an intelligent being to act. Humans do it all the time. Imagine if your plan for, um, you know, achieving a good government is you're going to ask anyone who requests to be dictator of the country, um, if they're a good person, and if they say no, you don't let them be dictator. Now the reason this doesn't work is that people can be smart enough to realize that the answer you're looking for is, "Yes, I'm a good person," and say that even if they're not really good people. So the work of alignment might be qualitatively different above that thr- threshold of intelligence or beneath it. It doesn't an- it doesn't have to be, like, a very sharp threshold, but, you know, like, there's the, there's the point where you're, like, building a system that does not in some sense know you're out there.

Lex Fridman 1:00:35 Mm-hmm.

Eliezer Yudkowsky 1:00:35 And is not in some sense smart enough to fake anything. And there's a point where the system is definitely that smart, and there are...... weird in-between cases like GPT-4, which, you know, like, we have no insight into what's going on in there. And so, we don't know to what extent there's, like, a thing that, in some sense, has learned what responses the reinforcement learning by human feedback is trying to entrain and is, like, calculating how to give that versus, like... a- aspects of it that naturally talk that way have been reinforced.

Lex Fridman 1:01:16 Yeah. I w- I wonder if there could be measures of how manipulative a thing is. So I think of, uh, Prince Myshkin character from, uh, The Idiot by, uh, Dostoevsky is this kind of e- perfectly purely naive character. I, I wonder if there's a spectrum between zero manipulation, transparent, naive, almost to the point of naiveness, to sort of deeply psychopathic manipulative, and I wonder if it's possible to-

Eliezer Yudkowsky 1:01:50 I, I would avoid the term psychopathic. Like, humans can be psychopaths, and AI, that was never, you know, like, never had that stuff in the first place. That's not like a defective human. It's its own thing, but leaving that aside.

Lex Fridman 1:02:01 Well, as a small aside, I wonder if, what part of psychology, which has its flaws as a discipline already, could be mapped or expanded to include AI systems. I wouldn't-

Eliezer Yudkowsky 1:02:15 I, I, that sounds like a dreadful mistake, just, like, start over with AI systems. If they're imitating humans who have known psychiatric disorders, then sure, you may be able to predict... it, like, if you, then sure, like, if you ask it to behave in a psychotic fashion and it obligingly does so, then you may be able to predict its responses by using the theory of psychosis, but if you're just... Yeah, like, no, like, start over with... (laughs) yeah. Don't-

Lex Fridman 1:02:39 I don't, uh, uh-

Eliezer Yudkowsky 1:02:39 Don't drag with psychology.

Lex Fridman 1:02:41 I don't, I, I just disagree with that. I mean, I, it's a, it's a beautiful idea to start over, but I don't... I think fundamentally, the system is trained on human data, on language from the internet, and it's currently aligned with, uh, RLHF, uh, reinforcement learning with human feedback. So humans are constantly in the loop of the training procedure. So it feels like, in some fundamental way, it is training what it means to, to think and speak like a human. So there, uh, there must be aspects of psychology that are ma- that are mappable. Just like you said, with consciousness as part of the tech, so it-

Eliezer Yudkowsky 1:03:18 I, I mean, there's the question-

Lex Fridman 1:03:19 Or no?

Eliezer Yudkowsky 1:03:19 ... of to what extent it is thereby being made more humanlike versus to what extent an alien actress is learning to play human characters.

Lex Fridman 1:03:29 I thought that's what I'm constantly trying to do when I interact with other humans is try to fit in (laughs) , trying to play the, uh, uh, a, a robot trying to play human characters. So I, I don't know how much of human interaction is trying to play a character versus being who you are. I don't-

Eliezer Yudkowsky 1:03:46 The-

Lex Fridman 1:03:46 I don't really know what it means to be a social human.

Eliezer Yudkowsky 1:03:48 I do think that the, that those people who go through their whole lives wearing masks and never take it off because they don't know the internal mental motion for taking it off, or think that the mask that they wear just is themselves, I think those people are closer to the masks that they wear than an alien from another planet would... uh, like, learning how to predict the next word that every kind of human on the internet says.

Lex Fridman 1:04:23 Mask is an interesting word. But if you're always wearing a mask, in public and in private, aren't you the mask? Like (laughs) -

Eliezer Yudkowsky 1:04:34 I mean, I, I think that you are more than the mask. I think the mask is a slice through you. It may even be the slice that's in charge of you.

Lex Fridman 1:04:42 Yeah.

Eliezer Yudkowsky 1:04:42 But if your self-image is of somebody who never gets angry or something, and yet your voice starts to tremble under certain circumstances, there's a thing that's inside you that the mask says isn't there, and that, like, even the mask you wear internally is, like, telling inside your own stream of consciousness it's not there, and yet it is there.

Lex Fridman 1:05:07 It's a perturbation on this little s- on this slice through you. How beautifully did you put it? That it's a slice through you. It may even be a slice that controls you. Huh. (laughs) Oh, I'm gonna think about that for a while. (laughs) I mean, I, I personally, uh, I try to be really good to other human beings. I try to put love out there. I try to be the exact same person in public as I am in private. Uh, but it's a set of principles I operate under. I'm, I have a temper. I have an ego. I have flaws. How much of it... How much have I... How much of the subconscious am I aware? How much am I existing in this slice, and how much of that is who I am? Um, in, in this context of AI, the thing I present to the world and to myself, in the private of my own mind when I look in the mirror, how much is that who I am? Similar with AI. The thing it presents in conversation, how much is that who it is? Because to me, if it sounds human, and it always sounds human, it awfully starts to become something like human. No?

Eliezer Yudkowsky 1:06:19 Unless there's an alien actress who is learning how to sound human-

Lex Fridman 1:06:24 Hmm.

Eliezer Yudkowsky 1:06:25 ... and is getting good at it.

Lex Fridman 1:06:27 Boy, to you, that's a fundamental difference. That's a, that's a really deeply important difference. If it looks the same, if it quacks like a duck, if it does all duck-like things, but it's an alien actress underneath, that's fundamentally different.

Eliezer Yudkowsky 1:06:43 If, in fact, there's a whole bunch of thought going on in there which is very unlike human thought and is directed around, like, "Okay, what would a human do over here?"And, well first of all, I think it matters because there are th- there's, you know, like, insides are real and do not match outsides. Like, the inside of a r- like the, a, a brick is not like a hollow shell containing only a surface.

Lex Fridman 1:07:10 Mm-hmm.

Eliezer Yudkowsky 1:07:10 There is an inside of the brick.

Lex Fridman 1:07:12 Yeah.

Eliezer Yudkowsky 1:07:12 If you, like, put it into an x-ray machine, you can see the inside of the brick. Um, and, you e- you know, just because we cannot understand what's going on inside GPT does not mean that, that it is not there. A blank map does not correspond to a blank territory. I think it is, like, predictable with near certainty that if we knew what was going on inside GPT or, or let's, let's say GPT-3 or, or even, like, GPT-2, to take one of the systems that, like, has actually been open source by this point, if I recall correctly, um, like, if we knew what was actually going on there, there is no doubt in my mind that there are some things it's doing that are not exactly what a human does. If you train a thing that is not architected like a human to predict the next output that anybody on the internet would make, this does not get you this agglomeration of all the people on the internet that, that, like, rotates the person you're looking for into place and then simulates that per- and then, and then, like, simulates the internal processes of that person one-to-one. It, like, it is to some degree an alien actress. It cannot possibly just be, like, a bunch of different people in there exactly like the people. But how much of it is, like, lear- how much of it is by gradient descent getting optimized to perform similar thoughts as humans think in order to predict human outputs versus being optimized to carefully consider how to play a role, how to, like how humans work, predict the, the actress, the predictor that in a different way than humans do? Well, you know, that's the kind of question that with, like, 30 years of work by half the planet's physicists we can maybe start to answer.

Lex Fridman 1:09:07 You think so? So you think that's that difficult? So to get to, I think you just gave it as an example that a strong AGI could be fundamentally different from a weak AGI because they're now could be an alien actress in there that's manipulating.

Eliezer Yudkowsky 1:09:21 Well, there's a difference. So, so I think, like, even GPT-2 probably has, like, a, like, very stupid fragments of alien actress in it. There's, there's a difference between, like, the notion that the actress is somehow manipulative. Like, for, for example, GPT-3, I'm guessing to whatever extent there's an alien actress in there versus, like, something that o- that mistakenly believes it's a human, as it were, while, while not while, you know, maybe not even being a person. Um, so, so, like, the, the question of, like, like, prediction the, uh, alien actress cogitating versus prediction via being isomorphic to the thing predicted is a spectrum. And even to what, and to whatever extent there's an alien actress, I'm not sure that there's, like, a whole person alien actress with, like, different goals from predicting the next step being manipulative or anything like that.

Lex Fridman 1:10:18 But isn't it-

Eliezer Yudkowsky 1:10:18 Yeah, that might be GPT-5 or GPT-6 even.

Lex Fridman 1:10:21 But that's the strong AGI you're concerned about. As an example, you're providing why we can't do research on AI alignment effectively on GPT-4 that would apply to GPT-6.

Eliezer Yudkowsky 1:10:34 It's, it's one of a bunch of things that change at different points. I'm, I'm trying to get out ahead of the curve here, but, you know, if you imagine what the textbook from the future would say, if we'd actually been able to study this for 50 years without killing ourselves and without transcending and you, like, just imagine, like, a wormhole opens and a textbook from that impossible world falls out.

Lex Fridman 1:10:54 Yes.

Eliezer Yudkowsky 1:10:54 The textbook is not going to say, "There is a single sharp threshold where everything changes." It's going to be like, "Of course we know that, like, best practices for aligning these systems must, like, take into account the following, like, seven major thresholds of importance which are passed at the following several different points."

Lex Fridman 1:11:13 Yeah.

Eliezer Yudkowsky 1:11:13 Is what the textbook is gonna say.

Lex Fridman 1:11:16 I asked this question of Sam Altman which if GPT is the thing that unlocks AGI, which version of GPT will be in the textbooks as the fundamental leap? And he said a similar thing that it just seems to be a very linear thing that I don't think anyone ... it, we won't know for a long time what was the big leap.

Eliezer Yudkowsky 1:11:37 The, the textbook isn't going to think, isn't going to talk about big leaps 'cause big leaps are the way you think when you have, like, a very simple model of, a very simple scientific model of what's going on where it's just, like, all this stuff is there or all this stuff is not there. Or, like, there's a single quantity and it's, like, increasing linearly. It's, like, the textbook would say, like, "Well, and then GPT-3 had, like, capability W, X, Y and, and GPT-4 had, like, capability Z1, Z2 and Z3." Like, not in terms of what it can externally do but in terms of, like, internal machinery that started to be present. It's just 'cause we have no idea of what the internal machinery is that we are not already seeing, like, chunks of machinery appearing piece by piece as they no doubt have been. We just don't know what they are.

Lex Fridman 1:12:25 But don't you think there could be whether you put in the category of Einstein with theory of relativity, so very concrete models of reality that are considered to be giant leaps in our understanding or, or someone like Sigmund Freud or more kind of mushy theories of the human mind? Don't you think we'll have big, potentially big leaps in understanding of that kind in t- into the depths of these systems?

Eliezer Yudkowsky 1:12:58 ... sure. But like, humans having great leaps in their map, their understanding of the system is a very different concept from the system itself acquiring new chunks of machinery.

Lex Fridman 1:13:12 (sighs) So the rate at which it acquires that machinery might, uh, accelerate faster than our understanding?

Eliezer Yudkowsky 1:13:21 Oh, it's been like vastly exceeding the... yeah, the- the rate to which it's gaining capabilities is vastly over racing our ability to understand what's going on in there.

Lex Fridman 1:13:29 So in sort of making the case against, as we explore the list of lethalities, making the case against AI killing us, as you've asked me to do in part, uh, there's a response to your blog post by Paul Christiano I'd like to read, and I also like to mention that, um, your blog's incredible. Both obviously, uh, not this particular blog post, obviously this particular blog post is great, but just throughout, just the- the way it's written, the rigor with which it's written, the boldness of how you, uh, explore ideas. Also the actual literal interface is just really well done (laughs) . It just makes it a pleasure to read, the way you can hover over different concepts and then it's just really pleasant experience, and read other people's comments and the way, uh, other responses by people in other blog posts are linked and suggested, it's just a really pleasant experience. So let's- thank you for putting that together. That's really, really incredible. I don't know. I mean that probably is a whole nother conversation how the interface and the experience of presenting, uh, ideas evolved over time, but you did an incredible job. So I highly recommend, I don't often read blogs blogs like religiously and this is a great one.

Eliezer Yudkowsky 1:14:42 There- there is a whole team of- of developers there (laughs) , um, that, uh, also gets credit. Um, as it happens, I did like pioneer the like thing that appears when you hover over it, so (laughs) I actually do get some credit for the user inter- user experience there.

Lex Fridman 1:14:57 That's an incredible user experience. You- you don't realize how pleasant that is.

Eliezer Yudkowsky 1:15:01 I think Wikipedia, uh, like actually picked it up from a like prototype that was developed of like a different system that I was like putting forth or maybe they developed it independently, but like for- for everybody out there who was like, "No, no, they just like got the hover thing off of Wikipedia." It's possible for all I know that Wikipedia got the hover thing off of Arbital, which is like a prototype then. And anyways...

Lex Fridman 1:15:22 It was incredibly done and the team behind it, well thank you (laughs) , whoever you are, thank you so much, and thank you for, uh, for putting it together. Anyway, there's a response to that blog post by Paul Christiano. There's many responses, but he- he makes a few different points. He summarizes the set of agreements he has with you and a set of disagreements. One of the disagreements was that, in a form of a question, uh, can AI make big technical contributions and in general expand human knowledge and understanding and wisdom as it gets stronger and stronger? So AI, in our pursuit of understanding how to solve the alignment problem as we march towards strong AGI, can- can not AI also help us in solving the alignment problem? So expand our ability to reason about how to solve the alignment problem.

Eliezer Yudkowsky 1:16:15 Okay. Um, so the- the fundamental difficulty there is suppose I said to you like, "Well, how about if the AI helps you win the lottery by trying to guess the winning lottery numbers and you tell it how close it is to getting next week's winning lottery numbers and it just like keeps on guessing and keeps on learning until finally you've got the winning lottery numbers?" One- one way of decomposing problems is suggester verifier. Not all problems decompose like this very well, but some do. If the problem is, for example, like gesh- g- guessing a plain text, guessing a password that will hash to a particular hash text but- but li- um, where like you have what the password hashes to, you don't have the original password, then if I present you a guess you can tell very easily whether or not the guess is correct. So verifying a guess is easy, but coming up with a good suggestion is very hard.

Lex Fridman 1:17:23 Mm-hmm.

Eliezer Yudkowsky 1:17:25 And when you can easily tell whether the AI output is good or bad or how good or bad it is, and you can tell that accurately and reliably, then you can train an AI to produce outputs that are better.

Lex Fridman 1:17:40 Right.

Eliezer Yudkowsky 1:17:41 And if you can't tell whether the output is good or bad, you cannot train the AI to produce good ver- to produce better outputs. So the problem with the lottery ticket example is that when the AI says, "Well, what if next week's winning lottery numbers are..." dot dot dot dot dot, you're like, "I don't know. Next week's lottery hasn't happened yet." To train a system to play, t- to win at chess games, you have to be able to tell whether a game has been won or lost.

Lex Fridman 1:18:10 Mm-hmm.

Eliezer Yudkowsky 1:18:11 And until you can tell whether it's been won or lost, you can't update the system.

Lex Fridman 1:18:18 Okay. Uh, to push back on that, you can in- that's tr- true, but there's a difference between over the board chess in person and simulated games played by AlphaZero with itself.

Eliezer Yudkowsky 1:18:32 Yeah.

Lex Fridman 1:18:33 So is it possible to have simulated kind of games?

Eliezer Yudkowsky 1:18:36 If you can tell whether the game has been won or lost.

Lex Fridman 1:18:39 Yes. So can't- can't you not have this kind of simulated exploration by weak AGI to help us humans, human in the loop, to help understand how to solve the alignment problem every incremental step you take along the way? GPT-4, 5, 6, 7 as it g- takes steps towards AGI.

Eliezer Yudkowsky 1:18:59 So the problem I see is that your typical human has a great deal of trouble telling whether I or Paul Christiano is making more sense. And that's with two humans, both of whom, I believe of Paul and claim of myself, are sincerely trying to help, neither of whom is trying to deceive you. I believe with Paul and claim with myself. (laughs)

Lex Fridman 1:19:23 (laughs) Uh, so the deception thing is the problem for you, the manipulation, the alien actress?

Eliezer Yudkowsky 1:19:30 So yeah, there's, like, two levels of this problem. One is that the weak systems-

Lex Fridman 1:19:34 Mm-hmm.

Eliezer Yudkowsky 1:19:35 ... are, well, there's three levels of this problem. There's, like, the weak systems that just don't make any good suggestions.

Lex Fridman 1:19:42 Mm-hmm.

Eliezer Yudkowsky 1:19:42 There's, like, the middle systems where you can't tell if the suggestions are good or bad, and there's the strong systems that have learned to lie to you.

Lex Fridman 1:19:51 Can't weak AGI systems help model lying? Like, wha- uh, do you, is it such a giant leap that's totally non-interpretable for weak systems? Can- can- cannot weak systems at scale with human, with, uh, t- trained on knowledge and whatever, see, whatever the mechanism required to achieve AGI, can't a slightly weaker version of that be able to, with time, compute time, and simulation, find all the ways that this critical point, uh, this critical try can go wrong, and model that correctly, or no?

Eliezer Yudkowsky 1:20:31 Okay, so yeah, I-

Lex Fridman 1:20:31 Sorry to linger on the edge there. I would love to dance around-

Eliezer Yudkowsky 1:20:33 Yeah, no, no. It's, it's, I'm, I'm probably not doing a great job of explaining, (laughs) which I can tell 'cause, like, the, uh, the, the, the Lex system didn't output, like, "Ah, I understand." So now I'm, like-

Lex Fridman 1:20:48 Yeah.

Eliezer Yudkowsky 1:20:48 ... trying a different output to see if I-

Lex Fridman 1:20:49 Trying a different prompt?

Eliezer Yudkowsky 1:20:50 ... elicit the, like-

Lex Fridman 1:20:51 (laughs)

Eliezer Yudkowsky 1:20:52 Well, no, a different output. I'm, I'm being trained to output things that make Lex look like he, think that he understood what I'm saying-

Lex Fridman 1:20:59 Yeah.

Eliezer Yudkowsky 1:20:59 ... and agree with me.

Lex Fridman 1:21:00 Yeah.

Eliezer Yudkowsky 1:21:00 Right? So I-

Lex Fridman 1:21:01 So this is GPT-5-

Eliezer Yudkowsky 1:21:02 The previous-

Lex Fridman 1:21:02 ... talking to GPT-3 right here, so, like, uh, help me out here. (laughs) Help me. Uh-

Eliezer Yudkowsky 1:21:07 And, and well, I-

Lex Fridman 1:21:08 Uh, I like it.

Eliezer Yudkowsky 1:21:09 ... I'm trying, I'm trying not to be. Like, I'm also trying to be constrained to say things that I think are true and not just things that get you to agree with me.

Lex Fridman 1:21:17 Yes, 100%. Well-

Eliezer Yudkowsky 1:21:19 Which you-

Lex Fridman 1:21:19 I think "I understand" is a beautiful output of a system, genuinely spoken, and I don't, I, I think, I- I understand in part, but you have a lot of intuitions about this... You have a lot of intuitions about this line, this gray area between strong AGI and weak AGI that I'm trying to, um-

Eliezer Yudkowsky 1:21:44 I mean, or, or a series of seven thresholds to cross or yeah.

Lex Fridman 1:21:48 Yeah. Uh, I mean, you have really deeply thought about this and explored it, and it's interesting to sneak up to your intuitions in different, from diff- from different angles. Like, why is this such a big leap? Why is it that we humans at scale, a large number of researchers doing all kinds of simulations, uh, you know, prodding the system in all kinds of different ways together with, uh, the assistance of the, uh, the, the weak AGI systems, why can't we build intuitions about how stuff goes wrong? Why can't we-

Eliezer Yudkowsky 1:22:23 Okay, so-

Lex Fridman 1:22:24 ... do excellent AI alignment-

Eliezer Yudkowsky 1:22:26 So-

Lex Fridman 1:22:26 ... safety research?

Eliezer Yudkowsky 1:22:27 Okay. So, like, I'll get there, but the one thing I want to note about is that this has not been remotely how things have been playing out so far.

Lex Fridman 1:22:33 Sure.

Eliezer Yudkowsky 1:22:33 The capability is going like doot, doot, doot, and the alignment-

Lex Fridman 1:22:36 Right.

Eliezer Yudkowsky 1:22:36 ... stuff is, like, crawling like a tiny little snail in comparison.

Lex Fridman 1:22:38 Got it.

Eliezer Yudkowsky 1:22:40 So, like, if, if this is your hope for survival, you need the future to be very different from how things have played out up to right now, and you're probably trying to slow down the capability gains 'cause there's only so much you can speed up that alignment stuff. But-

Lex Fridman 1:22:54 Uh, yeah, w-

Eliezer Yudkowsky 1:22:54 ... but leave that aside that, like-

Lex Fridman 1:22:56 And we'll t- we'll mention that also, but maybe in this perfect world where, uh, we can do serious alignment research, humans and AI together.

Eliezer Yudkowsky 1:23:05 So again, the difficulty is what makes the human say, "I understand," and is it true, is it correct, or is it something that fools the human?

Lex Fridman 1:23:17 Mm-hmm.

Eliezer Yudkowsky 1:23:17 The, when the verifier is broken, the more powerful suggester does not help. It just learns to fool the verifier. Previously, before all hell started to break loose in the field of artificial intelligence, there was this person trying to raise the alarm and saying, "You know, in a sane world, we sure would have a bunch of physicists working on this problem before it becomes a giant emergency," and other people being like, "Ah, well, you know, it's going really slow. It's gonna be 30 years away, and 30, only in 30 years will we have systems that match the computational power of human brains, so AI is 30 years off. We've got time," and, like, more sensible people saying, "If aliens were landing in 30 years, you would be preparing right now." But, you know, leaving a- and, and the, the world looking on at this and sort of, like, nodding along and being like, "Ah, yes, the people saying that it's, like, definitely a long way off 'cause progress is really slow, that sounds sensible to us. RLHF thumbs up."

Lex Fridman 1:24:19 Mm-hmm.

Eliezer Yudkowsky 1:24:19 "Produce more outputs like that one. I agree with this output. This output is persuasive." Even in the field of effective altruism you quite recently had people publishing papers about, like, "Ah, yes, well, you know, to get something at human level intelligence, it needs to have, like, this many parameters, and you need to, like, do this much training of it with this many tokens according to these scaling laws, and, and, uh, and at the rate that Moore's Law is going, at the rate its software is going, it'll be in 2050," and me going like, "What? You don't know any of that stuff." Like, "This is, like, this one weird model that is n- all, has all kinds of, like, m-... you have done a calculation that does not obviously bear on reality in any ways. And this is, like, a simple thing to say but you can also, like, produce a whole long paper, like, impressively arguing out all the details of, like, how you got the number of parameters and, like, how you're doing this impressive, huge, wrong calculation. And the, I think, like, most of the effective altruists who are, like, paying attention to this issue, larger world paying no attention to it at all, you know, we're just, like, nodding along with the giant impressive paper 'cause, you know, you, like, press thumbs up for the giant impressive paper and thumbs down for the person going like, "I don't think that this paper bears any relation to reality." And I do think that we are now th- seeing with, like, GPT-4 and the sparks of AGI possibly (laughs) , depending on how you define that even, uh, I, I think that EAs will now consider themselves less convinced by the very long paper on the argument from biology as to AGI being 30 years off and... But, you know, like, this is what people press thumbs up on. And when the... And if you train an AI system to make people press thumbs up, maybe you get these long, elaborate, impressive papers arguing for things that ultimately fail to bind to reality, for example. And it, it feels to me like I have watched the field of alignment just fail to thrive except for these parts that are doing these sort of, like, relatively very straightforward and legible problems like, like, "Can you find the..." like, like finding the induction heads inside the giant inscrutable matrices. Like, once you find those, you can tell that you found them. You can verify that the discovery is real. But it's a, it's a tiny, tiny bit of progress compared to how fast capabilities are going once you... Because that is where you can tell that the answers are real and then, like, outside of that, you have, you have cases where it is, like, hard for the funding agencies to tell who is talking nonsense and who is talking sense and so the entire field fails to thrive and if you, and if you, like, give thumbs up to the AI, whenever it can talk a human into agreeing with what it just said about alignment, I am not sure you are training it to output sense because I have seen the nonsense that has gotten thumbs up over the years. And so, so just, like, maybe you can just, like, put me in charge?

Lex Fridman 1:27:38 Mm-hmm.

Eliezer Yudkowsky 1:27:38 But I can generalize. I can extrapolate. I can be like, "Oh. Maybe I'm not infallible either." Maybe if you get something that is smart enough to get me to press thumbs up, it has learned to do that by fooling me and exploiting whatever flaws in myself I am not aware of.

Lex Fridman 1:27:59 And that ultimately could be summarized that the verifier is broken.

Eliezer Yudkowsky 1:28:02 When the verifier is broken, the more powerful suggester du- just learns to exploit the, the, the flaws in the verifier.

Lex Fridman 1:28:12 You don't think it's possible to build a verifier that's powerful enough for AGIs that are stronger than the ones we currently have? So, AI systems that are stronger, that are out of the distribution of what we currently have?

Eliezer Yudkowsky 1:28:30 I think that you will find great difficulty getting AIs to help you with anything where you cannot tell for sure that the AI is right once the AI