George Hotz vs Eliezer Yudkowsky

1. 0:00 – 3:33

   Hotz’s opening challenge: skepticism about “foom” and singularity narratives

   1. DPDwarkesh Patel0:00

      Okay. We are gathered here to witness George Hotz and Eliezer Yudkowsky debate and discuss, live on Twitter and YouTube, AI safety and related topics. You guys already know who George and Eliezer are, so I, I don't feel like introduction is necessary. I'm Dwarkesh. I'll be moderating. I'll mostly stay out of the way, um, except to kick things off by letting George explain his basic position. And we'll take things from there. George, I'll kick it off to you.

   2. GHGeorge Hotz0:31

      Sure. Um, so I took an existentialism class in high school, and you'd read about these people, Sartre, Kierkegaard, Nietzsche, and you wonder, "Who were these people alive today?" And I think I'm sitting across from one of them now. Um, rationality and the sequences, uh, this whole field, the whole Less Wrong cinematic universe, uh, have impacted so many people's lives in, I think, a very positive way, including mine. Um, not only are you a philosopher, you're also a, a great storyteller. Um, there's two books that I've picked up and, you know, it was like crack. I couldn't put them down. Uh, one was Atlas Shrugged and the other one was Harry Potter and the Methods of Rationality. Um, it's a great book. Now, those are fictional stories. Um, you've also told some stories pertaining to the real world. Um, one was a story you told when you were younger about how "I remember the day I found staring into the singularity when I was 15." And it starts talking about Moore's law and how Moore's law is fundamentally a human law that says humans double the power of processors every two years. So once computers are doing it, it's going to be two years, but then next time it'll be one year, and then six months, and then three months, and then 1.5 and so on. And this is a hyperbolic sequence. Um, this is a singularity, and that's why it's called staring into the singularity. Then this document said that we were gonna... you know, the AI was gonna do wonderful things for us, we were gonna go colonize the universe, we were gonna go, you know, go forth and do all things till the end of all ages. Um, then you changed your views, and super intelligence does not imply super morality. The orthogonality thesis, I'm not going to challenge it. It is obviously a true statement. Then you kept the basic premise of the story, the recursively self-improving, foom, criticality AI. But instead of saving us, it was gonna kill us. I don't think either of these stories is right, and I don't think either of these stories is right for the same reason. I don't think AI can foom. I don't think AI can go critical. I don't think intelligence can go critical. I think this is an absolutely extraordinary claim. I'm not saying that recursive self-improvement is impossible. Recursive self-improvement is of course possible, humanity has done it. Every time you have used a tool to make a better tool, you have recursively self-improved. What I don't believe in is the AI that's sitting in a basement somewhere running on a thousand GPUs that is suddenly gonna crack the secret to thinking, recursively self-improve overnight, and then flood the world with diamond nanobots. This is an extraordinary claim and it requires extraordinary evidence, and I hand it over to you to deliver that evidence.

2. 3:33 – 5:20

   Yudkowsky reframes: doom doesn’t require fast takeoff

   1. EYEliezer Yudkowsky3:33

      Heh. Well, first, let me say that I don't think that the scenario of us all perishing to non-super moral super intelligence requires that particularly rapid rate of ascent. It requires a large enough gap open up with humanity that hasn't followed along in time. And why be- be... is this a crux? Be- be- before we, we start arguing about whether like self-improvement of things on the large internet connected server clusters rather than basements that now prevail, um, before we start arguing about that part, let's first check where the disagreement lies. So from my perspective, if you've got a trillion beings that are, you know, sufficiently intelligent and smarter than us and not super moral, I think that's kind of game over for us. It... even if you got there via a slow 10 year process instead of a 10 hour process or a 10 week- day process or whatever, if you are at the end point where there's this like large mass of intelligence that doesn't care about you, I think that we are, we are dead. And I worry that our s- and, and more importantly, I worry that our successors will go on to do nothing very much worthwhile with the galaxies. So presumably you think that if things don't go quickly, then we're safe. I dispute that, and maybe that's the part we need to talk about.

3. 5:20 – 7:29

   Timelines and forecasting difficulty: AlphaFold as an example

   1. GHGeorge Hotz5:20

      Sure. Um, well, let's start with, let's give an approximate timeline. We don't, we don't need an exact timeline, but you seem to think this is gonna happen in your lifetime?

   2. EYEliezer Yudkowsky5:33

      That's my wild guess. It is far easier to predict the end point than all the details of the process that takes us, that take us there. Timing is one of those details. Timing is really, really hard. In 2004, I made a prediction that super intelligence would eventually be able to solve the, a special case of the protein folding problem, which is you get to choose the DNA sequence, but you wanna choose a DNA sequence that folds into a shape with a chemical property.... and so I predicted that super intelligence would en- eventually be able to solve this easy special case of protein folding. Now, in reality, protein folding was cracked f- for the much harder general case of biology, was cracked by AI come about 2020 or so, AlphaFold2. Um, there was no way I could've made the timing. I could not even have been confident that the bio- biological case of protein folding was going to be crackable by something so much shorter of super intelligence. A- of course, people at the time said it wasn't possible, you know, for the AI can't do this, like, how do you know this problem was even solvable, et cetera, et cetera. And, you know, I could try to explain how I knew, but that would be a technical story. I would point the fact that a much easier s- sp- um, pardon me, that a much harder general case of the problem I pointed to was solved by a non-super intelligence not all that far in the future as, as proof that I, like, was making a prediction with a lot of safety margin. But in, in 2004, that would've been pretty hard to convince you of 'cause there wouldn't have actually been an AI solving the harder general case of protein folding, a- and the timing, you know, the, the, or, and this particular form of AI that did it, that's, like, incredibly hard. So do, do I, nonetheless, taking a wild guess, expect this to happen in my lifetime? Yeah. My, my wild guess is that I'm very confident of that, if I don't get run over by a truck.

4. 7:29 – 9:37

   Does “godlike” capability matter? Chess analogies and what counts as dangerous power

   1. GHGeorge Hotz7:29

      Okay. Um, let's talk about AlphaFold. So I think the form does matter. I think the form is very important. Uh, when you were maybe talking about this in 2005, when I read all the Sequences Less Wrong stuff, 2010, you were thinking about Bayesian AIs that were going to figure out the world from first principles. Now, maybe not exactly that, but that's kind of where we were. But it's important how AlphaFold did it. AlphaFold did not start with the basic laws of physics and then figure out how proteins will fold. AlphaFold was trained on a huge amount of experimental data to extrapolate from that data. I don't doubt that these systems are going to get better. I don't doubt that they're eventually going to surpass us. I do doubt that they are going to have magical or godlike properties like solving the protein structure prediction problem from, you know, the, the, from quantum field theory, right? I, I-

   2. EYEliezer Yudkowsky8:34

      They don't have to.

   3. GHGeorge Hotz8:35

      Well-

   4. EYEliezer Yudkowsky8:36

      Right? Like, why, why do, what, they, they don't need to. There's protein-

   5. GHGeorge Hotz8:39

      Right.

   6. EYEliezer Yudkowsky8:39

      ... structure data to learn from. They don't need to do it-

   7. GHGeorge Hotz8:42

      Yes.

   8. EYEliezer Yudkowsky8:42

      ... from quantum field theory. Something can be not godlike and still more powerful than you, right? Like, like, like you look at the world, world chess champion Magnus Carlsen, who by objective, by which I mean AI measurements is probably the strongest human player who ever lived.

   9. GHGeorge Hotz8:59

      Sure.

   10. EYEliezer Yudkowsky8:59

       He's not God. He's not infinitely smart. He starts off on a chessboard that with no more resources than you have, and he predictably wipes the board with you 'cause he doesn't have to be godlike to defeat you or me, to be clear. I also can be defeated by being short of godhood.

   11. GHGeorge Hotz9:17

       Um, Magnus Carlsen can't make diamond nanobots. Do we agree on that statement?

   12. EYEliezer Yudkowsky9:23

       I, uh, well, we, we haven't ac- well, not quickly. I'm not sure what happens if you give him a million-

   13. GHGeorge Hotz9:30

       (laughs)

   14. EYEliezer Yudkowsky9:30

       ... if you give him a million years to work on it, then, then I'm not sure what happens. Like, I, I agree that, that he probably can't do it quickly.

5. 9:37 – 15:52

   Why timing matters (or doesn’t): economic growth, thresholds, and “pause button” governance

   1. GHGeorge Hotz9:37

      Okay. Um, so let's talk about timing, because timing, uh, sort of matters a lot.

   2. EYEliezer Yudkowsky9:44

      Why?

   3. GHGeorge Hotz9:46

      Well, because it depends when we should shut it down, right? Well, it definitely does.

   4. EYEliezer Yudkowsky9:53

      I mean, if there's like a predictive, or if there, if there's some kind of predictable phenomenon where you, you can, like, dance around the bullets and know that, like, like, things will become dangerous at, like, this time, but, like, no earlier than that, and we're like, okay, if we put the following, like, precautions into place at this future time, which is not now, we're sure we're going to do it later, 'cause people sure do talk a lot of crap about stuff that they claim will be done later and that never gets done. But-

   5. GHGeorge Hotz10:26

      Sure.

   6. EYEliezer Yudkowsky10:26

      ... so, so, you know, like, there's, there's this possibility that we could, like, be clever and dance around bullets if we knew exactly where the bullets were and we could actually coordinate on clever future strategies like that, which I don't think we can.

   7. GHGeorge Hotz10:38

      Okay.

   8. EYEliezer Yudkowsky10:39

      So that said, why do, why does timing matter?

   9. GHGeorge Hotz10:41

      Well, let's, let's start with the basic, and this is related to your question of why timing matters. Um, do you accept that it will not be hyperbolic, right? Staring into the singularity talks about a hyperbolic sequence, a sequence that has a singularity, that has a finite-

   10. EYEliezer Yudkowsky10:55

       Important context, I wrote this when I was 16 years old.

   11. GHGeorge Hotz10:58

       Okay, so you-

   12. EYEliezer Yudkowsky10:59

       And I, and I think that should be said out loud for, for the viewers. That said, yeah, I, I, I doubt-

   13. GHGeorge Hotz11:03

       Okay.

   14. EYEliezer Yudkowsky11:03

       ... it's going to be hyperbolic. Like, it-

   15. GHGeorge Hotz11:05

       Okay.

   16. EYEliezer Yudkowsky11:05

       ... it, it could be, like, very roughly hyperbolic up until a point, or it could be expon- you know, like, exponential on a sharp exponent up until a point. It, it, it could be-

   17. GHGeorge Hotz11:19

       Oh.

   18. EYEliezer Yudkowsky11:19

       ... some other weird curve that was like doo, doo, doo, doo, doo, doo, doo, doo, doo.

   19. GHGeorge Hotz11:23

       Yeah. I, I, I don't mean to, I don't mean to pin, but okay. Like, I, I, like, the timing definitely does matter, right? Because-

   20. EYEliezer Yudkowsky11:30

       But why?

   21. GHGeorge Hotz11:31

       Well, because without AI, we're on the same trajectory, right? AI might be an accelerant to diamond nanobots. But if, you, you, you, you would-

   22. EYEliezer Yudkowsky11:41

       I agree.

   23. GHGeorge Hotz11:41

       ... do you believe... I mean, you said... Okay. You said this about Magnus Carlsen too, right? Um, that he would eventually-

   24. EYEliezer Yudkowsky11:46

       Hum-

   25. GHGeorge Hotz11:46

       ... get there. Yeah.

   26. EYEliezer Yudkowsky11:47

       Humans would get there. Yup.

   27. GHGeorge Hotz11:49

       Humans will get there, right?

   28. EYEliezer Yudkowsky11:49

       The, the endpoint is much more predictable than the pathway. I don't know when humans would get there, but we would get there.

   29. GHGeorge Hotz11:55

       Yes. And I agree with you. I agree that we will get there. I, actually, I really hope we get there. Um, I don't want it to be tomorrow. That would be terrifying. Um, if we do it slowly-... if we do it, not super slowly, but if we start to expand out across the galaxy and we eventually unlock these wild and amazing technologies, that sounds pretty awesome to me. What doesn't sound awesome to me is a bunch of GPUs, uh, you know, going from chat GPT can kinda talk to you to boom, diamond nanobots overnight. I agree, that sounds horrifying. But it sounds like-

   30. EYEliezer Yudkowsky12:30

       What if it's a week instead of overnight? What if it's a month-
6. 15:52 – 25:27

   “Humans + tools” vs AI as a separate center of gravity

   1. GHGeorge Hotz15:52

      Oh, I very much disagree with this. Well, so I also... I somewhat object to the line between humanity and the machines, right? A lot of our intelligence is externalized. Um-

   2. EYEliezer Yudkowsky16:08

      Um, I, I mean, that's the way it is when you've got an intelligence over here that's using a bunch of responsive tools out there. There's, there's no que- there's only one center of gravity there. It, it, it's like looking at a star system and be- and being like, "Well, there's no point in drawing a firm boundary between the sun and the planets. They're all just in space." And, you know, like they're all just oc-... and, you know, sure, they're all ultimately just like objects in space, but one of them is far more massive than the others, and that's humans with the tools we have now.

   3. GHGeorge Hotz16:41

      Is your concern the bandwidth of the link? Is that what you're saying? Like, I'm not one with my tools because of the bandwidth of the link?

   4. EYEliezer Yudkowsky16:48

      Um-

   5. GHGeorge Hotz16:48

      Why are me and... Why am... Why are me and my computer not, like, a shared intelligence?

   6. EYEliezer Yudkowsky16:53

      Well, because there's one thi-... Because your brain is much more powerful than the computer at present. Like, not in terms of operations per second, but in terms of what you can do.

   7. GHGeorge Hotz17:01

      I'm not that sure about that. I think GPT-4 is... I'm a bit smarter than it, but not that-

   8. EYEliezer Yudkowsky17:09

      It's-

   9. GHGeorge Hotz17:09

      It's getting there.

   10. EYEliezer Yudkowsky17:09

       It's, it's, it's a little, but it-

   11. GHGeorge Hotz17:11

       Particularly, yeah.

   12. EYEliezer Yudkowsky17:12

       It's, it's not its own center of gravity. It's, it's like Jupiter to, like, the, the, the Mars of GBT-3 or something.

   13. GHGeorge Hotz17:20

       Yeah. I mean-

   14. EYEliezer Yudkowsky17:21

       But, you know, it's nowhere, nowhere near the sun.

   15. GHGeorge Hotz17:23

       An- another thing also is that, like, I don't think that capabilities... I don't think that intelligence falls on a nice line, right? Computers have been superhuman at adding for a long, long time. Computers are still far subhuman at plumbing, all right? And somewhere in the middle, we have things like chess and Go. Um, so when I mean that, like, like the tools that I use, the information age tools make me way smarter, all right? And you can use the, the, like, operant definition of intelligence and being able to, like, what I could affect in the world, right? Like, again, it's not instantaneous. Your intelligence ain't gonna save you against a bear. But if you asked me to, like, with my modern stuff on my computer, understand the operation of a-... 1800s era, like, Dutch India Trading Company. Oh, I think I could understand their operations super well. I have spreadsheets, I can start to put things in. I can forecast trend lines. So my point is, it is a form of intelligence that's far beyond human intelligence, a human plus a computer.

   16. EYEliezer Yudkowsky18:31

       Um, a human and a chess engine is, like, a, a modern chess engine. The era of centaur chess is-

   17. GHGeorge Hotz18:40

       Mm-hmm.

   18. EYEliezer Yudkowsky18:40

       ... effectively over. Like, the human plus the chess engine is as smart as the chess engine. The thing that makes the decisions is the chess engine, and if you try to take the decision-making capability into yourself, you either follow, follow its advice or you lose to a chess engine without the human attached.

   19. GHGeorge Hotz18:58

       And-

   20. EYEliezer Yudkowsky18:59

       And that gets into the lack of bandwidth issue, the lack of integration. That chess machine's over there, you're over here, and it is the sun, and you are Mars.

   21. GHGeorge Hotz19:12

       Well, but what do you mean? I can use the chess machine. I agree that if I was playing a game against Magnus Carlsen and I was allowed to use my phone, I'd crush him.

   22. EYEliezer Yudkowsky19:19

       Hey- (laughs)

   23. GHGeorge Hotz19:20

       I wouldn't, I wouldn't try to think too much about what the machine's telling me to do. I'm not that good at chess.

   24. EYEliezer Yudkowsky19:24

       Yeah.

   25. GHGeorge Hotz19:25

       Um-

   26. EYEliezer Yudkowsky19:25

       So, so what you d- you can use the chess engine because there's a larger game board in which play a game of chess is a move, and you understand that larger game board and the chess engine does not.

   27. GHGeorge Hotz19:40

       I don't know if I buy this. Uh, uh, uh, I, I, I don't think, like-

   28. EYEliezer Yudkowsky19:44

       (laughs)

   29. GHGeorge Hotz19:44

       ... this is a... I don't think this is that relevant to... My, my only point is that, um, humanity, like, we have super intelligences, right? They're corporations-

   30. EYEliezer Yudkowsky19:57

       No.
7. 25:27 – 26:31

   Multiplicity doesn’t save you: moons, suns, and instrumental convergence

   1. EYEliezer Yudkowsky25:27

      Then I think that all those peop- so there's, there's sort of... so, like, there's AIs that are heavier than the humans, more powerful than the humans, that are the suns to our planets, and then there's the AIs that are the moons to our planets, that still orbit us. And m- and the problem I have is that the humans and their moons cannot defeat the suns, and the suns are working with, with each other rather to... rather than working-

   2. GHGeorge Hotz25:55

      Uh-huh.

   3. EYEliezer Yudkowsky25:55

      ... with the humans.

   4. GHGeorge Hotz25:56

      Okay.

   5. EYEliezer Yudkowsky25:56

      This, this is sort of metaphorically my concern here, that there are-

   6. GHGeorge Hotz25:59

      Oh, good.

   7. EYEliezer Yudkowsky25:59

      ... things around that are much smarter than us.

   8. GHGeorge Hotz26:01

      Mm-hmm.

   9. EYEliezer Yudkowsky26:02

      They are not working for humans. They are not yours. It doesn't matter that you own, in human legal terms, the hardware that they're running on. They can... th- any attempts that you make to play them off against each other, they will laugh at. They will see through. They're not dumb like us. And th- and that system of AIs, for all its multiplicity, ends up killing you in much the same way as a single AI.

   10. GHGeorge Hotz26:29

       Sure.

   11. EYEliezer Yudkowsky26:29

       It doesn't care about you. It runs you over.

8. 26:31 – 32:46

   From atoms to negentropy: how humans could be wiped out as a side effect

   1. GHGeorge Hotz26:31

      Are they racist?

   2. EYEliezer Yudkowsky26:34

      Um, they don't care.

   3. GHGeorge Hotz26:36

      No, I mean-

   4. EYEliezer Yudkowsky26:37

      If that... so you'd have to tell me-

   5. GHGeorge Hotz26:38

      Okay.

   6. EYEliezer Yudkowsky26:38

      ... wheth- whether that's racism that they don't care.

   7. GHGeorge Hotz26:40

      Maybe, maybe not racist. Maybe are they speciesist, right? You think they're all gonna gang up against the humans.

   8. EYEliezer Yudkowsky26:47

      Well, I think they're going to eat the surrounding galaxies.

   9. GHGeorge Hotz26:51

      Okay.

   10. EYEliezer Yudkowsky26:51

       And insofar as humans have the conceit that they were playing off AIs against each other, that will not happen, and they will eat the galaxies in a cooperative fashion, possibly eating some of their own kind, if those ones were too weak to be part of the bargaining process.

   11. GHGeorge Hotz27:06

       So this isn't what happens almost ever. Um, it... if you look at almost all human conflict throughout history, it's not only been between groups of humans, right? We, we didn't fight World War II against the bears. We fought World War II against humans, and not just humans, but humans that looked surprisingly similar to us. This notion that it's the machines versus humanity is a very common sci-fi trope, but in reality, you fight against things that have resources you care about. I'm gonna say your line about the atoms, unless... do you wanna say it?

   12. EYEliezer Yudkowsky27:42

       Sorry, what?

   13. GHGeorge Hotz27:43

       Your line about the atoms. I'm not... the AI doesn't love me or hate me. I'm made of atoms it could use for something else.

   14. EYEliezer Yudkowsky27:50

       You are made of atoms that can be used-

   15. GHGeorge Hotz27:52

       Yeah.

   16. EYEliezer Yudkowsky27:52

       ... for something else. That's not the primary reason it would wipe you out in a hurry.

   17. GHGeorge Hotz27:56

       Okay.

   18. EYEliezer Yudkowsky27:56

       But you are made of a- but it will want all the atoms, and you are made of atoms it can use for something else.

   19. GHGeorge Hotz28:03

       Um, I'm not made of rare atoms. I'm made of-

   20. EYEliezer Yudkowsky28:08

       You're, you're made of negentropy.

   21. GHGeorge Hotz28:10

       Okay.

   22. EYEliezer Yudkowsky28:10

       Okay, yeah. It's not quite the atoms. It's the negentropy.

   23. GHGeorge Hotz28:13

       Okay.

   24. EYEliezer Yudkowsky28:13

       But-

   25. GHGeorge Hotz28:14

       So wait.

   26. EYEliezer Yudkowsky28:14

       ... you know, same effect.

   27. GHGeorge Hotz28:15

       Is it gonna... is it gonna enslave me or disassemble me? 'Cause if it disassembles me, it doesn't get the negentropy, right?

   28. EYEliezer Yudkowsky28:20

       No, i- i- if it, if it disassembles you, it gets... yeah, it does get the negentropy. You have chemical energy. You are, like, not in the c-... you are not in the configuration of-

   29. GHGeorge Hotz28:30

       (laughs)

   30. EYEliezer Yudkowsky28:30

       ... minimum ch- chemical potential energy. It can set you on fire to release chemical energy, not only... not literally, 'cause that wastes a lot of the chemical potential energy. You are made of atoms that are not iron on the periodic table. You can be fused, or a few bits fizzed, and above all, you're made of mass, and you can be thrown into things to generate-
9. 32:46 – 51:45

   Self-modification and “giant inscrutable matrices”: will deep learning rewrite itself?

   1. GHGeorge Hotz32:46

      So I'm not sure. And actually, this is another like... Let's talk about, let's talk about AIs rewriting their own source code. This is a common thing you bring up, right?

   2. EYEliezer Yudkowsky32:57

      I mean, I do talk a bit less about it nowadays, but I used to talk about it a lot, yeah.

   3. GHGeorge Hotz33:02

      Um, do you talk less about it now because you see how expensive and long the training runs are?

   4. EYEliezer Yudkowsky33:07

      Uh, that's not why I talk less about it now. (laughs)

   5. GHGeorge Hotz33:09

      Oh. Why, why do you talk less about it now?

   6. EYEliezer Yudkowsky33:12

      Um, I mean, in part because it no longer... because it sparks incredulity and you no longer need to postulate that in order to explain to people where intelligence can come from. People are manufacturing intelligence right now, that you don't need to, like, trip them up on the concept of an AI writing an AI.

   7. GHGeorge Hotz33:34

      Yeah. Well, but people are manufacturing intelligence right now. You know, it's interesting. The AI that we try to build, you know, we try to build these AIs to mimic humans as closely as possible.

   8. EYEliezer Yudkowsky33:49

      To predict humans, and then we use them to imitate humans. But they are trained to predict and used to imitate.

   9. GHGeorge Hotz33:58

      So are we.

   10. EYEliezer Yudkowsky33:59

       Sure. Go on.

   11. GHGeorge Hotz34:00

       I'm, I'm trained to predict, and then I imitate?

   12. EYEliezer Yudkowsky34:03

       Well, GPT-4 is trained to predict the next word, and then they produce an imitation via asking it over and over again to predict what a human would say in that circumstance.

   13. GHGeorge Hotz34:14

       Mm-hmm.

   14. EYEliezer Yudkowsky34:15

       But it is not like a generative adversarial network where it's like being trained to produce a typical output, and then another thing is checking that to see if it looks typical or not. It is being trained to predict over and over, and, and these are like somewhat different complexity classes. Though you can, like, switch around like GANs and do conditional GANs, and then it's the same class. But there, there is a difference between, like, be a typical human-

   15. GHGeorge Hotz34:42

       Yes.

   16. EYEliezer Yudkowsky34:43

       ... and be able to predict any human you found on the internet.

   17. GHGeorge Hotz34:47

       So sure, yes. You're, you're, you're asking for the probability of the next symbol, and you're not talking about the, the, uh, like the probability space. Like, you're not talking about like... But why do you think humans are the other thing? Why do you think humans are not just what GPT is?

   18. EYEliezer Yudkowsky35:01

       I'm... Well, humans ha- uh, I think have a lot of structural properties that so far as we, that we have for which we have not yet detected analogs in, within GPT-4, although like heaven knows we, we can't look in there very well. Um, you know-

   19. GHGeorge Hotz35:14

       We can't look in the brain either.

   20. EYEliezer Yudkowsky35:16

       Hu- humans got like a, a cerebellum, which is motor control and error correction, and maybe you, you could make a transformer layer do that, but we don't, like-

   21. GHGeorge Hotz35:25

       And GPT-

   22. EYEliezer Yudkowsky35:25

       ... there's GPT doing it yet. I, I mean-

   23. GHGeorge Hotz35:27

       GPT has a matrix at layer 970.

   24. EYEliezer Yudkowsky35:32

       Yeah. So, so, so, like, hu- humans, humans predict, humans manipulate, humans have this whole complicated brain that is like, but that like the, like, at least looks on the outside like a more complicated architecture than GPT has. Humans are clearly doing a bunch of prediction, but we're also doing like a bunch of decision problems.

   25. GHGeorge Hotz35:51

       Yes. Um, I, I, one of my big questions, 'cause I wanna build it, is what is the loss function for life?

   26. EYEliezer Yudkowsky35:58

       Inclusive genetic fitness. Do you have any other questions?

   27. GHGeorge Hotz36:00

       Oh, okay, no. No, sorry. I, I don't mean life in general. I mean an individual human. Of course that's the loss function for life.

   28. EYEliezer Yudkowsky36:07

       I mean, I don't think a human has a loss f- you know, there, there's gonna be like... So we've got like pain, and pleasure, and like our brains flinching away from future anticipated pain, and prediction errors where we're like, "What?" And like-

   29. GHGeorge Hotz36:29

       You mean like how-

   30. EYEliezer Yudkowsky36:29

       ... perception errors.
10. 51:45 – 1:01:10

    Goals, optimization, and how agency emerges (humans vs models)

    1. GHGeorge Hotz51:45

       I'll tell, I'll tell a story from personal experience. What I've found is that machines are almost always aligned with me. I have almost never come across a machine, certainly not a machine that I owned, that was not aligned with me.

    2. EYEliezer Yudkowsky52:00

       Very few of the machines that you own have goals such that they could be aligned or misaligned with you.

    3. GHGeorge Hotz52:07

       I mean, goals are-

    4. EYEliezer Yudkowsky52:07

       I would essentially say none.

    5. GHGeorge Hotz52:09

       Goals are an interesting, goals are an interesting word, right? Like, like, like, when do the machines decide to have goals to get rid of me? Right? Like, when does this happen? How does this happen? They all agree by exchanging inscrutable matrices with each other and saying, "We're all gonna cooperate. Fuck the humans."

    6. EYEliezer Yudkowsky52:29

       Um, so as you make thi- ... So as natural selection built humans to be better and cognitively better at the problems of chipping flint hand axes, throwing things in a way that hit other things, and above all, outwitting their other humans for status, and mates, and resources-

    7. GHGeorge Hotz52:51

       Chimpanzee politics.

    8. EYEliezer Yudkowsky52:52

       ... and political power. Hm?

    9. GHGeorge Hotz52:54

       Chimpanzee politics.

    10. EYEliezer Yudkowsky52:55

        Turned into human politics, only not literally, 'cause actually branching point in the past, but... Um, so it's not that squishy things naturally have goals. It's that having goals is a natural way of solving problems, and natural selection in the process of hill climbing, not aiming for things with goals, not even aiming explicitly for things with intelligence, just trying to maximize inclusive genetic fitness, just solve the problem of chipping the, the hand axes, eventually spit out things with the ability to reason across a very wide range of problems, learn new problems, solve new problems, combine knowledge from multiple domains, um, invent writing so that it starts to cumulate in a way it hadn't cumulated in the ancestral environment. Um, and it turned out that, that the, what hill climbing found for the intelligence that turned out to generalize in this way that started to, like, cohere and bootstrap, although that process is by no means completed. Humans are still pretty incoherent. But, like, they invented science and some of them were able to use it, and they, like, uh, they had this, like, knowledge transmitted through writing that they'd invented about how to science and some people could use it. And, you know, and when y- ... And in the course of hill climbing, building an intelligence that was powerful enough to start to coalesce and become more powerful, that intelligence turned out to be structured around a set of wants, desires, preferences, and it's a mathematical fact that if you just have a bunch of things pointing in different directions, they will step on each other and not be as resource efficient as they could be. So, as those things start to coalesce, they even started to imagine themselves as having goals and ask, "What are my goals?" instead of just, like, running off in lots of little local directions. You know, some of them.... um, the, you know, John von Neumann even contributed to the notion of a utility function, although this was invented, you know, like, thousands and thousands of years after writing. Um, (clears throat) and, you know, this is the story of humanity. It's a complicated story, um, but the moral is, is that, uh, the moral I would say is that sort of like I, I, one thing seems to me to be fairly inextricable from intelligence, especially the way hill climbing does it. Like, when you run acr- when you have this, like, larger environmental problem like chipping a hand axe, like, to solve this at a sufficient level, smarter than the bees, smarter than the, smarter than the bees building hives, smarter than the beavers building dams. At the human level, you got a thing that looks at the, the hand axe and it, it starts to think that symmetrical things are prettier. And it chips away at it until it looks symmetrical, and it says, like, "Well, if I chip here, then the thing I look at will be more symmetrical. It will be prettier."

    11. GHGeorge Hotz55:57

        Mm-hmm.

    12. EYEliezer Yudkowsky55:58

        And this is not without valence. This is not without wanting valence.

    13. GHGeorge Hotz56:04

        Okay. Why do you think the AIs are gonna be different, right? So right now, when we-

    14. EYEliezer Yudkowsky56:09

        Uh, uh, I mean, it wouldn't be v- very surprising... I, I mean, part of my thesis is that they indeed, like, in the process of people training AIs to be better and better at stuff, they got smarter, and the smartness goes along with desires laced through it.

    15. GHGeorge Hotz56:24

        You know, doesn't the orthogonality thesis apply to humans too?

    16. EYEliezer Yudkowsky56:30

        Um, that's a very strange cons-

    17. GHGeorge Hotz56:33

        What do you mean? That-

    18. EYEliezer Yudkowsky56:34

        I-

    19. GHGeorge Hotz56:34

        ... do you think everyone who's 150 IQ is nice and everyone who's 70 IQ is mean, or vice versa? It seems like intelligence and how good of a person you are are completely uncorrelated.

    20. EYEliezer Yudkowsky56:46

        Okay. First of all, uh, very few things are uncorrelated with intelligence.

    21. GHGeorge Hotz56:51

        (laughs) Sure.

    22. EYEliezer Yudkowsky56:51

        That, I bet you's saying it's empirically false. Um-

    23. GHGeorge Hotz56:54

        Smart people are actually really dicks. (laughs)

    24. EYEliezer Yudkowsky56:56

        Yeah. I- i- they, they, you know, like, I, if I had to guess if it would, like, lean mean or lean nice, I would guess nice, you know, at least within, like, my culture that defined what nice was in the first place. Um, but I, but I sure wouldn't bet a zer- bet zero correlation. Very few things are not correlated. Um-

    25. GHGeorge Hotz57:13

        Sure.

    26. EYEliezer Yudkowsky57:13

        Orthogonality is, like, a statement about the whole mind design space that for every kind of goal that could be stated, like, the question, like, to the extent that you can ask, "How would one pursue this goal if one had it?" you can have a mind that pursues that goal. To the extent that it's coherent to ask, "What would I need to do in order to turn a galaxy into spaghetti?" To the extent that it's coherent to ask, like, "How would I go about turning a galaxy into spaghetti if aliens offer to pay us, you know, like, some vast, super-universal quantity of resources to do that?" If you can coherently ask that question, there is also some mind design that seeks to turn a galaxy into spaghetti. And that, that's how I would describe the orthogonality.

    27. GHGeorge Hotz57:59

        Yes, yes.

    28. EYEliezer Yudkowsky57:59

        Now, me personally, my goals could very well be affected if you, if you dropped another 20 IQ points on me.

    29. GHGeorge Hotz58:07

        Well, sure. But, uh, let's even come back to, like, you can certainly desire turning the galaxy into spaghetti. I wanna bet-

    30. EYEliezer Yudkowsky58:17

        I mean-
11. 1:01:10 – 1:11:13

    Hardness of ‘endgame’ tech: nanobots, biotech risk, and limits of search

    1. GHGeorge Hotz1:01:10

       Okay. Well, I mean, let's also, like, let's really drill down on what these end of world scenarios are. Do you want to posit, like, protein synthesis and diamond nanobots?

    2. EYEliezer Yudkowsky1:01:22

       I mean, if I'm going to lose a bunch of viewers that way, I might have to pick some, you know, like, easier to understand process. Like, we're talking about like-

    3. GHGeorge Hotz1:01:30

       Okay.

    4. EYEliezer Yudkowsky1:01:30

       ... 1823 versus 2023. You know, if you, if you're trying to explain it to 1823, maybe you just talk about, like, the powerful explosive artillery shells, and you don't mention the nuclear weapons.

    5. GHGeorge Hotz1:01:40

       Sure.

    6. EYEliezer Yudkowsky1:01:41

       'Cause they don't get that part. So, similarly, you know, like, if we don't wanna start diving into this book over here, then maybe, may- maybe we want to talk about something like, you know, like standard biological weapons or something.

    7. GHGeorge Hotz1:01:57

       (laughs) .

    8. EYEliezer Yudkowsky1:01:57

       But, you know, but in, in real life, sure. In r- in real life, it, it, you know, doesn't-

    9. GHGeorge Hotz1:02:02

       Well-

    10. EYEliezer Yudkowsky1:02:02

        ... use the squishy stuff.

    11. GHGeorge Hotz1:02:03

        No, I'm not trying to, I'm not trying to say that, that nanobots are impossible. What I'm trying to say is that nanobots are extremely, extremely hard, right?

    12. EYEliezer Yudkowsky1:02:13

        Why?

    13. GHGeorge Hotz1:02:13

        And... To figure out?

    14. EYEliezer Yudkowsky1:02:16

        Why?

    15. GHGeorge Hotz1:02:16

        Well, 'cause it, 'cause it's a really hard search problem, right?

    16. EYEliezer Yudkowsky1:02:19

        Why?

    17. GHGeorge Hotz1:02:21

        Why is it a hard search problem?

    18. EYEliezer Yudkowsky1:02:23

        Yeah.

    19. GHGeorge Hotz1:02:24

        I mean, can you make nanobots? I know I can't.

    20. EYEliezer Yudkowsky1:02:28

        No, but I'm a very weak search process. I can't even solve the protein folding problem, which, which, you know, like some lesser s- you knows dumber than human AIs have already done.

    21. GHGeorge Hotz1:02:37

        You know what else you-

    22. EYEliezer Yudkowsky1:02:37

        I can't even beat Stockfish 15 at chess.

    23. GHGeorge Hotz1:02:39

        You know what else you can't do? You can't, uh, find the key in AES 256.

    24. EYEliezer Yudkowsky1:02:45

        Well, that possi- I'm, I'm not sure how quantum hardened that is, but is, is that the kind of problem which you can't solve even with a Dyson sphere?

    25. GHGeorge Hotz1:02:53

        Uh, a Dyson sphere, I'm not sure. A quantum hardened, I think so. I mean, look, we don't actually know. It's possible that... Do you think P equals NP?

    26. EYEliezer Yudkowsky1:03:02

        Uh, I defer to the experts who guess n-o.

    27. GHGeorge Hotz1:03:05

        (laughs) Yeah. Okay. So, as long as we agree about this, then, you know, I'm not even... Look, I don't know enough to say exactly, does that imply that one-way functions are possible? But you say that it's a search problem, right? Well, AES is a search problem too.

    28. EYEliezer Yudkowsky1:03:19

        Yeah, you can't solve all the search problems, even if you're G- even if you're God, 'cause you know, for God we just state-

    29. GHGeorge Hotz1:03:24

        Exactly.

    30. EYEliezer Yudkowsky1:03:24

        ... like, trans-God search problems.
12. 1:11:13 – 1:19:23

    Compute, efficiency, and headroom above biology (Landauer limit dispute)

    1. EYEliezer Yudkowsky1:11:13

       Like, like, here we are. We- we- we- we... There's, like, enormous amounts of room above biology. I didn't even get into the part about, like, the fundamental, like, constraints that natural selection are under and, like, how we know that there's, like, enormous amounts of headroom above b- biology for artificial biology.

    2. GHGeorge Hotz1:11:28

       Wait.

    3. EYEliezer Yudkowsky1:11:29

       Um...

    4. GHGeorge Hotz1:11:29

       How close do you think the brain is to-

    5. EYEliezer Yudkowsky1:11:31

       We don't have that. We don't have that yet. And I- and-

    6. GHGeorge Hotz1:11:33

       How close do you think the brain is to the Landauer limit?

    7. EYEliezer Yudkowsky1:11:36

       The Landauer limit?

    8. GHGeorge Hotz1:11:37

       Yeah, the limit of possible compute.

    9. EYEliezer Yudkowsky1:11:43

       All right. So I'm 100 watts, and let's say I'm about 10 to the 17th operations per second. And I don't actually remember the Landauer limit, um, but I would guess somewhere about, uh, six orders of magnitude.

    10. GHGeorge Hotz1:11:57

        A lot closer. Okay. So, I- I can, I can give you a, like... If you wanna buy 20 petaflops of compute today, you need 16 H100s, right? It's gonna cost you about half a million dollars, and that machine's gonna use 20 kilowatts, right? Brain does the same amount, 10 to the 17th, 10... We're in the same order of magnitude. I think you even said a little bit more than me. I think it's, yeah, like, two, 2E16 or something, 20 petaflops. Um, so in order to get that much compute in a silicon computer, you need 1000X the power.

    11. EYEliezer Yudkowsky1:12:33

        Yeah?

    12. GHGeorge Hotz1:12:33

        I did the math. I did the math. Using the kind of silicon computers we're using today, we are really close to the Landauer limit. We're off by a factor of about a hundred or a thousand. The brain may very well be at the Landauer limit for compute. The brain is really good in terms of efficiency.

    13. EYEliezer Yudkowsky1:12:52

        Deeply, biologically implausible. The reason being that each of your synaptic, um, s- like, each- each time your, one of your synap- W- well, axon terminal releases a bunch of neurotransmitter molecules onto a waiting synapse.... um, the, all of those molecules need to be pumped back in to the axon terminal.

    14. GHGeorge Hotz1:13:15

        Right.

    15. EYEliezer Yudkowsky1:13:15

        And each of those, in each, and every time it gets pumped from out to in, that's an irreversible operation. That must be at least one Flash of Landauer. And then you've got your neural impulses being transmitted via sections of neural, of, of neural membrane depolarizing, and the potassium item, ions going out... That, that, or sodium. I don't remember what it... which ion it is.

    16. GHGeorge Hotz1:13:40

        You, you-

    17. EYEliezer Yudkowsky1:13:40

        But the point is, you've got all these ions going in and out, and every one of those is one Landauer minute with 10 to the 17th power.

    18. GHGeorge Hotz1:13:45

        You, you might know a lot more bio than me. I, I, I don't know how to speak to this. But you agree. You said 10 to the 17th, so that's 100 petaflops, right?

    19. EYEliezer Yudkowsky1:13:52

        Yeah.

    20. GHGeorge Hotz1:13:53

        Okay. So, how much power does an 100 petaflop computer take today?

    21. EYEliezer Yudkowsky1:13:59

        Uh...

    22. GHGeorge Hotz1:13:59

        It's 100 kilowatts.

    23. EYEliezer Yudkowsky1:14:02

        Sounds, sounds legit.

    24. GHGeorge Hotz1:14:04

        So, the brain is so much more efficient than these computers are, right? The brain... Look, these superintelligences you're talking about, and I know we're kinda coming close to the end, I think these things are possible. But I think that the orders of magnitude of power and compute we need are so, so much more than anything like what humanity has today. Then I think even when they do exist, they're mostly gonna leave us alone because, not 'cause it can't mess with us, because why would it? What incentive does it have? I don't have anything it wants.

    25. EYEliezer Yudkowsky1:14:37

        Okay. To

    26. DPDwarkesh Patel1:14:37

        Permit this-

    27. GHGeorge Hotz1:14:37

        After colonizing the whole galaxy? Fine, it comes back for me.

    28. EYEliezer Yudkowsky1:14:40

        To prevent us from making other superintelligences that could compete with it for resources, is, is like the first, is, is like the reason to wipe us out on purpose. If it is doing a bunch of compute on Earth's surface because it started there, or before, or like before spreading, then we've got a bunch of water in our oceans that can be turned into fusion energy. And the main limit on that is how fast Earth can radiate heat once you've used all the existing stuff as a heat sink. That's not very survivable. Like that, that kills us off as a side effect.

    29. GHGeorge Hotz1:15:15

        This is, again, assuming that this thing is a god, not kinda close to humans, but a bit smarter. And yes, might it get to a god, but the timing matters. It's not 10 years.

    30. EYEliezer Yudkowsky1:15:25

        Humans-
13. 1:19:23 – 1:27:33

    Coordination among AIs: bargaining, prisoner’s dilemma, and whether conflict is inevitable

    1. GHGeorge Hotz1:19:23

       that I've heard-... about the foom scenario is uncovering new algorithms on silicon stack things that are 10,000x better and it fooms overnight. If you're imagining that this thing is going to need to build new fabs, like, how is it doing this overnight? It's gonna, it's gonna-

    2. EYEliezer Yudkowsky1:19:44

       Um.

    3. GHGeorge Hotz1:19:44

       ... do it with one protein? It's gonna-

    4. EYEliezer Yudkowsky1:19:45

       I feel like we've got one... Uh, no, it doesn't, I don't, uh, yeah, I don't, I, I don't think it needs the, like, actual, like, tiny reversible computing elements-

    5. GHGeorge Hotz1:19:54

       Okay.

    6. EYEliezer Yudkowsky1:19:55

       ... to foom. I'm just saying that, like, once it fooms, it is not sticking with the silicon stack. Does this-

    7. GHGeorge Hotz1:19:59

       It, like, but we've established that it doesn't foom. It slowly increases power on an exponential along with humanity.

    8. EYEliezer Yudkowsky1:20:11

       (sighs) Humanity grows more slowly. So as the things that we are u- that we were using as our tools back when we were planets and they were moons become planets and then suns, they... If we are using that scenario instead of the one that I think is, is a bit more likely, the suns do eventually collaborate and wipe us out. And in that scenario, then yes, they might be running on GPUs at the point where they figure out how to build their own, you know, non-van der Waals based collection of carbon atoms. And, um, and, and that is the point where I think they can wipe out humanity from there. It's a technological bootstrapping process, that the-

Episode duration: 1:34:29