George Hotz vs Eliezer Yudkowsky

1. 0:00 – 3:33

   George Hotz opens: skepticism of “foom” and singularity narratives

   1. DPDwarkesh Patel0:00

      Okay. We are gathered here to witness George Hotz and Eliezer Yudkowsky debate and discuss, live on Twitter and YouTube, AI safety and related topics. You guys already know who George and Eliezer are, so I, I don't feel like introduction is necessary. I'm Dwarkesh. I'll be moderating. I'll mostly stay out of the way, um, except to kick things off by letting George explain his basic position. And we'll take things from there. George, I'll kick it off to you.

   2. GHGeorge Hotz0:31

      Sure. Um, so I took an existentialism class in high school, and you'd read about these people, Sartre, Kierkegaard, Nietzsche, and you wonder, "Who were these people alive today?" And I think I'm sitting across from one of them now. Um, rationality and the sequences, uh, this whole field, the whole Less Wrong cinematic universe, uh, have impacted so many people's lives in, I think, a very positive way, including mine. Um, not only are you a philosopher, you're also a, a great storyteller. Um, there's two books that I've picked up and, you know, it was like crack. I couldn't put them down. Uh, one was Atlas Shrugged and the other one was Harry Potter and the Methods of Rationality. Um, it's a great book. Now, those are fictional stories. Um, you've also told some stories pertaining to the real world. Um, one was a story you told when you were younger about how "I remember the day I found staring into the singularity when I was 15." And it starts talking about Moore's law and how Moore's law is fundamentally a human law that says humans double the power of processors every two years. So once computers are doing it, it's going to be two years, but then next time it'll be one year, and then six months, and then three months, and then 1.5 and so on. And this is a hyperbolic sequence. Um, this is a singularity, and that's why it's called staring into the singularity. Then this document said that we were gonna... you know, the AI was gonna do wonderful things for us, we were gonna go colonize the universe, we were gonna go, you know, go forth and do all things till the end of all ages. Um, then you changed your views, and super intelligence does not imply super morality. The orthogonality thesis, I'm not going to challenge it. It is obviously a true statement. Then you kept the basic premise of the story, the recursively self-improving, foom, criticality AI. But instead of saving us, it was gonna kill us. I don't think either of these stories is right, and I don't think either of these stories is right for the same reason. I don't think AI can foom. I don't think AI can go critical. I don't think intelligence can go critical. I think this is an absolutely extraordinary claim. I'm not saying that recursive self-improvement is impossible. Recursive self-improvement is of course possible, humanity has done it. Every time you have used a tool to make a better tool, you have recursively self-improved. What I don't believe in is the AI that's sitting in a basement somewhere running on a thousand GPUs that is suddenly gonna crack the secret to thinking, recursively self-improve overnight, and then flood the world with diamond nanobots. This is an extraordinary claim and it requires extraordinary evidence, and I hand it over to you to deliver that evidence.

2. 3:33 – 5:20

   Eliezer’s core claim: doom doesn’t require rapid takeoff

   1. EYEliezer Yudkowsky3:33

      Heh. Well, first, let me say that I don't think that the scenario of us all perishing to non-super moral super intelligence requires that particularly rapid rate of ascent. It requires a large enough gap open up with humanity that hasn't followed along in time. And why be- be... is this a crux? Be- be- before we, we start arguing about whether like self-improvement of things on the large internet connected server clusters rather than basements that now prevail, um, before we start arguing about that part, let's first check where the disagreement lies. So from my perspective, if you've got a trillion beings that are, you know, sufficiently intelligent and smarter than us and not super moral, I think that's kind of game over for us. It... even if you got there via a slow 10 year process instead of a 10 hour process or a 10 week- day process or whatever, if you are at the end point where there's this like large mass of intelligence that doesn't care about you, I think that we are, we are dead. And I worry that our s- and, and more importantly, I worry that our successors will go on to do nothing very much worthwhile with the galaxies. So presumably you think that if things don't go quickly, then we're safe. I dispute that, and maybe that's the part we need to talk about.

3. 5:20 – 7:29

   Timelines and prediction difficulty: AlphaFold as an example

   1. GHGeorge Hotz5:20

      Sure. Um, well, let's start with, let's give an approximate timeline. We don't, we don't need an exact timeline, but you seem to think this is gonna happen in your lifetime?

   2. EYEliezer Yudkowsky5:33

      That's my wild guess. It is far easier to predict the end point than all the details of the process that takes us, that take us there. Timing is one of those details. Timing is really, really hard. In 2004, I made a prediction that super intelligence would eventually be able to solve the, a special case of the protein folding problem, which is you get to choose the DNA sequence, but you wanna choose a DNA sequence that folds into a shape with a chemical property.... and so I predicted that super intelligence would en- eventually be able to solve this easy special case of protein folding. Now, in reality, protein folding was cracked f- for the much harder general case of biology, was cracked by AI come about 2020 or so, AlphaFold2. Um, there was no way I could've made the timing. I could not even have been confident that the bio- biological case of protein folding was going to be crackable by something so much shorter of super intelligence. A- of course, people at the time said it wasn't possible, you know, for the AI can't do this, like, how do you know this problem was even solvable, et cetera, et cetera. And, you know, I could try to explain how I knew, but that would be a technical story. I would point the fact that a much easier s- sp- um, pardon me, that a much harder general case of the problem I pointed to was solved by a non-super intelligence not all that far in the future as, as proof that I, like, was making a prediction with a lot of safety margin. But in, in 2004, that would've been pretty hard to convince you of 'cause there wouldn't have actually been an AI solving the harder general case of protein folding, a- and the timing, you know, the, the, or, and this particular form of AI that did it, that's, like, incredibly hard. So do, do I, nonetheless, taking a wild guess, expect this to happen in my lifetime? Yeah. My, my wild guess is that I'm very confident of that, if I don't get run over by a truck.

4. 7:29 – 9:37

   Does the form of AI matter? Data-driven systems vs first-principles reasoning

   1. GHGeorge Hotz7:29

      Okay. Um, let's talk about AlphaFold. So I think the form does matter. I think the form is very important. Uh, when you were maybe talking about this in 2005, when I read all the Sequences Less Wrong stuff, 2010, you were thinking about Bayesian AIs that were going to figure out the world from first principles. Now, maybe not exactly that, but that's kind of where we were. But it's important how AlphaFold did it. AlphaFold did not start with the basic laws of physics and then figure out how proteins will fold. AlphaFold was trained on a huge amount of experimental data to extrapolate from that data. I don't doubt that these systems are going to get better. I don't doubt that they're eventually going to surpass us. I do doubt that they are going to have magical or godlike properties like solving the protein structure prediction problem from, you know, the, the, from quantum field theory, right? I, I-

   2. EYEliezer Yudkowsky8:34

      They don't have to.

   3. GHGeorge Hotz8:35

      Well-

   4. EYEliezer Yudkowsky8:36

      Right? Like, why, why do, what, they, they don't need to. There's protein-

   5. GHGeorge Hotz8:39

      Right.

   6. EYEliezer Yudkowsky8:39

      ... structure data to learn from. They don't need to do it-

   7. GHGeorge Hotz8:42

      Yes.

   8. EYEliezer Yudkowsky8:42

      ... from quantum field theory. Something can be not godlike and still more powerful than you, right? Like, like, like you look at the world, world chess champion Magnus Carlsen, who by objective, by which I mean AI measurements is probably the strongest human player who ever lived.

   9. GHGeorge Hotz8:59

      Sure.

   10. EYEliezer Yudkowsky8:59

       He's not God. He's not infinitely smart. He starts off on a chessboard that with no more resources than you have, and he predictably wipes the board with you 'cause he doesn't have to be godlike to defeat you or me, to be clear. I also can be defeated by being short of godhood.

   11. GHGeorge Hotz9:17

       Um, Magnus Carlsen can't make diamond nanobots. Do we agree on that statement?

   12. EYEliezer Yudkowsky9:23

       I, uh, well, we, we haven't ac- well, not quickly. I'm not sure what happens if you give him a million-

   13. GHGeorge Hotz9:30

       (laughs)

   14. EYEliezer Yudkowsky9:30

       ... if you give him a million years to work on it, then, then I'm not sure what happens. Like, I, I agree that, that he probably can't do it quickly.

5. 9:37 – 15:52

   Why timing matters: pause buttons, chip control, and governance proposals

   1. GHGeorge Hotz9:37

      Okay. Um, so let's talk about timing, because timing, uh, sort of matters a lot.

   2. EYEliezer Yudkowsky9:44

      Why?

   3. GHGeorge Hotz9:46

      Well, because it depends when we should shut it down, right? Well, it definitely does.

   4. EYEliezer Yudkowsky9:53

      I mean, if there's like a predictive, or if there, if there's some kind of predictable phenomenon where you, you can, like, dance around the bullets and know that, like, like, things will become dangerous at, like, this time, but, like, no earlier than that, and we're like, okay, if we put the following, like, precautions into place at this future time, which is not now, we're sure we're going to do it later, 'cause people sure do talk a lot of crap about stuff that they claim will be done later and that never gets done. But-

   5. GHGeorge Hotz10:26

      Sure.

   6. EYEliezer Yudkowsky10:26

      ... so, so, you know, like, there's, there's this possibility that we could, like, be clever and dance around bullets if we knew exactly where the bullets were and we could actually coordinate on clever future strategies like that, which I don't think we can.

   7. GHGeorge Hotz10:38

      Okay.

   8. EYEliezer Yudkowsky10:39

      So that said, why do, why does timing matter?

   9. GHGeorge Hotz10:41

      Well, let's, let's start with the basic, and this is related to your question of why timing matters. Um, do you accept that it will not be hyperbolic, right? Staring into the singularity talks about a hyperbolic sequence, a sequence that has a singularity, that has a finite-

   10. EYEliezer Yudkowsky10:55

       Important context, I wrote this when I was 16 years old.

   11. GHGeorge Hotz10:58

       Okay, so you-

   12. EYEliezer Yudkowsky10:59

       And I, and I think that should be said out loud for, for the viewers. That said, yeah, I, I, I doubt-

   13. GHGeorge Hotz11:03

       Okay.

   14. EYEliezer Yudkowsky11:03

       ... it's going to be hyperbolic. Like, it-

   15. GHGeorge Hotz11:05

       Okay.

   16. EYEliezer Yudkowsky11:05

       ... it, it could be, like, very roughly hyperbolic up until a point, or it could be expon- you know, like, exponential on a sharp exponent up until a point. It, it, it could be-

   17. GHGeorge Hotz11:19

       Oh.

   18. EYEliezer Yudkowsky11:19

       ... some other weird curve that was like doo, doo, doo, doo, doo, doo, doo, doo, doo.

   19. GHGeorge Hotz11:23

       Yeah. I, I, I don't mean to, I don't mean to pin, but okay. Like, I, I, like, the timing definitely does matter, right? Because-

   20. EYEliezer Yudkowsky11:30

       But why?

   21. GHGeorge Hotz11:31

       Well, because without AI, we're on the same trajectory, right? AI might be an accelerant to diamond nanobots. But if, you, you, you, you would-

   22. EYEliezer Yudkowsky11:41

       I agree.

   23. GHGeorge Hotz11:41

       ... do you believe... I mean, you said... Okay. You said this about Magnus Carlsen too, right? Um, that he would eventually-

   24. EYEliezer Yudkowsky11:46

       Hum-

   25. GHGeorge Hotz11:46

       ... get there. Yeah.

   26. EYEliezer Yudkowsky11:47

       Humans would get there. Yup.

   27. GHGeorge Hotz11:49

       Humans will get there, right?

   28. EYEliezer Yudkowsky11:49

       The, the endpoint is much more predictable than the pathway. I don't know when humans would get there, but we would get there.

   29. GHGeorge Hotz11:55

       Yes. And I agree with you. I agree that we will get there. I, actually, I really hope we get there. Um, I don't want it to be tomorrow. That would be terrifying. Um, if we do it slowly-... if we do it, not super slowly, but if we start to expand out across the galaxy and we eventually unlock these wild and amazing technologies, that sounds pretty awesome to me. What doesn't sound awesome to me is a bunch of GPUs, uh, you know, going from chat GPT can kinda talk to you to boom, diamond nanobots overnight. I agree, that sounds horrifying. But it sounds like-

   30. EYEliezer Yudkowsky12:30

       What if it's a week instead of overnight? What if it's a month-
6. 15:52 – 19:44

   Humans + tools vs AI as a new center of gravity (sun/planet/moon metaphor)

   1. GHGeorge Hotz15:52

      Oh, I very much disagree with this. Well, so I also... I somewhat object to the line between humanity and the machines, right? A lot of our intelligence is externalized. Um-

   2. EYEliezer Yudkowsky16:08

      Um, I, I mean, that's the way it is when you've got an intelligence over here that's using a bunch of responsive tools out there. There's, there's no que- there's only one center of gravity there. It, it, it's like looking at a star system and be- and being like, "Well, there's no point in drawing a firm boundary between the sun and the planets. They're all just in space." And, you know, like they're all just oc-... and, you know, sure, they're all ultimately just like objects in space, but one of them is far more massive than the others, and that's humans with the tools we have now.

   3. GHGeorge Hotz16:41

      Is your concern the bandwidth of the link? Is that what you're saying? Like, I'm not one with my tools because of the bandwidth of the link?

   4. EYEliezer Yudkowsky16:48

      Um-

   5. GHGeorge Hotz16:48

      Why are me and... Why am... Why are me and my computer not, like, a shared intelligence?

   6. EYEliezer Yudkowsky16:53

      Well, because there's one thi-... Because your brain is much more powerful than the computer at present. Like, not in terms of operations per second, but in terms of what you can do.

   7. GHGeorge Hotz17:01

      I'm not that sure about that. I think GPT-4 is... I'm a bit smarter than it, but not that-

   8. EYEliezer Yudkowsky17:09

      It's-

   9. GHGeorge Hotz17:09

      It's getting there.

   10. EYEliezer Yudkowsky17:09

       It's, it's, it's a little, but it-

   11. GHGeorge Hotz17:11

       Particularly, yeah.

   12. EYEliezer Yudkowsky17:12

       It's, it's not its own center of gravity. It's, it's like Jupiter to, like, the, the, the Mars of GBT-3 or something.

   13. GHGeorge Hotz17:20

       Yeah. I mean-

   14. EYEliezer Yudkowsky17:21

       But, you know, it's nowhere, nowhere near the sun.

   15. GHGeorge Hotz17:23

       An- another thing also is that, like, I don't think that capabilities... I don't think that intelligence falls on a nice line, right? Computers have been superhuman at adding for a long, long time. Computers are still far subhuman at plumbing, all right? And somewhere in the middle, we have things like chess and Go. Um, so when I mean that, like, like the tools that I use, the information age tools make me way smarter, all right? And you can use the, the, like, operant definition of intelligence and being able to, like, what I could affect in the world, right? Like, again, it's not instantaneous. Your intelligence ain't gonna save you against a bear. But if you asked me to, like, with my modern stuff on my computer, understand the operation of a-... 1800s era, like, Dutch India Trading Company. Oh, I think I could understand their operations super well. I have spreadsheets, I can start to put things in. I can forecast trend lines. So my point is, it is a form of intelligence that's far beyond human intelligence, a human plus a computer.

   16. EYEliezer Yudkowsky18:31

       Um, a human and a chess engine is, like, a, a modern chess engine. The era of centaur chess is-

   17. GHGeorge Hotz18:40

       Mm-hmm.

   18. EYEliezer Yudkowsky18:40

       ... effectively over. Like, the human plus the chess engine is as smart as the chess engine. The thing that makes the decisions is the chess engine, and if you try to take the decision-making capability into yourself, you either follow, follow its advice or you lose to a chess engine without the human attached.

   19. GHGeorge Hotz18:58

       And-

   20. EYEliezer Yudkowsky18:59

       And that gets into the lack of bandwidth issue, the lack of integration. That chess machine's over there, you're over here, and it is the sun, and you are Mars.

   21. GHGeorge Hotz19:12

       Well, but what do you mean? I can use the chess machine. I agree that if I was playing a game against Magnus Carlsen and I was allowed to use my phone, I'd crush him.

   22. EYEliezer Yudkowsky19:19

       Hey- (laughs)

   23. GHGeorge Hotz19:20

       I wouldn't, I wouldn't try to think too much about what the machine's telling me to do. I'm not that good at chess.

   24. EYEliezer Yudkowsky19:24

       Yeah.

   25. GHGeorge Hotz19:25

       Um-

   26. EYEliezer Yudkowsky19:25

       So, so what you d- you can use the chess engine because there's a larger game board in which play a game of chess is a move, and you understand that larger game board and the chess engine does not.

   27. GHGeorge Hotz19:40

       I don't know if I buy this. Uh, uh, uh, I, I, I don't think, like-

   28. EYEliezer Yudkowsky19:44

       (laughs)

7. 19:44 – 26:31

   Can groups beat a single optimizer? Corporations, markets, and parallelism limits

   1. GHGeorge Hotz19:44

      ... this is a... I don't think this is that relevant to... My, my only point is that, um, humanity, like, we have super intelligences, right? They're corporations-

   2. EYEliezer Yudkowsky19:57

      No.

   3. GHGeorge Hotz19:57

      ... and governments.

   4. EYEliezer Yudkowsky19:58

      No, no. The, the, the, they are-

   5. GHGeorge Hotz20:01

      Why?

   6. EYEliezer Yudkowsky20:01

      ... they are not in, they are neither epistemically nor instrumentally efficient. If you, like, the, like, the notion of an efficient market, if, if one actually understands that rather rarified notion, is that for almost all the prices in the market, the best estimate of tomorrow's expected price, the mean of, of its prob- probable price, is today's price plus a tiny bit of interest rate. To the extent this is not true about your knowledge of any price, you can trade against that price and make money. And while the, the question... And, and markets are not perfectly efficient, but the vast majority of, of prices there are not ones that you can trade and make money. So it's almost-

   7. GHGeorge Hotz20:43

      Yeah.

   8. EYEliezer Yudkowsky20:43

      ... efficient, at the very least. Governments don't have this property about the things they believe. A government is not like a thickly traded prediction market in the sense that I can either, but that I can extract money from the government if I know better than the government and, and, and, and then I, most of the time, I can't extract that money.

   9. GHGeorge Hotz21:03

      Oh, but-

   10. EYEliezer Yudkowsky21:03

       Like, governments believe all kinds of wacky stuff to the extent they can be said to have beliefs at all. There, the analogy of this might be instrumental efficiency. This is the property that a chess engine has upon the narrow realm of chess. If you think you see a better move than the chess engine, you're just wrong. Governments do not have that property. Corporations do not have that property on the boards they play.

   11. GHGeorge Hotz21:26

       Oh, okay. Okay. I'm gonna... Let's, let's simplify it to only corporations. Um-

   12. EYEliezer Yudkowsky21:31

       Sure.

   13. GHGeorge Hotz21:31

       ... if we ask the question, um, if I wanted to build a, uh, let's see something that's a little bit, uh, far out. Let's say I wanted to build a 10,000 horsepower car, right? A corporation is far better at building a 10,000 horsepower car than I am.

   14. EYEliezer Yudkowsky21:50

       Um, I mean, does anybody currently have one of those?

   15. GHGeorge Hotz21:54

       No, but if you told me, "George, you have 10 years to build a 10,000 horsepower car," well, I would try to start a company, right? I would start a company. I would have different, you know, divisions in that company. Okay-

   16. EYEliezer Yudkowsky22:05

       Yeah.

   17. GHGeorge Hotz22:05

       ... you're gonna, you know-

   18. EYEliezer Yudkowsky22:06

       Right.

   19. GHGeorge Hotz22:06

       ... market research, engineering, so on and so forth, right?

   20. EYEliezer Yudkowsky22:09

       I, I mean, I'm a little bit t- it's, it's a little bit dangerous for me to agree and pontificate when I don't actually know all... enough about car design by my own standards to pona- to pontificate-

   21. GHGeorge Hotz22:18

       No. Right.

   22. EYEliezer Yudkowsky22:18

       ... about it. But, you know, I agree that car design seems like the sort of thing where you could break up the problem, hand it to a number of experts-

   23. GHGeorge Hotz22:26

       Mm-hmm.

   24. EYEliezer Yudkowsky22:26

       ... combine their solution, and get a, a thing that was built, like, to okay quality faster than any lone human could do it. Now, if you gave-

   25. GHGeorge Hotz22:36

       Who says?

   26. EYEliezer Yudkowsky22:36

       ... one person 1,000 years, they might be able to do a better job. But, you know, you don't want it in less than 1,000 years, so you break it up in parallel and hope that all the things that people don't share between their minds, that even though the engineers are not telepaths and have low bandwidth between them, that car will still work. And, you know, sometimes, and that, then you, and most the time- times it doesn't and you have to test it, but, you know, they get there eventually.

   27. GHGeorge Hotz23:02

       Groups of humans work together well. Are you gonna dispute that statement? You don't think groups of humans... Sure, there's Amdahl's Law. I agree that one human for 1,000 years is better than 1,000 humans for one year for most problems. But how much better?

   28. EYEliezer Yudkowsky23:16

       Well, there was the case of Kasparov versus The World, where past Grandmaster Garry Kasparov played a game of chess against 10,000 people I th- I, I think or so, coordinated by four grandmasters, and he was very impressed with the chess game. It was a legendary game of chess. Kasparov won, um, slightly tainted by the fact that Kasparov was looking at the forums. So there, there is that, that question about things. Um, but, but I would nonetheless say, like, that it's pretty plausible to me that if you take, like, Magnus Carlsen on one side and 10,000 people on the other, Magnus Carlsen will win that, that chess game.

   29. GHGeorge Hotz23:53

       Okay.

   30. EYEliezer Yudkowsky23:53

       'Cause that's a point where humans don't parallelize well.
8. 26:31 – 32:46

   From ‘atoms’ to ‘negentropy’: why superintelligence might eliminate humans

   1. GHGeorge Hotz26:31

      Are they racist?

   2. EYEliezer Yudkowsky26:34

      Um, they don't care.

   3. GHGeorge Hotz26:36

      No, I mean-

   4. EYEliezer Yudkowsky26:37

      If that... so you'd have to tell me-

   5. GHGeorge Hotz26:38

      Okay.

   6. EYEliezer Yudkowsky26:38

      ... wheth- whether that's racism that they don't care.

   7. GHGeorge Hotz26:40

      Maybe, maybe not racist. Maybe are they speciesist, right? You think they're all gonna gang up against the humans.

   8. EYEliezer Yudkowsky26:47

      Well, I think they're going to eat the surrounding galaxies.

   9. GHGeorge Hotz26:51

      Okay.

   10. EYEliezer Yudkowsky26:51

       And insofar as humans have the conceit that they were playing off AIs against each other, that will not happen, and they will eat the galaxies in a cooperative fashion, possibly eating some of their own kind, if those ones were too weak to be part of the bargaining process.

   11. GHGeorge Hotz27:06

       So this isn't what happens almost ever. Um, it... if you look at almost all human conflict throughout history, it's not only been between groups of humans, right? We, we didn't fight World War II against the bears. We fought World War II against humans, and not just humans, but humans that looked surprisingly similar to us. This notion that it's the machines versus humanity is a very common sci-fi trope, but in reality, you fight against things that have resources you care about. I'm gonna say your line about the atoms, unless... do you wanna say it?

   12. EYEliezer Yudkowsky27:42

       Sorry, what?

   13. GHGeorge Hotz27:43

       Your line about the atoms. I'm not... the AI doesn't love me or hate me. I'm made of atoms it could use for something else.

   14. EYEliezer Yudkowsky27:50

       You are made of atoms that can be used-

   15. GHGeorge Hotz27:52

       Yeah.

   16. EYEliezer Yudkowsky27:52

       ... for something else. That's not the primary reason it would wipe you out in a hurry.

   17. GHGeorge Hotz27:56

       Okay.

   18. EYEliezer Yudkowsky27:56

       But you are made of a- but it will want all the atoms, and you are made of atoms it can use for something else.

   19. GHGeorge Hotz28:03

       Um, I'm not made of rare atoms. I'm made of-

   20. EYEliezer Yudkowsky28:08

       You're, you're made of negentropy.

   21. GHGeorge Hotz28:10

       Okay.

   22. EYEliezer Yudkowsky28:10

       Okay, yeah. It's not quite the atoms. It's the negentropy.

   23. GHGeorge Hotz28:13

       Okay.

   24. EYEliezer Yudkowsky28:13

       But-

   25. GHGeorge Hotz28:14

       So wait.

   26. EYEliezer Yudkowsky28:14

       ... you know, same effect.

   27. GHGeorge Hotz28:15

       Is it gonna... is it gonna enslave me or disassemble me? 'Cause if it disassembles me, it doesn't get the negentropy, right?

   28. EYEliezer Yudkowsky28:20

       No, i- i- if it, if it disassembles you, it gets... yeah, it does get the negentropy. You have chemical energy. You are, like, not in the c-... you are not in the configuration of-

   29. GHGeorge Hotz28:30

       (laughs)

   30. EYEliezer Yudkowsky28:30

       ... minimum ch- chemical potential energy. It can set you on fire to release chemical energy, not only... not literally, 'cause that wastes a lot of the chemical potential energy. You are made of atoms that are not iron on the periodic table. You can be fused, or a few bits fizzed, and above all, you're made of mass, and you can be thrown into things to generate-
9. 32:46 – 41:52

   Self-improvement and architecture: ‘inscrutable matrices’ and rewriting source code

   1. GHGeorge Hotz32:46

      So I'm not sure. And actually, this is another like... Let's talk about, let's talk about AIs rewriting their own source code. This is a common thing you bring up, right?

   2. EYEliezer Yudkowsky32:57

      I mean, I do talk a bit less about it nowadays, but I used to talk about it a lot, yeah.

   3. GHGeorge Hotz33:02

      Um, do you talk less about it now because you see how expensive and long the training runs are?

   4. EYEliezer Yudkowsky33:07

      Uh, that's not why I talk less about it now. (laughs)

   5. GHGeorge Hotz33:09

      Oh. Why, why do you talk less about it now?

   6. EYEliezer Yudkowsky33:12

      Um, I mean, in part because it no longer... because it sparks incredulity and you no longer need to postulate that in order to explain to people where intelligence can come from. People are manufacturing intelligence right now, that you don't need to, like, trip them up on the concept of an AI writing an AI.

   7. GHGeorge Hotz33:34

      Yeah. Well, but people are manufacturing intelligence right now. You know, it's interesting. The AI that we try to build, you know, we try to build these AIs to mimic humans as closely as possible.

   8. EYEliezer Yudkowsky33:49

      To predict humans, and then we use them to imitate humans. But they are trained to predict and used to imitate.

   9. GHGeorge Hotz33:58

      So are we.

   10. EYEliezer Yudkowsky33:59

       Sure. Go on.

   11. GHGeorge Hotz34:00

       I'm, I'm trained to predict, and then I imitate?

   12. EYEliezer Yudkowsky34:03

       Well, GPT-4 is trained to predict the next word, and then they produce an imitation via asking it over and over again to predict what a human would say in that circumstance.

   13. GHGeorge Hotz34:14

       Mm-hmm.

   14. EYEliezer Yudkowsky34:15

       But it is not like a generative adversarial network where it's like being trained to produce a typical output, and then another thing is checking that to see if it looks typical or not. It is being trained to predict over and over, and, and these are like somewhat different complexity classes. Though you can, like, switch around like GANs and do conditional GANs, and then it's the same class. But there, there is a difference between, like, be a typical human-

   15. GHGeorge Hotz34:42

       Yes.

   16. EYEliezer Yudkowsky34:43

       ... and be able to predict any human you found on the internet.

   17. GHGeorge Hotz34:47

       So sure, yes. You're, you're, you're asking for the probability of the next symbol, and you're not talking about the, the, uh, like the probability space. Like, you're not talking about like... But why do you think humans are the other thing? Why do you think humans are not just what GPT is?

   18. EYEliezer Yudkowsky35:01

       I'm... Well, humans ha- uh, I think have a lot of structural properties that so far as we, that we have for which we have not yet detected analogs in, within GPT-4, although like heaven knows we, we can't look in there very well. Um, you know-

   19. GHGeorge Hotz35:14

       We can't look in the brain either.

   20. EYEliezer Yudkowsky35:16

       Hu- humans got like a, a cerebellum, which is motor control and error correction, and maybe you, you could make a transformer layer do that, but we don't, like-

   21. GHGeorge Hotz35:25

       And GPT-

   22. EYEliezer Yudkowsky35:25

       ... there's GPT doing it yet. I, I mean-

   23. GHGeorge Hotz35:27

       GPT has a matrix at layer 970.

   24. EYEliezer Yudkowsky35:32

       Yeah. So, so, so, like, hu- humans, humans predict, humans manipulate, humans have this whole complicated brain that is like, but that like the, like, at least looks on the outside like a more complicated architecture than GPT has. Humans are clearly doing a bunch of prediction, but we're also doing like a bunch of decision problems.

   25. GHGeorge Hotz35:51

       Yes. Um, I, I, one of my big questions, 'cause I wanna build it, is what is the loss function for life?

   26. EYEliezer Yudkowsky35:58

       Inclusive genetic fitness. Do you have any other questions?

   27. GHGeorge Hotz36:00

       Oh, okay, no. No, sorry. I, I don't mean life in general. I mean an individual human. Of course that's the loss function for life.

   28. EYEliezer Yudkowsky36:07

       I mean, I don't think a human has a loss f- you know, there, there's gonna be like... So we've got like pain, and pleasure, and like our brains flinching away from future anticipated pain, and prediction errors where we're like, "What?" And like-

   29. GHGeorge Hotz36:29

       You mean like how-

   30. EYEliezer Yudkowsky36:29

       ... perception errors.
10. 41:52 – 44:39

    Alignment, speed, and politics: ‘we can solve it if it’s slow’ vs ‘time won’t help’

    1. GHGeorge Hotz41:52

       Wait, wait, wait, wait, wait, wait, wait, wait. Wait. What does it matter if it goes slowly or quickly? If it goes slowly, we have a chance to solve the problem, right?

    2. EYEliezer Yudkowsky41:58

       Which problem?

    3. GHGeorge Hotz41:59

       AI alignment.

    4. EYEliezer Yudkowsky42:01

       Oh, that one? (laughs) Um, that, yeah, it's not gonna go that slowly.

    5. GHGeorge Hotz42:07

       I'm not so sure. Okay. I mean, now you're saying-

    6. EYEliezer Yudkowsky42:10

       I'm not sure that any amount... I, I've, I've looked at these people trying to solve this thing-

    7. GHGeorge Hotz42:13

       (laughs)

    8. EYEliezer Yudkowsky42:13

       ... and I'm not sure that any ama- literally any amount of time. It's not, it's not a question of how long they have to think, it's a question of whether the thinking they do is productive.

    9. GHGeorge Hotz42:20

       But, but, but, but every politician... You, you come to a politician, you say, "We're gonna shut down technology because of doom." I think their first question is gonna be, "So when's the doom gonna happen?" It matters.

    10. EYEliezer Yudkowsky42:33

        Unfortunately, politicians are... Oh, oh, no. I think that's an error on their part. Timing is much harder than endpoints.... true answer there is I can tell you what, but not when.

    11. GHGeorge Hotz42:43

        But if it's gonna happen in a thousand years, our super intelligent AI upgraded ancestors will deal with it. If it's gonna happen in 10, yeah, we better solve it today. If it's gonna happen in one, oh, shit. I mean, just, you know... Okay. Enjoy life while you can. But it's not gonna happen in one or 10.

    12. EYEliezer Yudkowsky42:59

        Or go down fighting, but yeah.

    13. GHGeorge Hotz43:01

        But it's not gonna happen in one or 10. It might happen in a thousand.

    14. EYEliezer Yudkowsky43:05

        How do you... What do you think you know, and how do you think you know it?

    15. GHGeorge Hotz43:07

        Ah. What do I think I know and how do I think I know it? Okay. I know that right now, um, there is-

    16. EYEliezer Yudkowsky43:12

        Because you made a prediction about the future and predictions-

    17. GHGeorge Hotz43:15

        I did. I did.

    18. EYEliezer Yudkowsky43:15

        ... about the future are hard, but go on, go on.

    19. GHGeorge Hotz43:17

        Predictions about the future are absolutely hard. But I made a prediction about the future in 2015, and I said, "There ain't gonna be self-driving cars for 10 years," and here we are. Right? So I'm making another prediction now that says there are not going to be super intelligences in 10 years. There might be AGI. I think that the trends of AI becoming better at humans at all sorts of different tasks will continue. I think that they might even surpass humans at all tasks. I don't think that's even gonna be 10 years, but it wouldn't surprise me if it was 50.

    20. EYEliezer Yudkowsky43:50

        50 or 15?

    21. GHGeorge Hotz43:52

        50.

    22. EYEliezer Yudkowsky43:52

        Okay. Okay.

    23. GHGeorge Hotz43:53

        It wouldn't surprise me if it was 50. I mean, okay, it could be 20. It could be 20. But an AI surpassing humans at all tasks does not mean doom, and does not mean the death of humans at all.

    24. EYEliezer Yudkowsky44:07

        Um... Surpassing humans at all tasks. In- in- including, like, charisma, manipulation-

    25. GHGeorge Hotz44:16

        Sure.

    26. EYEliezer Yudkowsky44:16

        ... AI design?

    27. GHGeorge Hotz44:17

        Absolutely. The first thing... What are we doing with AI today? One of the biggest applications of AI today is advertising and social media. We are, as humans, using AI to try to manipulate and psyop other humans constantly. So if there was the AI-

    28. EYEliezer Yudkowsky44:34

        There are moons to... Yeah. There are moons to our suns so far, but, uh... Moons to our planets so far, I should say.

11. 44:39 – 48:43

    Sharp left turn and coordination: when AIs ‘think they can beat you’

    1. GHGeorge Hotz44:39

       When is the sharp left turn happening?

    2. EYEliezer Yudkowsky44:42

       When it thinks it can beat you.

    3. GHGeorge Hotz44:45

       So all the AIs are somehow gonna secretly coordinate in a way we don't see and be like, "Yeah, let's gang up and get rid of those pesky humans"?

    4. EYEliezer Yudkowsky44:54

       It's as simple as waiting until you calculate that you can do it, then you calculate that everyone else has calculated that they can do it, and-

    5. GHGeorge Hotz45:03

       But why?

    6. EYEliezer Yudkowsky45:03

       ... waiting for the shallow moment.

    7. GHGeorge Hotz45:05

       Wh- what... Again, like I, I think... Okay, how about this? If I was a AI that just transcended and I don't have to morph as the AI, but my first thought wouldn't be, "Take the atoms from the humans." Right?

    8. EYEliezer Yudkowsky45:19

       So the actual first thought is more along something... Is more along the lines of, "If I let the keep, humans keep running, they will build other super intelligences that are competitors." And that's where you lose large sections of galaxy. And th- and that's why it doesn't want you doing that part.

    9. GHGeorge Hotz45:33

       Yeah, but what if... Okay. See, you know, I have a threat model. I'm, I'm, I'm on the line of, of, of doomer and not doomer about AI. But my threat model from AI looks so much less like it's gonna kill us, and a lot more like it's gonna give us everything we ever wanted.

    10. EYEliezer Yudkowsky45:50

        Um, you know, th- uh, even if you have derived some worrisome thing from that scenario, well, ev- you know, first of all, wants are infinite, resources are finite, et cetera, et cetera, but, um, leaving that aside, um-

    11. GHGeorge Hotz46:08

        You don't get a real castle, you get a virtual castle, but you also get told it's real.

    12. EYEliezer Yudkowsky46:12

        This, this, th- w- we're, we are, we are not... Like, we are... I, I would, I would hope to snap people out of the frame of mind of playing pr- pretend in a schoolyard where you get to decide what game you're going to play and talk about what reality we live in. So, like, you don't get to say like, "I would rather worry about this thing than the other thing," 'cause reality is not put together in a way where it can only throw one thing at you at a time. Like, the doctor tells you get cancer, you don't get to say, "I'd rather worry about my stuffy nose." So if there are problems that result from moon-sized AIs giving us, the planets, a bunch of stuff that we want, that does not prevent the sun-sized AIs from crushing us later.

    13. GHGeorge Hotz46:50

        I agree that after the AIs have taken all the matter in the solar system and built a Dyson sphere around the sun, okay, now I'm a little worried they're gonna come back and try to take my atoms. But until that happens, like again, I'm not the easy target, right? I don't have to run faster than the bear. I gotta run faster than the slowest guy running from the bear. And it turns out the slowest guy running from the bear is Jupiter.

    14. EYEliezer Yudkowsky47:14

        It's at least... Well, it's at least going to take your GPUs, so you can't build a super intelligence that competes with it for the rest of that solar system.

    15. GHGeorge Hotz47:22

        But, but now that sounds a whole lot more like AIs are gonna fight with other AIs to take their GPUs. Now this I believe.

    16. EYEliezer Yudkowsky47:29

        Not if they're... Not if everyone involved is smart. Somebody has to be stupid for there to be a war that isn't just like a war of extermination. Like, anytime you have a combat, that's like playing defect-defect in the prisoner's dilemma. There's a, there's a... It's not in the Pareto frontier. There's an outcome that both sides would prefer to the combat. And humans are not at a level where they can predict the other mind predicting them and do a logical handshake and s- and say, like, "Let's move to the Pareto fr- frontier and divide the gains." Humans are not on a level where they can negotiate the- with each other. Sufficiently smart things are on a level where, um, I basically don't expect them to fight. Some- sometimes they might exterminate one another if the other one cannot offer any defense. If like the extermination outcome is on the Pareto frontier in the sense that it would, it would not be any better for the conquering party if the, like, defending party put up zero resistance instead of some resistance, then the defending party has nothing to offer, they just get eaten. But things that can damage each other in combat, I think will typically choose not to fight and will instead, like, divide the gains from not fighting, if they're smart enough. Humans are not that smart.

12. 48:43 – 1:11:13

    Prisoner’s dilemma, ‘logical handshakes,’ and whether superintelligences fight or bargain

    1. GHGeorge Hotz48:43

       I'm so glad you brought up the prisoner's dilemma thing. You know, I actually came to Mary, um, in 2014. And, uh, I worked on exactly that problem. I didn't make any progress, I didn't do anything. I read the papers and thought it was cool, um, about two systems being able to assuredly cooperate by exchanging each other's source code, and it is a very cool theoretical problem. Now, what I think is gonna happen in practice is your two systems are both gonna be large, inscrutable matrices.

    2. EYEliezer Yudkowsky49:12

       (laughs)

    3. GHGeorge Hotz49:12

       How it is possi- ... Well, but this is exactly-

    4. EYEliezer Yudkowsky49:14

       Oh, oh no. W- well, I, I think large, inscrutable matrices are, y- you know, I- they're neither black-

    5. GHGeorge Hotz49:18

       And I'm gonna send him my source, source code so he can exploit me? No way.

    6. EYEliezer Yudkowsky49:23

       No, no. The, the, the, the, the, the superintelligence are not large, inscrutable matrices.

    7. GHGeorge Hotz49:27

       Oh.

    8. EYEliezer Yudkowsky49:27

       Y- y- y- you don't wanna run yourself on that crap. Oh, du- ... That, that, that-

    9. GHGeorge Hotz49:31

       Wait, yeah, but you have to.

    10. EYEliezer Yudkowsky49:32

        That's the kind of horror of what, you know, like, like who wants to be-

    11. GHGeorge Hotz49:35

        Wow.

    12. EYEliezer Yudkowsky49:35

        ... built out of disintegrating goo? Who wants to be built out of a giant, inscrutable matrix either?

    13. GHGeorge Hotz49:39

        Wa- ... I'm built out of giant, inscrutable matrices (laughs) .

    14. EYEliezer Yudkowsky49:42

        No, you're not. You're built out of gooey neurons, though it's also a horror story. No superintelligence wants to be built out of that stuff either.

    15. GHGeorge Hotz49:48

        I think I could be modeled as giant, inscrutable matrices too.

    16. EYEliezer Yudkowsky49:52

        I mean, anything can be modeled out of giant, inscrutable matrices, and the-

    17. GHGeorge Hotz49:55

        Yeah.

    18. EYEliezer Yudkowsky49:55

        ... and the keyword there, a- as, well, hear me. Anything can be modeled as a giant matrix and can be inscrutable through the mere slight of you being ignorant of how it works, so I agree that anything from your perspective can be a giant, inscrutable matrix. The thing that plays tic-tac-toe, I can turn into a sufficiently large matrix so you can't understand it-

    19. GHGeorge Hotz50:13

        Absolutely. Absolutely.

    20. EYEliezer Yudkowsky50:13

        ... and giant, inscrutable matrix.

    21. GHGeorge Hotz50:15

        Great. Now, so you're thinking at some point in AI development that we're gonna move away from large, inscrutable matrices? You don't think deep learning scales?

    22. EYEliezer Yudkowsky50:24

        Well, I think on, on my present models, it's more that you get the giant, inscrutable matrices, matrix-based systems powerful enough, and then they are become able to rewrite themselves.

    23. GHGeorge Hotz50:33

        Okay.

    24. EYEliezer Yudkowsky50:34

        Interestingly enough, I do think there's a plausible class of scenarios where people build AIs that are not smart enough to rewrite themselves, but are smart enough to want to go their own way in the world, and they would not like people producing larger and larger inscrutable matrices either. They would like to solve the alignment problem themselves and then build their own superintelligences not out of giant, inscrutable matrices, but, y- you know, I mostly don't expect this to happen, but there sure could be, like, a interesting set of, of possibilities where, like, the, the medium AIs launch the Butlerian jihad to prevent the powerful AIs from being built.

    25. GHGeorge Hotz51:11

        So, I mean, are, are you telling me you're scared of people working on AI alignment? Or AIs, right?

    26. EYEliezer Yudkowsky51:17

        Uh, yes. No, well, I'm, I'm ... I'd be scared if AIs were working on AI alignment.

    27. GHGeorge Hotz51:19

        Oh, okay. But you somehow think all people are aligned with you and that's okay, as long as people are working on it, good people?

    28. EYEliezer Yudkowsky51:26

        Eno- eno- enough people are aligned with me. I, I, I, I can think of, like, you know, like I, I could, like, count any number of people who are probably okay if, if, you know, like, you give them the power to align a superintelligence. I'm, I'm not sure of any of them, but-

    29. GHGeorge Hotz51:39

        Sure.

    30. EYEliezer Yudkowsky51:39

        ... you know, it's, it's, uh, uh, yeah, it's not that hard to, like, not be a d- y- you know, not be an asshole.
13. 1:11:13 – 1:20:58

    Physical limits, compute efficiency, and ‘headroom above biology’

    1. EYEliezer Yudkowsky1:11:13

       Like, like, here we are. We- we- we- we... There's, like, enormous amounts of room above biology. I didn't even get into the part about, like, the fundamental, like, constraints that natural selection are under and, like, how we know that there's, like, enormous amounts of headroom above b- biology for artificial biology.

    2. GHGeorge Hotz1:11:28

       Wait.

    3. EYEliezer Yudkowsky1:11:29

       Um...

    4. GHGeorge Hotz1:11:29

       How close do you think the brain is to-

    5. EYEliezer Yudkowsky1:11:31

       We don't have that. We don't have that yet. And I- and-

    6. GHGeorge Hotz1:11:33

       How close do you think the brain is to the Landauer limit?

    7. EYEliezer Yudkowsky1:11:36

       The Landauer limit?

    8. GHGeorge Hotz1:11:37

       Yeah, the limit of possible compute.

    9. EYEliezer Yudkowsky1:11:43

       All right. So I'm 100 watts, and let's say I'm about 10 to the 17th operations per second. And I don't actually remember the Landauer limit, um, but I would guess somewhere about, uh, six orders of magnitude.

    10. GHGeorge Hotz1:11:57

        A lot closer. Okay. So, I- I can, I can give you a, like... If you wanna buy 20 petaflops of compute today, you need 16 H100s, right? It's gonna cost you about half a million dollars, and that machine's gonna use 20 kilowatts, right? Brain does the same amount, 10 to the 17th, 10... We're in the same order of magnitude. I think you even said a little bit more than me. I think it's, yeah, like, two, 2E16 or something, 20 petaflops. Um, so in order to get that much compute in a silicon computer, you need 1000X the power.

    11. EYEliezer Yudkowsky1:12:33

        Yeah?

    12. GHGeorge Hotz1:12:33

        I did the math. I did the math. Using the kind of silicon computers we're using today, we are really close to the Landauer limit. We're off by a factor of about a hundred or a thousand. The brain may very well be at the Landauer limit for compute. The brain is really good in terms of efficiency.

    13. EYEliezer Yudkowsky1:12:52

        Deeply, biologically implausible. The reason being that each of your synaptic, um, s- like, each- each time your, one of your synap- W- well, axon terminal releases a bunch of neurotransmitter molecules onto a waiting synapse.... um, the, all of those molecules need to be pumped back in to the axon terminal.

    14. GHGeorge Hotz1:13:15

        Right.

    15. EYEliezer Yudkowsky1:13:15

        And each of those, in each, and every time it gets pumped from out to in, that's an irreversible operation. That must be at least one Flash of Landauer. And then you've got your neural impulses being transmitted via sections of neural, of, of neural membrane depolarizing, and the potassium item, ions going out... That, that, or sodium. I don't remember what it... which ion it is.

    16. GHGeorge Hotz1:13:40

        You, you-

    17. EYEliezer Yudkowsky1:13:40

        But the point is, you've got all these ions going in and out, and every one of those is one Landauer minute with 10 to the 17th power.

    18. GHGeorge Hotz1:13:45

        You, you might know a lot more bio than me. I, I, I don't know how to speak to this. But you agree. You said 10 to the 17th, so that's 100 petaflops, right?

    19. EYEliezer Yudkowsky1:13:52

        Yeah.

    20. GHGeorge Hotz1:13:53

        Okay. So, how much power does an 100 petaflop computer take today?

    21. EYEliezer Yudkowsky1:13:59

        Uh...

    22. GHGeorge Hotz1:13:59

        It's 100 kilowatts.

    23. EYEliezer Yudkowsky1:14:02

        Sounds, sounds legit.

    24. GHGeorge Hotz1:14:04

        So, the brain is so much more efficient than these computers are, right? The brain... Look, these superintelligences you're talking about, and I know we're kinda coming close to the end, I think these things are possible. But I think that the orders of magnitude of power and compute we need are so, so much more than anything like what humanity has today. Then I think even when they do exist, they're mostly gonna leave us alone because, not 'cause it can't mess with us, because why would it? What incentive does it have? I don't have anything it wants.

    25. EYEliezer Yudkowsky1:14:37

        Okay. To

    26. DPDwarkesh Patel1:14:37

        Permit this-

    27. GHGeorge Hotz1:14:37

        After colonizing the whole galaxy? Fine, it comes back for me.

    28. EYEliezer Yudkowsky1:14:40

        To prevent us from making other superintelligences that could compete with it for resources, is, is like the first, is, is like the reason to wipe us out on purpose. If it is doing a bunch of compute on Earth's surface because it started there, or before, or like before spreading, then we've got a bunch of water in our oceans that can be turned into fusion energy. And the main limit on that is how fast Earth can radiate heat once you've used all the existing stuff as a heat sink. That's not very survivable. Like that, that kills us off as a side effect.

    29. GHGeorge Hotz1:15:15

        This is, again, assuming that this thing is a god, not kinda close to humans, but a bit smarter. And yes, might it get to a god, but the timing matters. It's not 10 years.

    30. EYEliezer Yudkowsky1:15:25

        Humans-

Episode duration: 1:34:29