Jim Keller: Moore's Law, Microprocessors, and First Principles | Lex Fridman Podcast #70

1. 0:00 – 2:12

   Introduction

   1. LFLex Fridman0:00

      The following is a conversation with Jim Keller, legendary microprocessor engineer who has worked at AMD, Apple, Tesla, and now Intel. He's known for his work on AMD K7, K8, K12, and Zen microarchitectures, Apple A4 and A5 processors, and co-author of the specification for the x86-64 instruction set and HyperTransport Interconnect. He's a brilliant first principles engineer and out of the box thinker, and just an interesting and fun human being to talk to. This is the Artificial Intelligence podcast. If you enjoy it, subscribe on YouTube, give it five stars on Apple Podcast, follow on Spotify, support it on Patreon, or simply connect with me on Twitter @lexfridman, spelled F-R-I-D-M-A-N. I recently started doing ads at the end of the introduction. I'll do one or two minutes after introducing the episode, and never any ads in the middle that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. I personally use Cash App to send money to friends, but you can also use it to buy, sell, and deposit Bitcoin in just seconds. Cash App also has a new investing feature. You can buy fractions of a stock, say $1 worth, no matter what the stock price is. Broker services are provided by Cash App Investing, a subsidiary of Square and member SIPC. I'm excited to be working with Cash App to support one of my favorite organizations called FIRST, best known for their FIRST robotics and Lego competitions. They educate and inspire hundreds of thousands of students in over 110 countries and have a perfect rating at Charity Navigator, which means the donated money is used to maximum effectiveness. When you get Cash App from the App Store or Google Play and use code LEXPODCAST, you'll get $10 and Cash App will also donate $10 to FIRST, which again is an organization that I've personally seen inspire girls and boys to dream of engineering a better world. And now, here's my conversation with Jim Keller.

2. 2:12 – 3:43

   Difference between a computer and a human brain

   1. LFLex Fridman2:12

      What are the differences and similarities between the human brain and a computer with a microprocessor at its core? Let's start with a philosophical question perhaps.

   2. JKJim Keller2:22

      Well, since people don't actually understand how human brains work... You think that's true?

   3. LFLex Fridman2:29

      I think that's true.

   4. JKJim Keller2:30

      Um, so it's hard to compare them. Computers are, you know, there's really two things. There's memory and there's computation, right? And to date, almost all computer architectures are global memory, which is a thing, right? And then computational where you pull data in and you do relatively simple operations on it and write data back.

   5. LFLex Fridman2:54

      So it's decoupled in moder- in modern computers.

   6. JKJim Keller2:57

      Right.

   7. LFLex Fridman2:57

      And you- you- you think in the human brain, everything's a mesh- a mess that's combined together?

   8. JKJim Keller3:02

      Well, what people observe is there's, you know, some number of layers of neurons which have local and global connections, and information is stored in some distributed fashion, and people build things called neural networks in computers where the information is distributed in some kind of fashion. You know, there's a mathematics behind it. Um, I don't know that the understanding of that is super deep. Uh, the computations we run on those are straightforward computations. I don't believe anybody has said a neuron does this computation. So, to date, it's hard to compare them, I

3. 3:43 – 17:53

   Computer abstraction layers and parallelism

   1. JKJim Keller3:43

      would say.

   2. LFLex Fridman3:44

      So let's get into the basics before we zoom back out. How do you build a computer from scratch? What is a microprocessor? What is a microarchitecture? What's an instruction set architecture? Maybe even as far back as what is a transistor?

   3. JKJim Keller4:01

      So, the special charm of computer engineering is there's a relatively good understanding of abstraction layers. So down at the bottom, you have atoms, and atoms get put together in materials like silicon or dope silicon or metal, and we build transistors. On top of that, we build logic gates, right? And then functional units, like an adder, a subtractor, or an instruction parsing unit, and then we assemble those into, you know, processing elements. Modern computers are built out of, you know, probably 10 to 20 locally, you know, organic processing elements or coherent processing elements, and then that runs co- computer programs, right? So there's abstraction layers, and then software, you know, there's an instruction set you run, and then there's assembly language, C, C++, Java, JavaScript, you know. There's abstraction layers, you know, e- essentially from the atom to the data center, right? So when you, when you build a computer, you know, first there's a target, like what's it for? Like how fast does it have to be? Which, you know, today there's a whole bunch of metrics about what that is. And then in an organization of, you know, a thousand people who build a computer, there's lots of different disciplines that you have to operate on. Does that make sense? And so-

   4. LFLex Fridman5:26

      So the, so there's a bunch of levels of abstraction of, in, in organizational, I can tell, and in your own vision, there's a lot of brilliance that comes in at every one of those layers. Some of it is science, some of it is engineering, some of it is art. What's the most, uh, if you could pick favorites, what's the most important, your favorite layer, um, o- on these layers of abstractions? Where does the magic enter this hierarchy?

   5. JKJim Keller5:54

      Uh, I don't really care.

   6. LFLex Fridman5:57

      (laughs)

   7. JKJim Keller5:57

      That's the fun, you know, I'm somewhat agnostic to that. So I would say for relatively long periods of time, instruction sets are stable.... so the x86 instruction set, the ARM instruction set.

   8. LFLex Fridman6:12

      What's an instruction set?

   9. JKJim Keller6:13

      So it says, how do you encode the basic operations, load, store, multiply, add, subtract, conditional branch, you know. Th- there, there aren't that many interesting instructions. Like if you look at a program when it runs, you know, 90% of the execution is on 25 op codes, you know, 25 instructions, and those are stable, right?

   10. LFLex Fridman6:34

       What does it mean, stable?

   11. JKJim Keller6:35

       Intel architecture has been around for 25 years.

   12. LFLex Fridman6:38

       It works.

   13. JKJim Keller6:38

       It works. A- and that's because the basics, you know, were defined a long time ago, right? Now, the way um, an old computer ran is you fetched instructions and you executed them in order. Do the load, do the add, do the compare. The way a modern computer works is you fetch large numbers of instructions, say 500, and then you find the dependency graph between the instructions, and then you e- you execute in independent units those little micrographs. So a modern computer, like people like to say, "Computers should be simple and clean." But it turns out the market for a simple, complete, clean, slow computers is zero, right? We don't sell any simple, clean computers. Now you can... there's... how you build it can be clean, but the computer people want to buy, that's say in a phone or a data center, fetches a large number of instructions, computes the dependency graph, and then executes it in a way that gets the right answers.

   14. LFLex Fridman7:49

       And optimizes that graph somehow-

   15. JKJim Keller7:50

       Yeah.

   16. LFLex Fridman7:51

       ... so it executes it faster.

   17. JKJim Keller7:51

       S- they run deeply out of order and then there's semantics around how memory ordering works and other things work. So the, the computer sort of has a bunch of bookkeeping tables. It says, "What order should these operations finish in or appear to finish in?" But to go fast, you have to fetch a lot of instructions and find all the parallelism. Now there's a second kind of computer which we call GPUs today, and, and I, I call it the difference... There's found parallelism, like you have a program with a lot of dependent instructions, you fetch a bunch and then you go figure out the dependency graph and you issue instructions out of order. That's because you have one serial narrative to execute, which in fact is in... can be done out of order.

   18. LFLex Fridman8:35

       Did you call it a narrative?

   19. JKJim Keller8:37

       Yeah.

   20. LFLex Fridman8:37

       Wow.

   21. JKJim Keller8:38

       So yeah, so humans think in serial narrative.

   22. LFLex Fridman8:40

       Yeah.

   23. JKJim Keller8:40

       So read, read a book, right? There's a, you know, there's a sentence after sentence after sentence and there's paragraphs. Now you could diagram that. Imagine you diagrammed it properly and you said, "Which sentences could be read in any order, any order without changing the meaning?" Right? Like-

   24. LFLex Fridman9:00

       (laughs) That's a fascinating question to ask-

   25. JKJim Keller9:01

       R- right?

   26. LFLex Fridman9:01

       ... of a book. Yeah.

   27. JKJim Keller9:02

       Yeah, you could do that.

   28. LFLex Fridman9:04

       In theory, yeah.

   29. JKJim Keller9:04

       Right? So some paragraphs could be reordered, some sentences can be reordered. You could say, "He is tall and smart and X," right? And it could... it doesn't matter the order of tall and smart. But if you say, "The tall man is wearing a red shirt," what colors, you know... like you c- you can create dependencies, right?

   30. LFLex Fridman9:27

       Mm-hmm.
4. 17:53 – 20:43

   If you run a program multiple times, do you always get the same answer?

   1. JKJim Keller17:54

   2. LFLex Fridman17:54

      At the end of the day, if you run the same program multiple times, does it always produce the same result? Is, is, is there some room for fuzziness there?

   3. JKJim Keller18:04

      That's a math problem. So if you run a correct C program, the definition is every time you run it, you get the same answer.

   4. LFLex Fridman18:12

      Yeah. Th- well, that's a math statement, but, uh-

   5. JKJim Keller18:14

      Well, that's a, that's a language definitional statement. So-

   6. LFLex Fridman18:18

      Yes, for language.

   7. JKJim Keller18:18

      ... for years, when people did, when we first did 3D acceleration of graphics, you could run the same scene multiple times and get different answers.

   8. LFLex Fridman18:28

      Right.

   9. JKJim Keller18:29

      ... right? And then some people thought that was okay and some people thought it was a bad idea. And then when the HPC world used GPUs for calculations, they thought it was a really bad idea, okay? Now, in modern AI stuff, people are looking at networks where the precision of the data is low enough that the data is somewhat noisy.

   10. LFLex Fridman18:53

       Hmm.

   11. JKJim Keller18:53

       And the observation is the input data is unbelievably noisy, so why should the calculation be not noisy? And people have experimented with algorithms that say you can get faster answers by being noisy. Like as the network starts to converge, if you look at the computation graph, it starts out really wide and then it gets narrower, and you could say, "Is that last little bit that important or should I start the graph on the next wrap- rev before we whittle it all the way down to the answer?" Right? So you can create algorithms that are noisy.

   12. LFLex Fridman19:23

       Mm-hmm.

   13. JKJim Keller19:24

       Now if you're developing something and every time you run it, you get a different answer, it's really annoying.

   14. LFLex Fridman19:29

       (laughs)

   15. JKJim Keller19:29

       And so most people think, even today, every time you run a program, you get the same answer.

   16. LFLex Fridman19:36

       No, I know, but the- the question is, that's the formal definition of a programming language.

   17. JKJim Keller19:42

       There is a definition of languages that don't get the same answer, but people who use those... Y- you always want something, 'cause you get a bad answer and then you're wondering, "Is it because-"

   18. LFLex Fridman19:52

       Right.

   19. JKJim Keller19:53

       ... "of something in the algorithm or because of this?" And so everybody wants a little switch that says, "No matter what-"

   20. LFLex Fridman19:58

       Yeah.

   21. JKJim Keller19:58

       ... "do it deterministically." And it's really weird, 'cause almost everything going into modern calculations is noisy. So why-

   22. LFLex Fridman20:05

       Right.

   23. JKJim Keller20:05

       ... the answers have to be so clear? It's-

   24. LFLex Fridman20:08

       All right, so where do you stand?

   25. JKJim Keller20:09

       I design computers for people who run programs.

   26. LFLex Fridman20:12

       So you're agnostic.

   27. JKJim Keller20:12

       So if somebody says, "I want a deterministic answer," like most people want that.

   28. LFLex Fridman20:18

       Can you deliver a deterministic answer, I guess is the question? Like when you-

   29. JKJim Keller20:22

       Yeah, hopefully, sure.

   30. LFLex Fridman20:24

       That- that's
5. 20:43 – 22:41

   Building computers and teams of people

   1. LFLex Fridman20:43

      You've achieved, in the eyes of, uh, many people, a legend status as a chip art-

   2. JKJim Keller20:51

      Mm-hmm.

   3. LFLex Fridman20:51

      ... architect. Uh, what design creation are you most proud of, perhaps because it was challenging, because of its impact, or because of the set of brilliant ideas that, um, that were involved in bringing it to life?

   4. JKJim Keller21:06

      Well, I- I find that description odd. And I have two small children-

   5. LFLex Fridman21:11

      Thank you.

   6. JKJim Keller21:11

      ... and I promise you, uh, (laughs) they think it's hilarious.

   7. LFLex Fridman21:16

      This question.

   8. JKJim Keller21:16

      Yeah. So, uh-

   9. LFLex Fridman21:17

      I do it for them.

   10. JKJim Keller21:18

       So I- I am, uh... I'm really interested in building computers. And I've worked with really, really smart people. I'm not unbelievably smart. I'm fascinated by how they go together, both as a- as a thing to do and as a endeavor that people do.

   11. LFLex Fridman21:38

       How people and computers go together?

   12. JKJim Keller21:40

       Yeah. Like, how people think and build a computer. And I find sometimes that the best computer architects aren't that interested in people, or the best people managers aren't that good at designing computers. So-

   13. LFLex Fridman21:54

       So the whole stack of human beings is fascinating. So the managers, the individual engineers-

   14. JKJim Keller21:58

       Yeah, yeah. So, yeah, I de- I said- I realized after a lot of years of building computers, where you sort of build them out of transistors, logic gates, functional units, com-

   15. LFLex Fridman22:06

       Yeah.

   16. JKJim Keller22:06

       ... computational elements, that you could think of people the same way. So people are functional units.

   17. LFLex Fridman22:12

       Yes.

   18. JKJim Keller22:12

       And then you could think of organizational design as a computer architectural problem. And then it was like, oh, that's super cool, 'cause the people are all different, just like the computational elements are all different, and they like to do different things, and... And so I had a- a lot of fun like reframing how I think about organizations.

   19. LFLex Fridman22:31

       Just like with, uh, with computers, w- we were saying execution paths, you can have a lot of different paths that end up at a- at a- at a- at the same good destination.

6. 22:41 – 30:05

   Start from scratch every 5 years

   1. LFLex Fridman22:41

      So what have you learned about the human abstractions, from individual functional human units to the- the broader organization? What- what does it take to create something special?

   2. JKJim Keller22:55

      Well, most people don't think simple enough, all right? So do you know the difference between a recipe and the understanding? I- y- there's probably a philosophical description of this. So imagine you're gonna make a loaf of bread.

   3. LFLex Fridman23:11

      Yup.

   4. JKJim Keller23:11

      The recipe says get some flour, add some water, add some yeast, mix it up, let it rise, put it in a pan, put it in the oven. It's a recipe, right? Understanding bread, you can understand biology, supply chains, f- you know, f- grain grinders, yeast, physics, you know, thermodynamics. Like, there's so many levels of understanding there. And then when people build and design things, they frequently are executing some stack of recipes, right? And the problem with that is the recipes all have limited scope. Look, if you have a really good recipe book for making bread, it won't tell you anything about how to make an omelet.

   5. LFLex Fridman23:54

      Right.

   6. JKJim Keller23:54

      Right? But if you have a deep understanding of cooking, right? Then bread, omelets, you know, sandwich, you know, there's- there's a different, you know, way of viewing everything. And most people, when you get to be an expert at something, you know, you're- you're hoping to achieve deeper understanding, not just a large set of recipes to go execute. And it's interesting to watch groups of people, because executing recipes is unbelievably efficient, if it's what you want to do. If it's not what you wanna do, you're really stuck.... and, and that difference is crucial. And ev- and everybody has a balance of, let's say, deeper understanding and recipes, and some people are really good at recognizing when the problem is to understand something deeply, deeply. Does that make sense?

   7. LFLex Fridman24:49

      That totally makes sense. Uh, does at every stage of development deep un- understanding on the team needed?

   8. JKJim Keller24:55

      Oh, this goes back to the art versus science question.

   9. LFLex Fridman24:58

      Sure.

   10. JKJim Keller24:59

       If you constantly unpacked everything for deeper understanding, you'd never get anything done.

   11. LFLex Fridman25:03

       Right.

   12. JKJim Keller25:04

       And if you don't unpack understanding when you need to, you'll do the wrong thing, and then at every juncture... Like human beings are these really weird things because everything you tell 'em has a million possible outputs-

   13. LFLex Fridman25:17

       Right.

   14. JKJim Keller25:18

       ... right? And then they all interact in a hilarious way.

   15. LFLex Fridman25:21

       Yeah, it's very

7. 30:05 – 55:47

   Moore's law is not dead

   1. JKJim Keller30:41

      ... operational model is we increase the performance of computers by 2X every two or three years, and it's wiggled around substantially over time, and also, in how we deliver performance has changed.

   2. LFLex Fridman30:57

      So the... Right, so you mentioned-

   3. JKJim Keller30:58

      But, but the, the, the, the foundational-

   4. LFLex Fridman30:59

      ... what, what performance means.

   5. JKJim Keller30:59

      ... idea was 2X the transistors every two years. The current cadence is something like, they call it a shrink factor, like 0.6 every two years, which is not 0.5.

   6. LFLex Fridman31:11

      But that, that's referring strictly, again, to the original definition of just-

   7. JKJim Keller31:15

      Yeah, of transistor count.

   8. LFLex Fridman31:16

      And shrink factor's just getting them smaller and smaller and smaller.

   9. JKJim Keller31:19

      Well as you, use for a constant chip area-

   10. LFLex Fridman31:21

       Right.

   11. JKJim Keller31:21

       ... if you make the transistors smaller by 0.6, then you get one over 0.6 more transistors.

   12. LFLex Fridman31:27

       So can you linger on it a little longer? What's the, what's a broader... What do you think should be the broader definition of Moore's law? When you mentioned perfor- how you think of performance, just broadly, what's a good way to think about Moore's law?

   13. JKJim Keller31:42

       Well, first of all... So I, I've, I've been aware of Moore's law for 30 years.

   14. LFLex Fridman31:48

       In which sense?

   15. JKJim Keller31:49

       Well, when I arrived, I've been designing computers for 40.

   16. LFLex Fridman31:52

       You're just watching it before your eyes kind of thing.

   17. JKJim Keller31:55

       Well... And (clears throat) somewhere where I became aware of it, I was also informed that Moore's law was gonna die in 10 to 15 years. And I thought that was true at first, but then after 10 years, it was gonna die in 10 to 15 years, and then at one point it was gonna die in five years, and then it went back up to 10 years, and at some point I decided not to worry about that particular prognostication for the rest of my life. Which is, which is fun, and then I joined Intel and everybody said Moore's law is dead.

   18. LFLex Fridman32:22

       Right.

   19. JKJim Keller32:22

       And I thought, "That's sad 'cause it's the Moore's law company," and it's not dead, and it's always been gonna die. And, you know, humans like these apocryphal kind of statements like, "We'll run out of food," or, "We'll run out of air," or, "We'll run out of room," or, "We'll run out of," you know, something.

   20. LFLex Fridman32:40

       Right, but it's still incredible that it's lived for as long as it ha- ha- has. And yes, there's many people who believe now that Moore's law (laughs) is, is dead.

   21. JKJim Keller32:50

       I know.

   22. LFLex Fridman32:51

       (laughs)

   23. JKJim Keller32:51

       And they can join the last 50 years of people who had the same idea.

   24. LFLex Fridman32:53

       Yeah, there's a long tradition.

   25. JKJim Keller32:55

       Yeah.

   26. LFLex Fridman32:55

       But, uh, w- why do you think... if you can intex- uh, try to understand it, why do you think it's not dead-

   27. JKJim Keller33:03

       Well, first-

   28. LFLex Fridman33:03

       ... currently?

   29. JKJim Keller33:03

       ... let... Just think, um... People think Moore's law is one thing, transistors get smaller, but actually under the sheets there's literally thousands of innovations and almost all those innovations have their own diminishing return curves.

   30. LFLex Fridman33:16

       Yeah.
8. 55:47 – 1:00:02

   Is superintelligence the next layer of abstraction?

   1. JKJim Keller55:47

   2. LFLex Fridman55:47

      There's something reminiscent of that step from the, the basic operations of addition to taking a step towards neural networks that's reminiscent of what life on earth, at its origins, was doing. Do you think we're creating sort of the next step in our evolution in, in creating artificial intelligence systems that will-

   3. JKJim Keller56:08

      I, I don't know. I mean-

   4. LFLex Fridman56:09

      You don't-

   5. JKJim Keller56:09

      ... there's so much in the universe already, it's hard to say.

   6. LFLex Fridman56:12

      Where we stand-

   7. JKJim Keller56:13

      Like-

   8. LFLex Fridman56:13

      ... in this whole thing.

   9. JKJim Keller56:14

      ... are human beings working on additional abstraction layers and possibilities? Yeah, it appears so. Does that mean that human beings don't need dogs? You know, no. Like, like there's so many things that are all simultaneously interesting and useful.

   10. LFLex Fridman56:30

       But you've seen... Throughout your degree, you've seen greater and greater level abstractions built in artificial machines, right? Do you think... When you look at humans, do you think that th- the, uh, look of all life on Earth as a single organism building this thing, this machine with greater and greater levels of abstraction, do you think humans are the peak, the top of the food chain in this long arc of history on Earth or do you think we're just somewhere in the middle? Are we, are we the basic f- functional operations of a CPU? Are we the C++ program, the Python program-

   11. JKJim Keller57:09

       Well, like-

   12. LFLex Fridman57:09

       ... or we're the neural network or-

   13. JKJim Keller57:10

       ... like somebody's, you know, people have calculated like how many operations does the brain do and something-

   14. LFLex Fridman57:15

       Right.

   15. JKJim Keller57:15

       ... you know, I've seen the number 10 to the 18th of a bunch of times, arrived different ways. So could you make a computer that did 10 to the 20th operations?

   16. LFLex Fridman57:23

       Yes.

   17. JKJim Keller57:24

       Sure.

   18. LFLex Fridman57:24

       Right. So you think-

   19. JKJim Keller57:25

       We're gonna do that. Now, is there something magical about how brains compute things? I don't know. You know, my personal experiences is interesting 'cause, you know, you think you know how you think and then you have all these ideas and you can't figure out how they happened, and if you meditate, you know, the, like what, what you can be aware of is interesting. So I don't know if brains are magical or not. You know, the physical evidence says no, lots of people's personal experience says yes. So what would be f- funny is if brains are magical and yet we can make brains with more computation. You know, I don't know what to say about that, but...

   20. LFLex Fridman58:07

       Well, do you think, uh, magic is an emergent phenomena? What, what, uh-

   21. JKJim Keller58:11

       Could, could be. I have-

   22. LFLex Fridman58:11

       Let me a-

   23. JKJim Keller58:12

       ... I have no explanation-

   24. LFLex Fridman58:13

       (laughs) Yes.

   25. JKJim Keller58:13

       ... for it. I'm a-

   26. LFLex Fridman58:13

       Let me ask-

   27. JKJim Keller58:14

       ... I'm an engineer.

   28. LFLex Fridman58:14

       ... Jim Keller of what, what, what, what in your view is consciousness?

   29. JKJim Keller58:19

       What's, what's consciousness?

   30. LFLex Fridman58:20

       Yeah, like what, uh, you know, c- consciousness, uh, love, things that are these deeply human things that seems to emerge from our brain, is that something that we'll be able to make, encode in chips-
9. 1:00:02 – 1:03:00

   Is the universe a computer?

   1. JKJim Keller1:00:02

      so. Like the-

   2. LFLex Fridman1:00:03

      Do you think the universe is a computer? Like, do you think-

   3. JKJim Keller1:00:06

      Uh, well, it seems to be... It's a weird kind of computer because if it was a computer, right? Like when they do calculations on what it... How much calculation it takes to describe quantum effects is unbelievably high.

   4. LFLex Fridman1:00:20

      Right.

   5. JKJim Keller1:00:21

      So if it was a computer, wouldn't you have built it out of something that was easier to compute? Right? That's a, that's a funny... It's a funny system, but then the simulation guys have pointed out that the rules are kind of interesting, like when you look really close, it's uncertain and the speed of light says you can only look so far and things can't be simultaneous except for the odd entanglement problem where they seem to be. Like the rules are all kind of weird.

   6. LFLex Fridman1:00:44

      Yeah.

   7. JKJim Keller1:00:45

      And somebody said physics is like having 50 equations with 50 variables to define 50 variables. Like, you know, (laughs) it's, it's, you know, like physics itself has been a shit show for thousands of years. It seems odd when you get to the corners of everything and, you know-

   8. LFLex Fridman1:01:02

      Yeah.

   9. JKJim Keller1:01:02

      ... it's either uncomputable or u definable or uncertain.

   10. LFLex Fridman1:01:07

       It's almost like the designers of the simulation are trying to prevent us from understanding it-

   11. JKJim Keller1:01:11

       Yeah.

   12. LFLex Fridman1:01:11

       ... perfectly.

   13. JKJim Keller1:01:12

       But, but also the, the things that require calculations require so much calculation that our idea of the universe of a computer is absurd because every single little bit of it takes all the computation in the universe to figure out.

   14. LFLex Fridman1:01:26

       So, so do you-

   15. JKJim Keller1:01:26

       So that's a weird kind of computer. You know, you say simulations running in the computer-

   16. LFLex Fridman1:01:30

       Yeah.

   17. JKJim Keller1:01:30

       ... which has by definition infinite computation.

   18. LFLex Fridman1:01:34

       Not infinite. Oh, you mean if the universe is infinite?

   19. NANarrator1:01:37

       Oh.

   20. JKJim Keller1:01:37

       Well, yeah. Well, well-

   21. LFLex Fridman1:01:38

       It's just very large.

   22. JKJim Keller1:01:39

       Every little piece of our universe seems to take infinite computation to figure out.

   23. LFLex Fridman1:01:43

       Not infinite. Just a lot.

   24. JKJim Keller1:01:44

       Well, a lot's a pretty big number. Compute this little teeny spot takes all the ma- mass in the local one light year by one light year space, it's-

   25. LFLex Fridman1:01:53

       Yeah.

   26. JKJim Keller1:01:53

       ... close enough to infinite, so...

   27. LFLex Fridman1:01:55

       Well, that's a heck of a computer if it is one.

   28. JKJim Keller1:01:56

       I know, it's, it's, it's a weird, it's a weird description 'cause the simulation description seems to, to break when you look closely at it. But the rules of the universe seem to imply something's up.

   29. LFLex Fridman1:02:08

       (laughs)

   30. JKJim Keller1:02:08

       That seems a little arbitrary.
10. 1:03:00 – 1:04:33

    Ray Kurzweil and exponential improvement in technology

    1. LFLex Fridman1:03:00

       So what are your thoughts on Ray Kurzweil's sense that exponential improvement in technology will continue indefinitely? That, is that how you see Moore's Law? Do you see Moore's Law more broadly in the sense that technology of all kinds has a way of stacking S-curves on top of each other where it'll be exponential and then we'll see all kinds of-

    2. JKJim Keller1:03:24

       What does an exponential of a million mean? That's a, that's a pretty amazing number.

    3. LFLex Fridman1:03:29

       Yeah.

    4. JKJim Keller1:03:29

       And that's just for a local little piece of silicon. Now let's imagine you, say, decided to get 1,000 tons of silicon to collaborate in one computer at a million times the density. Like, now, now you're talking, I don't know, 10 to the 20th more computation power than our current already unbelievably fast computers. Uh, like, nobody knows what that's gonna mean. You know, the sci-fi guys call it, you know, computronium. Like when, like a local civilization turns the nearby star into a computer.

    5. LFLex Fridman1:04:04

       Right.

    6. JKJim Keller1:04:04

       Like, I don't think that's true, but...

    7. LFLex Fridman1:04:06

       So just even when you shrink a transistor, the-

    8. JKJim Keller1:04:11

       That's only one dimension.

    9. LFLex Fridman1:04:12

       ... the ripple effects of that-

    10. JKJim Keller1:04:14

        Like, like people tend to think about computers as a cost problem, right? So computers are made out of silicon and minor amounts of metals and, you know, this and that. None of those things cost any money. Like, there's plenty of sand. Like, like you could just turn to beach and a little bit of ocean water into computers.

11. 1:04:33 – 1:20:51

    Elon Musk and Tesla Autopilot

    1. JKJim Keller1:04:33

       So, all the cost is in the equipment to do it, and the trend on equipment is once you figure out how to build the equipment, the trend of cost is zero. Elon said first you figure out what configuration you want the atoms in, and then how to put them there.

    2. LFLex Fridman1:04:49

       (laughs)

    3. JKJim Keller1:04:49

       Right?

    4. LFLex Fridman1:04:50

       Yeah. Right.

    5. JKJim Keller1:04:50

       'Cause, well, what, here's the, the, you know, his, his great insight is people are how constrained. I have this thing, I know how it works, and then little tweaks to that will generate something. As opposed to what do I actually want, and then figure out how to build it. It's a very different mindset and almost nobody has it, obviously.

    6. LFLex Fridman1:05:12

       Well, let me ask on that topic, you were one of the key early people in the development of autopilot, at least in the hardware side. Elon Musk believes that autopilot and vehicle autonomy, if you just look at that problem, can follow this kind of exponential improvement. In terms of the ho- the how question-

    7. JKJim Keller1:05:31

       Mm-hmm.

    8. LFLex Fridman1:05:31

       ... that we're talking about, there's no reason why it can't. What are your thoughts on this particular space of vehicle autonomy and your part of it and Elon Musk's and Tesla's vision for Well, the computer- vehiclet autonomy.

    9. JKJim Keller1:05:46

       ... you need to build was straightforward. And you could argue, well, does it need to be f- two times faster or five times or 10 times? But that's just a matter of time, like, or, or price in the short run. So that's, that's not a big deal. You don't have to be especially smart to drive a car, so it's not like a super hard problem. I mean, the big problem with safety is attention, which computers are really good at, not skills.

    10. LFLex Fridman1:06:12

        Well, let me push back on one m- you see, everything you said is correct, but...

    11. JKJim Keller1:06:17

        Okay.

    12. LFLex Fridman1:06:17

        We as humans tend to, um, uh, tend to take for granted how, how incredible our vision system is. So...

    13. JKJim Keller1:06:27

        You can drive a car with 20/50 vision and you can train a neural network to extract the distance of any object and the shape of any surface from a video in data.

    14. LFLex Fridman1:06:38

        Yeah, but that-

    15. JKJim Keller1:06:39

        It's really simple.

    16. LFLex Fridman1:06:40

        No, it's not simple. I, uh... (laughs)

    17. JKJim Keller1:06:42

        That's a simple data problem. (laughs)

    18. LFLex Fridman1:06:44

        (laughs) It's not, it's not simple. Uh, it's, because you, uh, 'cause it's not just detecting objects, it's understanding the scene and it's being able to do it in a way that doesn't make errors. So the, the beautiful thing about the human vision system and our entire brain around the whole thing is we're able to fill in the gaps. It's not just about perfectly detecting cars.

    19. JKJim Keller1:07:08

        Mm-hmm.

    20. LFLex Fridman1:07:08

        It's inferring the occluded cars. It's trying to, it's, it's understanding the physics-

    21. JKJim Keller1:07:12

        I think that's mostly a data problem.

    22. LFLex Fridman1:07:14

        Y- so you think what data...

    23. JKJim Keller1:07:16

        Yeah.

    24. LFLex Fridman1:07:16

        ... with compute, with improvement of computation, with improvement in collection of data-

    25. JKJim Keller1:07:20

        Well, there is a, you know, when you're driving a car and somebody cuts you off, your brain has theories about why they did it.

    26. LFLex Fridman1:07:26

        Right.

    27. JKJim Keller1:07:26

        You know, they're a bad person, they're distracted, they're dumb. You know, you can listen to yourself.

    28. LFLex Fridman1:07:31

        Right.

    29. JKJim Keller1:07:32

        So, y- you know, if you think that narrative is important to be able to successfully drive a car, then current autopilot systems can't do it. But if cars are ballistic things with tracks and probabilistic changes of speed and direction, and roads are fixed and given, by the way, they don't change dynamically...... right? You can map the world really thoroughly. You can place every object really thoroughly, right? You can calculate trajectories of things really thoroughly, right?

    30. LFLex Fridman1:08:06

        But everything you said about really thoroughly has a different degree of difficulty, so-
12. 1:20:51 – 1:28:33

    Lessons from working with Elon Musk

    1. LFLex Fridman1:20:51

       Or what have you learned, have taken away from your time working with Elon Musk, working at Tesla? Which is known to be a place of chaos, innovation, craftsmanship, and all of those things.

    2. JKJim Keller1:21:03

       I really liked the way he thought. Like, you think you have an understanding about what first principles of something is, and then you talk to Elon about it, and you, you didn't scratch the surface, you know? He, he has a deep belief that no matter what you do, is a local maximum, right? I had a friend, he invented a better electric motor, and, uh, it was like a lot better than what we were using. And one day he came by, he said, "You know, I'm a little disappointed 'cause, you know, this is really great and you didn't seem that impressed." And I said, "You know when the super intelligent aliens come, are they gonna be looking for you?" Like, "Where is he?" "The guy who built the motor." (laughs)

    3. LFLex Fridman1:21:41

       (laughs) Yeah.

    4. JKJim Keller1:21:42

       Probably not. You know? Like, like, the, like... but doing interesting work that's both innovative and let's say craftsman's work on the current thing, is really satisfying and it's good. And, and that's cool. And then, Elon was good at taking everything apart, like, what's the deep first principle? Oh, no, what's really the f- no, what's really... you know, you know-

    5. LFLex Fridman1:22:05

       Yeah.

    6. JKJim Keller1:22:05

       ... you know, that, that, you know, ability to look at it without assumptions and, and how constraints is, is super wild. You know, he built a rocket ship and-

    7. LFLex Fridman1:22:16

       Using that same kind of process.

    8. JKJim Keller1:22:17

       ... an electric car and, you know, everything. And that's super fun, and he's into it too. Like, when they first landed two SpaceX rockets at Tesla, we had a video projector in the big room and like 500 people came down, and when they landed everybody cheered and some people cried. It was so cool.

    9. LFLex Fridman1:22:33

       Yeah.

    10. JKJim Keller1:22:34

        All right, but how did you do that? Well, it was super hard. And then people say, "Well, it's chaotic." Really? To get out of all your assumptions, you think that's not gonna be unbelievably painful? And, is Elon tough? Yeah, probably. Do people look back on it and say, "Boy, I'm really happy I had that experience to go take apart that many layers of assumptions"? Sometimes super fun, sometimes painful.

    11. LFLex Fridman1:23:05

        So it could be emotionally and intellectually painful, that whole process of just stripping away assumptions?

    12. JKJim Keller1:23:10

        Yeah. Imagine 99% of your thought process is protecting your self-conception. And 98% of that's wrong.

    13. LFLex Fridman1:23:19

        Yeah.

    14. JKJim Keller1:23:20

        Now you got the math right. How do you think you're feeling when you get back into that one bit that's useful? And now you're open and you have the ability to do something different. I don't know if I got the math right, it might be 99.9, but, it ain't 50.

    15. LFLex Fridman1:23:38

        Imagining it, the 50% is hard enough.

    16. JKJim Keller1:23:42

        Yeah. Now, for a long time I've suspected you could get better. Like, you can think better, you can think more clearly, you can take things apart. And there's lots of examples of that. People who do that. So-

    17. LFLex Fridman1:23:59

        And Elon is an example of that.

    18. JKJim Keller1:24:01

        Apparently.

    19. LFLex Fridman1:24:01

        You are an example. So is-

    20. JKJim Keller1:24:02

        I don't know if I am. I'm, I'm fun to talk to.

    21. LFLex Fridman1:24:06

        (laughs) Certainly.

    22. JKJim Keller1:24:07

        I've learned a lot of stuff.

    23. LFLex Fridman1:24:08

        Right.

    24. JKJim Keller1:24:09

        Well here's the other thing is like, I, I joke, like, like I read books.

    25. LFLex Fridman1:24:12

        Yeah.

    26. JKJim Keller1:24:13

        And people think, "Oh, you read books." Well, no, I've read a couple books a week for s- 55 years.

    27. LFLex Fridman1:24:19

        Wow.

    28. JKJim Keller1:24:20

        Well, maybe 50, 'cause I didn't read, learn to read until I was eight or something. And, uh, and, and it turns out when people write books, they often take 20 years of their life where they passionately did something, reduce it to two- 200 pages. That's kind of fun. And then, they go, you go online and you can find out who wrote the best books and who like, you know... that's kind of wild. So there's this wild selection process and then you can read it and, for the most part, understand it. And then you can go apply it. Like I went to one company, I thought, "I haven't managed much before." So I read 20 management books and I started talking to them and basically, compared to all the VPs running around, I'd run night- read 19 more management books than anybody else.

    29. LFLex Fridman1:25:07

        (laughs)

    30. JKJim Keller1:25:07

        It wasn't even that hard.
13. 1:28:33 – 1:32:38

    Existential threats from AI

    1. LFLex Fridman1:28:34

       Speaking of unpleasant surprises, many people have worries about a singularity in the development of AI. Forgive me for such questions. (laughs)

    2. JKJim Keller1:28:43

       Mm. Yeah. (laughs)

    3. LFLex Fridman1:28:44

       When, when AI improves exponentially and reaches a point of superhuman level general intelligence, uh, you know, beyond the point there's no looking back, do you share this worry of existential threats from artificial intelligence from computers becoming superhuman level intelligent?

    4. JKJim Keller1:29:02

       No, not really. You know, like we already have a very stratified society. And then if you look at the whole animal kingdom of capabilities and abilities and interests and, you know, smart people have their niche and, you know, normal people have their niche and craftsmen have their niche and, you know, animals have their niche. I, I suspect that the domains of interest for things that, you know, astronomically different, like the whole s- something got 10 times smarter than us and wanted to track us all down because what? We like to have coffee at Starbucks? Like it, it doesn't seem plausible. Now, is there an existential problem in that how do you live in a world where there's something way smarter than you and you, you based your kind of self-esteem on being the smartest local person? Well, there's what .1% of the population who thinks that 'cause the rest of the population's been dealing with it since they were born. So the, the, the breadth of possible experience that can be interesting is really big. And, you know, super intelligence seems likely, although we still don't know if we're magical, but I suspect we're not. And it seems likely that it'll create possibilities that are interesting for us and its, its interests will be interesting for that, for whatever it is. It's not obvious why its interest would somehow wanna fight over some square foot of dirt or, you know, whatever, you know, the usual fears are about.

    5. LFLex Fridman1:30:37

       So you don't think it will inherit some of the darker aspects of human nature?

    6. JKJim Keller1:30:42

       Depends on how you think reality's constructed.

    7. LFLex Fridman1:30:45

       (laughs)

    8. JKJim Keller1:30:45

       So for, for whatever reason-

    9. LFLex Fridman1:30:48

       Yeah.

    10. JKJim Keller1:30:48

        ... human beings are in, let's say, creative tension and opposition with both our good and bad forces. Like there's lots of philosophical understanding of that, right? I don't know why that would be different.

    11. LFLex Fridman1:31:03

        So you think the evil is, is necessary for the good? I mean-

    12. JKJim Keller1:31:07

        Why?

    13. LFLex Fridman1:31:07

        ... the tension.

    14. JKJim Keller1:31:08

        I don't know about evil, but like we live in a competitive world where your good is somebody else's-

    15. LFLex Fridman1:31:14

        Right.

    16. JKJim Keller1:31:15

        ... you know, evil. You know, there's, there's the malignant part of it, but that seems to be self-limiting, although occasionally it's, it's ho- super horrible. But-

    17. LFLex Fridman1:31:27

        But yes, the... there's a debate over ideas and some people have different beliefs and that, that debate itself is a process so that arriving at something-

    18. JKJim Keller1:31:37

        Yeah. And why wouldn't-

    19. LFLex Fridman1:31:38

        ... something-

    20. JKJim Keller1:31:38

        ... that continue?

    21. LFLex Fridman1:31:39

        Yeah.

    22. JKJim Keller1:31:40

        You know.

    23. LFLex Fridman1:31:40

        It just... You... But you don't think that whole process will leave humans behind in a way that's painful?

    24. JKJim Keller1:31:47

        No.

    25. LFLex Fridman1:31:47

        Emotionally painful, yes, for the one per- for the .1% they'll be-

    26. JKJim Keller1:31:50

        You know, why isn't it already painful for a large percentage of the population? And it is. I mean, society-

    27. LFLex Fridman1:31:55

        It is.

    28. JKJim Keller1:31:55

        ... does have a lot of stress in it about the 1% and the, about the this and about the that, but, you know, everybody has a lot of stress in their life about what they find satisfying and, and, you know, know yourself seems to be the proper dictum and pursue something that makes your life meaningful seems proper.... and there's so many avenues on that. Like, there's so much unexplored space at every single level. I, uh, you know, uh, I'm, I'm somewhat of a... Uh, my nephew called me a jaded optimist.

Episode duration: 1:34:43