Jim Keller: The Future of Computing, AI, Life, and Consciousness | Lex Fridman Podcast #162

Lex Fridman 0:00 The following is a conversation with Jim Keller. His second time on the podcast. Jim is a legendary microprocessor architect and is widely seen as one of the greatest engineering minds of the computing age. In a peculiar twist of space time in our simulation, Jim is also a brother-in-law of Jordan Peterson. We talk about this and about computing, artificial intelligence, consciousness, and life. Quick mention of our sponsors: Athletic Greens all-in-one nutrition drink, Brooklinen Sheets, ExpressVPN, and Belcampo grass-fed meat. Click the sponsor links to get a discount and to support this podcast. As a side note, let me say that Jim is someone who, on a personal level, inspired me to be myself. There was something in his words, on and off the mic, or perhaps that he even paid attention to me at all, that almost told me, "You're all right, kid." A kind of pat on the back that can make the difference between a mind that flourishes and a mind that is broken down by the cynicism of the world. So, I guess that's just my brief few words of thank you to Jim, and in general, gratitude for the people who have given me a chance on this podcast, in my work, and in life. If you enjoy this thing, subscribe on YouTube, review it on Apple Podcasts, follow on Spotify, support on our Patreon, or connect with me on Twitter @lexfridman. And now, here's my conversation with Jim Keller.

Lex Fridman 1:33 What's the value and effectiveness of theory versus engineering, this dichotomy, in, uh, building good software or s- hardware systems?

Jim Keller 1:43 Well, it's... Good design is both. I guess that's pretty obvious. By engineering, do you mean, you know, reduction to practice of known methods? And then science is the pursuit of discovering things that people don't understand or solving unknown problems.

Lex Fridman 2:00 Definitions are interesting here, but I was thinking more in theory, constructing models that kind of generalize about how things work.

Jim Keller 2:08 Mm-hmm.

Lex Fridman 2:08 Engineering is, uh, like actually building stuff. The pragmatic, like-

Jim Keller 2:14 Mm-hmm.

Lex Fridman 2:14 ... okay, we have these nice models, but how do we actually get things to work? Maybe economics is a nice example. Like economists have all these models of how the economy works and how different policies will have an effect, but then there's the actual, okay, let's call it engineering of like-

Jim Keller 2:30 Yeah.

Lex Fridman 2:31 ... actually deploying the policies.

Jim Keller 2:33 So, computer design is almost all engineering and reduction to practice of known methods. Now, because of the complexity of the computers we build, you know, you- you could think you're, well, we'll just go write some code and then we'll verify it and then we'll put it together, and then you find out that the combination of all that stuff is complicated, and then you have to be inventive to figure out how to do it. Right? So that's- that's definitely ha- happens a lot. And then every so often some big idea happens, but it might be one person.

Lex Fridman 3:06 And that idea is in what? In the space of engineering or is it a to- in the space of ideas?

Jim Keller 3:10 Well, I'll give you an example. So, one of the limits of computer performance is branch prediction. So... And there's- there's a whole bunch of ideas about how good you could predict a branch. And people said there's a limit to it, and that's been taught a curve, and somebody came up with a better way to do branch prediction. It was a lot better. And he published a paper on it, and every computer in the world now uses it. And it was one idea. So the- the engineers who build branch prediction hardware were happy to drop the one kind of training array and put it in another one.

Lex Fridman 3:42 Mm-hmm.

Jim Keller 3:42 So, it was- it was a real idea.

Lex Fridman 3:44 And branch prediction is- is one of the key problems underlying all of sort of the lowest level of software. It boils down to branch prediction.

Jim Keller 3:53 Boils down to uncertainty. Computers are limited by... You know, single thread computer's limited by two things. The- the predictability of the path of the branches and the predictability of the locality of- of data. So, we have predictors that now predict both of those pretty well.

Lex Fridman 4:09 Yeah.

Jim Keller 4:09 So, memory is, you know, a couple hundred cycles away. Local cache is a couple cycles away. When you're executing fast, virtually all the data has to be in the local cache. So, a simple program says, you know, add one to every element in an array. It's really easy to see what the stream of data will be.

Lex Fridman 4:26 Mm-hmm.

Jim Keller 4:26 But you might have a more complicated program that's, you know, says get a- get an element of this array, look at something, make a decision, go get another element, it's kind of random. And you can think, that's really unpredictable. And then you make this big predictor that looks at this kind of pattern and you realize, well, if you get this data and this data, then you probably want that one. And if you get this one and this one and this one, you probably want that one.

Lex Fridman 4:47 And is that theory or is that engineering? Like the paper that was written, was it, uh-

Jim Keller 4:52 Well, that was prob-

Lex Fridman 4:52 ... asymptotic kinda- kinda discussion or is it more like here's a hack that works well?

Jim Keller 4:57 Um, it's a little bit of both. Like there's information theory in it, I think somewhere.

Lex Fridman 5:01 Okay. So it's-

Jim Keller 5:02 Yeah.

Lex Fridman 5:02 ... it's actually trying to prove some kind of stuff.

Jim Keller 5:03 Yeah. But once- once you know the method, implementing it is an engineering problem. Now, there's a flip side of this which is in a big design team, what percentage of people think their- their- their, uh, their- their- their plan or their life's work is engineering versus design... inventing things? So, lots of companies will reward you for filing patents.

Lex Fridman 5:27 Yes.

Jim Keller 5:27 Some... Many big companies get stuck because to get promoted you have to come up with something new. And then what happens is everybody's trying to do some random new thing, 99% of which doesn't matter, and the basics get neglected. And... Or they get to... There's a dichotomy. They think like the cell library and the basic CAD tools, you know, or basic, you know, software validation methods, that's simple stuff, you know. They want to work on the exciting stuff. And then they- they- they spend lots of time trying to figure out how to patent something.And that's mostly useless.

Lex Fridman 6:02 But the breakthrough is on the simple stuff.

Jim Keller 6:04 No, no, you... No. You have to do the simple stuff really well. If you're build- building a building out of bricks, you want great bricks. So you go to two places that sell bricks, and one guy says, "Yeah, they're over there in an ugly pile." And the other guy is like, lovingly tells you about the 50 kinds of bricks and how hard they are and how beautiful they are and how square they are. And you know, which one are you gonna buy bricks from? Which is going to make a better house?

Lex Fridman 6:30 So you- you're talking about the craftsman, the- the person who understands bricks, who loves bricks, who loves the variety?

Jim Keller 6:35 Yeah. That's a good word. You know, good engineering is great craftsmanship and when you start thinking engineering is about invention and-

Lex Fridman 6:45 Uh-huh.

Jim Keller 6:45 ... you set up a system that rewards invention, the craftsmanship gets neglected.

Lex Fridman 6:50 Okay, so maybe one perspective is the theory, the science o- over-emphasizes invention and engineering emphasizes craftsmanship and therefore-

Jim Keller 7:01 Mm-hmm.

Lex Fridman 7:01 Like, so if you, it doesn't matter what you do with theory in engineering-

Jim Keller 7:04 Well, everybody does. Like, read the tech rags. They're always talking about some breakthrough or innovat- innovation.

Lex Fridman 7:09 Yeah.

Jim Keller 7:09 And every- everybody thinks that's the most important thing. But the number of innovative ideas is actually relatively low. We need them, right? An innovation creates a whole new opportunity. Like when-

Lex Fridman 7:20 Mm-hmm.

Jim Keller 7:20 ... when some guy invented the internet, right? Like, that was a big thing. The million people that wrote software against that were mostly doing engineering software writing. So the elaboration of that idea was huge.

Lex Fridman 7:34 I don't know if you know Brendan Eich, he wrote JavaScript in 10 days.

Jim Keller 7:37 Uh-huh.

Lex Fridman 7:38 And that's an interesting story. It makes me wonder... And it was, you know, famously for many years considered to be a pretty crappy programming language.

Jim Keller 7:47 Mm-hmm.

Lex Fridman 7:47 It still is perhaps. It's been improving sort of consistently. But the interesting thing about that guy is, you know, he doesn't get any awards. (laughs)

Jim Keller 7:58 (laughs)

Lex Fridman 7:58 You don't get a Nobel Prize or a Fields Medal or-

Jim Keller 8:01 Uh-huh.

Lex Fridman 8:01 ... uh, he might not even-

Jim Keller 8:03 For inventing a- a crappy piece of, you know, software code that-

Lex Fridman 8:06 That- well, that is currently the number one programming language in the world and runs, now is cons- i- increasingly running the backend of the internet, the frontend of the internet.

Jim Keller 8:13 Well, does he, does he know why everybody uses it? Like, that would be an interesting thing. Was it the right thing at the right time? 'Cause like when stuff like JavaScript came out, like there was a move from, you know, writing C programs in C++ to, let's call it what they call managed code frameworks.

Lex Fridman 8:32 Mm-hmm.

Jim Keller 8:32 Where you write simple code, it might be interpreted, it has lots of libraries, productivity is high and you don't have to be an expert. So, you know, Java was supposed to solve all the world's problems. It was complicated. JavaScript came out, you know, after a bunch of other scripting languages. I'm not an expert on it but-

Lex Fridman 8:49 Yeah.

Jim Keller 8:49 ... but was it the right thing at the right time?

Lex Fridman 8:51 The right thing at the right-

Jim Keller 8:51 Or was there something, you know, clever? 'Cause he wasn't the only one.

Lex Fridman 8:56 There's a few elements. One is-

Jim Keller 8:57 And maybe if he figured out what it was-

Lex Fridman 8:59 No, I think-

Jim Keller 8:59 ... then he'd get a prize. (laughs)

Lex Fridman 9:02 (laughs)

Jim Keller 9:02 Like that's-

Lex Fridman 9:02 Constructive theory. (laughs)

Jim Keller 9:04 Yeah. You know, maybe this problem is he hasn't defined this, or he just needs a good promoter.

Lex Fridman 9:08 (laughs) Well, I think there's a bunch of blog posts written about it, which is like wrong is right. Which is like doing the crappy thing fast, just like hacking together the thing that answers some of the needs-

Jim Keller 9:23 Mm-hmm.

Lex Fridman 9:23 ... and then iterating over time, listening to developers, like listening to people who actually use the thing.

Jim Keller 9:28 Mm-hmm.

Lex Fridman 9:28 This is something you can do more in software.

Jim Keller 9:31 Yep.

Lex Fridman 9:31 Uh, but the right time, like you have to sense, you have to have a good instinct of when is the right time for the right tool and make it super simple and just get it out there. The problem is, this is true with hardware, this is less true with software, is there's backward compatibility that just drags behind you as, you know, as you try to fix all the mistakes of the past. But the- the timing...

Jim Keller 9:55 Was good. Yeah.

Lex Fridman 9:56 There's something about that. And it wasn't accidental. You have to, like, give yourself over to the... You have to have this, like, broad sense of what's needed now. You know, both scientifically-

Jim Keller 10:09 Mm-hmm.

Lex Fridman 10:09 ... and, like, the community and just like... This, it was obvious that, uh, there was no... The- the interesting thing about JavaScript is everything that ran in the browser at the time, like Java and- and I think other, like Scheme, other programming languages, they were all in a separate external container.

Jim Keller 10:29 Mm-hmm.

Lex Fridman 10:30 And then JavaScript was literally just injected into the web page.

Jim Keller 10:34 Mm-hmm.

Lex Fridman 10:34 It was the dumbest possible thing in- running in the same thread as everything else. And like, uh, it was inserted as a comment. So JavaScript code is inserted as a comment in the HTML code.

Jim Keller 10:47 Mm-hmm.

Lex Fridman 10:47 And it was... I mean, it's, there's... (laughs) It's either genius or super dumb, but it's like it's-

Jim Keller 10:53 Right. So it has no apparatus for like a virtual machine, a container.

Lex Fridman 10:56 No.

Jim Keller 10:56 It just executed in the framework of the program that's already running.

Lex Fridman 10:59 Yeah. And it was-

Jim Keller 11:00 Yeah, that's cool.

Lex Fridman 11:00 And then, uh, because something about that accessibility, the ease of its use, uh, resulted in then developers innovating of how to actually use it. I mean, I don't even know what to make of that, but there, it does seem to echo across different software, like stories of different software. PHP has the same story, really crappy language-

Jim Keller 11:22 Mm-hmm.

Lex Fridman 11:22 ... that just took over the world. Yeah.

Jim Keller 11:25 Well, we always have a joke that the- the random length instructions, the variable length instruction sets always won, even though they're obviously worse.

Lex Fridman 11:32 (laughs)

Jim Keller 11:33 Like, nobody knows why. X86 is arguably the worst architecture, you know, on the planet. It's one of the most popular ones.

Lex Fridman 11:40 Well, I mean, isn't- isn't that also the story of RISC versus CISC? I mean, is that simplicity? There's something about simplicity that, uh, us in this evolutionary process is valued.

Jim Keller 11:53 Yeah.

Lex Fridman 11:53 If it's simple, it's, uh, gets... It spreads faster, it seems like.

Jim Keller 11:58 Yeah.

Lex Fridman 11:58 Or is that not always true?

Jim Keller 12:00 That's not always true. Yeah. It could be simple is good, but too simple is bad.

Lex Fridman 12:04 So why did RISC win, you think, so far?

Jim Keller 12:06 Did RISC win?

Lex Fridman 12:07 (laughs)

Jim Keller 12:08 We don't know.

Lex Fridman 12:08 In the long arc of history, maybe not. (laughs)

Jim Keller 12:10 We, we don't know.

Lex Fridman 12:11 So, who, who's gonna win? What, what's RISC, what's CISC, and who's gonna win in that space in these instruction sets?

Jim Keller 12:17 Well, A-, AI software's gonna win, but there'll be little computers that run little programs like normal all over the place. But, but we're, we're going through another transformation, so.

Lex Fridman 12:28 B- but y- you think instruction sets underneath it all will change?

Jim Keller 12:32 Yeah, they evolve slowly. They, they don't matter very much.

Lex Fridman 12:35 They don't matter very much, okay.

Jim Keller 12:36 Yeah. I mean, the, the limits of performance are, you know, predictability of instructions and data. I mean, that's the big thing. And then the usability of it is some, you know, quality of design, quality of tools, availability. Like, right now, x86 is proprietary with Intel and AMD, but they can change it any way they want independently.

Lex Fridman 12:59 Mm-hmm.

Jim Keller 12:59 Right? Arm is proprietary to Arm, and they won't let anybody else change it. So, it's like a sole point. And RISC-V is open source, so anybody can change it, which is super cool. But that also might mean it gets changed in too many random ways that there's no common sub- subset of it that people can use.

Lex Fridman 13:17 Do you like open or do you like closed? Like, if you were to bet all your money on one or the other, RISC-V versus A?

Jim Keller 13:23 No idea.

Lex Fridman 13:24 It's case dependent?

Jim Keller 13:25 Well, x86, oddly enough, when Intel first started developing it, they licensed it to, like, seven people. So, it was the open architecture.

Lex Fridman 13:32 (laughs)

Jim Keller 13:33 And then they moved faster than others and also bought one or two of them. But there was seven different people making x86. 'Cause at the time, there was 6502 and Z80s and, you know, 8086. And you could argue everybody thought Z80 was the better instruction set, but that was propriety to, proprietary to one place. Oh, and the 6800.

Lex Fridman 13:56 So-

Jim Keller 13:56 There was, like, five differe- four or five different microprocessors. Intel went open, got the market share 'cause people felt like they had multiple sources from it, and then over time, it narrowed down to two players.

Lex Fridman 14:07 So, why ... You as a historian, uh, wh- (laughs) why did Intel win for so long with, uh, with their processors? I mean, I-

Jim Keller 14:17 They were great. Their process development was great.

Lex Fridman 14:21 So, it's ... Just looking back to JavaScript and is, uh, and Microsoft and Netscape and all these, uh, internet browsers, Microsoft won the browser game because they aggressively stole other people's ideas, like, right after they did it.

Jim Keller 14:37 You know, I, I don't know if Intel was stealing other people's ideas. They started making-

Lex Fridman 14:41 In a good way, stealing in a good way-

Jim Keller 14:42 They star-

Lex Fridman 14:43 ... just to clarify.

Jim Keller 14:43 Yeah, they started making RAMs, random access memories.

Lex Fridman 14:47 Yes.

Jim Keller 14:48 And then at the time when the Japanese manufacturers came up, you know, they, they were getting out-competed on that, and they pivoted to microprocessors, and they made the first, you know, integrated microprocessor to-

Lex Fridman 14:59 When did that-

Jim Keller 14:59 ... run programs. It was the, uh, 4004 or something.

Lex Fridman 15:03 Who was behind that pivot? That's a hell of a pivot.

Jim Keller 15:05 An- Andy Grove.

Lex Fridman 15:06 Oh.

Jim Keller 15:06 And he was great.

Lex Fridman 15:08 That's a hell of a pivot in hardware.

Jim Keller 15:10 Right, and then they led semiconductor industry ... Like, they were just a little company. IBM, all kinds of big companies had boatloads of money, and they out-innovated everybody.

Lex Fridman 15:21 Out-innovated, okay.

Jim Keller 15:22 Yeah, yeah.

Lex Fridman 15:23 So, it's not like marketing, it's not any of that stuff?

Jim Keller 15:26 And, and their processor designs were pretty good. Um, I think the, you know, Core 2 was probably the, the first one I thought was great. It was a really fast processor, and then Haswell was great.

Jim Keller 15:40 And al-

Lex Fridman 15:40 What, what makes a great processor in that? What, you know?

Jim Keller 15:42 Oh, if you just look at its performance versus everybody else, it's, you know, the size of it, the, you know, usability of it.

Lex Fridman 15:49 So, it's not specific, some kind of element that makes it beautiful, it's just, like, literally just raw performance. Is that how you think about processors, as just, like, raw performance?

Jim Keller 15:59 Of course. (laughs) It's like a horse race.

Lex Fridman 16:02 So-

Jim Keller 16:02 The fastest one wins. Now-

Lex Fridman 16:04 You don't care how. (laughs) Just as long as it wins.

Jim Keller 16:08 Well, well, there's the, the fastest in an environment, like-

Lex Fridman 16:11 Right.

Jim Keller 16:11 ... you know, for years, you made the fastest one you could, and then people started to have power limits. So, then you made the fastest at the right power point.

Lex Fridman 16:17 Yeah.

Jim Keller 16:17 And then, and then when we started doing multiprocessors, like, if you could scale your processors more than the other guy, you could be 10% faster on, like, a single thread, but you have more threads. So, there's lots of variability. And then Arm really explored, like, you know, they have the A series and the R series and the M series, like a family of processors for all these different design points, from, like, unbelievably small and simple. And so then when you're doing the design, it's sort of like this big palette of CPUs.

Lex Fridman 16:48 Mm-hmm.

Jim Keller 16:49 Like, they're the only ones with a credible, you know, top-to-bottom palette, and ...

Lex Fridman 16:54 Wh- wh- what do you mean a credible, uh, top-to-bottom palette?

Jim Keller 16:56 Well, there's people who make microcontrollers that are small, but they don't have a fast one. There's people who make fast processors, but don't have a litt- a medium one or a small one.

Lex Fridman 17:04 Is that hard to do, that full palette? That's, that seems like a ...

Jim Keller 17:08 Yeah, it's a lot of different

Jim Keller 17:09 ...

Lex Fridman 17:09 So, what's the difference between, uh, the Arm folks and Intel in terms of the way they're approaching this problem?

Jim Keller 17:15 Well, Intel, almost all their processor designs were, you know, very custom, high-end, you know, for the last 15, 20 years.

Lex Fridman 17:23 For the fastest horse possible.

Jim Keller 17:24 Yeah.

Lex Fridman 17:25 (laughs) In one horse race.

Jim Keller 17:27 And ... Yeah, and they, they, architecturally, they're really good, but the company itself was fairly insular to what's going on in the industry with CAD tools and stuff. And there's this debate about custom design versus synthesis, and how do you approach that? I, I'd say Intel was slow on the getting to synthesize processors. Arm came in from the bottom, and they generated IP, which went to all kinds of customers, so they had very little say in how the customer implemented their IP. So, Arm is super friendly to the synthesis IP environment, whereas Intel said, "We're gonna make this great client chip or server chip with our own CAD tools, with our own process, with our own, you know, other supporting IP, and everything only works with our stuff."

Lex Fridman 18:11 So, was that, um... Is Arm winning the mobile platform space in terms of process?

Jim Keller 18:17 Of course, yeah.

Lex Fridman 18:17 And so, i- in that, in... What you're describing is why they're winning.

Jim Keller 18:22 Well, they had lots of people doing lots of different experiments, so they controlled the processor architecture and IP, but they let people put in lots of different chips. And there was a lot of variability in what happened there. Whereas Intel, when they made their mobile, their foray into mobile, they had one team doing one part, right? So, it wasn't 10 experiments. And then their mindset was PC mindset, Microsoft software mindset, and that brought a whole bunch of things along that, uh, the mobile world and the embedded world don't do.

Lex Fridman 18:52 Do you think it was possible for Intel to pivot hard and win the mobile market?

Jim Keller 18:58 Sure.

Lex Fridman 18:58 That's a hell of a difficult thing to do, right? For a huge company to just pivot. I mean, it's so interesting to... 'Cause we'll talk about your current work. It's like, it's clear that PCs were dominating for several decades, like desktop computers, and then mobile, it's unclear.

Jim Keller 19:17 It's a, it's a leadership question. Like Ap- like Apple under Steve Jobs, when he came back, they pivoted multiple times.

Lex Fridman 19:25 Yeah.

Jim Keller 19:25 You know, they built iPads and iTunes and phones and tablets and great Macs. Like, like, who knew computers could be made out of aluminum? Nobody knew that. But they're great. It's super fun.

Lex Fridman 19:37 That was Steve?

Jim Keller 19:37 Yeah, Steve Jobs. Like, they pivoted multiple times, and, uh, you know, the old Intel, they, they did that multiple times. They made DRAMs and processors and processes and...

Lex Fridman 19:50 I gotta ask this. What was it like working with Steve Jobs?

Jim Keller 19:53 I didn't work with him.

Lex Fridman 19:54 Did you interact with him?

Jim Keller 19:55 Twice.

Lex Fridman 19:56 (laughs)

Jim Keller 19:57 I said hi to him twice in the cafeteria.

Lex Fridman 19:59 What did he say? Hi?

Jim Keller 20:01 He said, "Hey, fellas."

Lex Fridman 20:03 (laughs)

Jim Keller 20:04 He was friendly.

Lex Fridman 20:05 (laughs)

Jim Keller 20:05 He was wandering around, uh, with somebody. He couldn't find a table 'cause the cafeteria was, was packed, and I gave him my table. But I worked for Mike Colbert, who talked to... Like, Mike, Mike was the unofficial CTO of Apple and a brilliant guy, and he worked for Steve for 25 years, maybe more. And he talked to Steve multiple times a day, and he was one of the people who could put up with Steve's, let's say, brilliance and intensity. And, and Steve really liked him, and Steve trusted Mike to translate the shit he thought up into engineering products that worked. And then Mike ran a group called Platform Architecture, and I was in that group. So, many times, I'd be sitting with Mike, and the phone would ring, and it'd be Steve. And Mike would hold the phone like this 'cause Steve would be yelling about something or other.

Lex Fridman 20:52 Yeah, and then he would translate.

Jim Keller 20:54 And he'd translate, and then he would say, "Steve wants us to do this." So...

Lex Fridman 20:59 Was Steve a good engineer or no?

Jim Keller 21:01 I don't know. He was a great idea guy.

Lex Fridman 21:03 Idea person.

Jim Keller 21:04 And he was a really good selector for talent.

Lex Fridman 21:07 Yeah. That seems to be-

Jim Keller 21:08 So, again-

Lex Fridman 21:08 ... one of the key elements of leadership, right?

Jim Keller 21:10 And then he was a really good first principles guy. Like, like, somebody would say something couldn't be done, and he would just think, "That's obviously wrong," right? But, you know, maybe it's hard to do. Maybe it's expensive to do. Maybe we need different people. You know, there's like a whole bunch of... You know, like, if you want to do something hard, you know, maybe it takes time. Maybe you have to iterate. There's a whole bunch of things y- you could think about. But saying it can't be done is stupid.

Lex Fridman 21:36 How would you compare... So, it seems like Elon Musk is more engineering centric, but is also... I think he considers himself a designer, too. He has a design mind.

Jim Keller 21:47 Yeah.

Lex Fridman 21:47 Steve Jobs feels like he is much more idea space, design space versus engineering.

Jim Keller 21:52 Yeah.

Lex Fridman 21:52 Just make it happen. Like, the world should be this way. Just figure it out.

Jim Keller 21:56 But, but he used computers. You know, he had computer people talk to him all the time. Like, Mike was a really good computer guy. He knew what computers could do.

Lex Fridman 22:04 Computer meaning computer hardware, like low-level stuff?

Jim Keller 22:06 Yeah, hardware, software, all the pieces.

Lex Fridman 22:08 The whole thing.

Jim Keller 22:09 And then he would, you know, have an idea about what could we do with this next that was grounded in reality. It wasn't like he was, you know, just finger painting on the wall and wishing somebody would interpret it. Like, so he had this interesting connection because, you know, he wasn't a computer architect or a designer, but he had an intuition from the computers we had to what could happen. And...

Lex Fridman 22:33 It's interesting you say intuition, because it seems like he was pissing off a lot of engineers in his intuition about what can and can't be done. Those, the, like, the... What is it? All these stories about, like, floppy disks and all that kind of stuff like that.

Jim Keller 22:49 Yeah, so in, in Steve's the first round, like, he'd go into a lab and look at what's going on and hate it and, and, uh, fire people or, or ask somebody in the elevator what they're doing for Apple and, you know, not be happy. When he came back, my impression was, is he surrounded himself with a relatively small group of people.

Lex Fridman 23:10 Yes.

Jim Keller 23:10 And didn't really interact outside of that as much. And then the joke was you'd see like a little, somebody moving a prototype through the, the quad with a, with a black blanket over it.

Lex Fridman 23:20 (laughs)

Jim Keller 23:20 And that was 'cause it was secret, you know, partly from Steve 'cause they didn't want Steve to see it until it was ready.

Lex Fridman 23:27 Yeah, the dynamic with Jony Ive and Steve is interesting. It's like you don't wanna... He ruins as many ideas as he generates.

Jim Keller 23:37 Yeah. Yeah.

Lex Fridman 23:38 It's a dangerous kind of line to walk. I, I-

Jim Keller 23:41 And if you have a lot of ideas, like... Like, Gordon Bell was famous for ideas, right? And it wasn't that the percentage of good ideas was way higher than anybody else.

Lex Fridman 23:50 (laughs)

Jim Keller 23:51 It was he had so many ideas, and, and he was also good at talking to people about it and getting the filters right and, you know, seeing through stuff. Whereas Elon was like, "Hey, I want to build rockets." So, Steve would hire a bunch of rocket guys, and Elon would go read rocket manuals.

Lex Fridman 24:08 So, Elon is a better engineer, in a sense. Like, or, like, more, uh...... like, a love and passion for the manuals. (laughs)

Jim Keller 24:16 Yeah. And the details-

Lex Fridman 24:17 The details.

Jim Keller 24:18 ... and the data and the other stuff.

Lex Fridman 24:19 The craftsmanship too, right? Well, I guess Steve had craftsmanship too, but of a different kind.

Jim Keller 24:24 Yeah.

Lex Fridman 24:24 What do you make of the, just to stay in there for just a little longer, what do you make of, like, the anger and the passion and all of that? The, the firing and the mood swings and the madness, the im-, you know, being emotional and all of that, that's Steve? And I, I guess Elon too. So, what, is that a, is that a bug or a feature?

Jim Keller 24:43 It's a feature. So there's a graph which is, uh, y-axis productivity.

Lex Fridman 24:49 (laughs) Yeah.

Jim Keller 24:50 X-axis, at zero it's chaos.

Lex Fridman 24:52 Yeah.

Jim Keller 24:53 And infinity is complete order.

Lex Fridman 24:55 Yeah.

Jim Keller 24:55 Right? So as you go from the e- you know, the origin, as you improve order, you improve productivity.

Lex Fridman 25:03 Yeah.

Jim Keller 25:04 And at some point, productivity peaks, and then it goes back down again.

Lex Fridman 25:08 Yeah.

Jim Keller 25:08 Too much order, nothing can happen.

Lex Fridman 25:09 Yes. But the question is, is th- how close to the chaos is that peak?

Jim Keller 25:13 No, there, no, here's the thing-

Lex Fridman 25:14 Yeah.

Jim Keller 25:15 ... is once you start moving in the direction of order, the force vector to drive you towards order is unstoppable.

Lex Fridman 25:21 Oh. So it's this slippery slope.

Jim Keller 25:22 And every organization will move to the place where their productivity is stymied by order.

Lex Fridman 25:27 So you need a s-

Jim Keller 25:28 So the question is, who's the counterforce? Like, and 'cause it also feels really good. As you get more organized, then productivity goes up. The organization feels it. They orient to it, towards it, right?

Lex Fridman 25:39 Mm-hmm.

Jim Keller 25:39 They hire more people. They get more guys who can run process. You get bigger, right? And then inevitably, inevitably, the organization gets captured by the bureaucracy that manages all the processes.

Lex Fridman 25:52 Yeah.

Jim Keller 25:53 Right? And then humans really like that. And so if you just walk into a room and say, "Guys, love what you're doing, but I need you to have less order," if you don't have some force behind that, nothing will happen.

Lex Fridman 26:09 I, I can't tell you on how many levels that's profound. So...

Jim Keller 26:12 So that's why I say it's a feature. Now, could you be nicer about it? I don't know. I don't know any good examples of being nicer about it.

Lex Fridman 26:20 (laughs)

Jim Keller 26:21 Well, the, the funny thing is to get stuff done, you need people who can manage stuff and manage people, 'cause humans are complicated.

Lex Fridman 26:26 Yeah.

Jim Keller 26:26 They need lots of care and feeding, and you need to tell them they, they look nice and they're doing good stuff and pat them on the back, right?

Lex Fridman 26:33 I don't know. Uh, you tell me. Is that, is that needed?

Jim Keller 26:36 Oh, yeah.

Lex Fridman 26:36 Do humans need that?

Jim Keller 26:37 I had a friend, he started to manage a group, and he said, "I figured it out. You have to praise them before they do anything."

Lex Fridman 26:43 (laughs)

Jim Keller 26:43 "I was waiting till they were done, and they were always mad at me. Now I tell them what a great job they're doing while they're doing it." But then you get stuck in that trap, 'cause then when they're not doing something, how do you confront these people?

Lex Fridman 26:54 I, I think a lot of people that had trauma in their childhood would disagree with you, successful people, that you need to s- first do the rough stuff and then be nice later. I don't know. Mm-hmm.

Jim Keller 27:03 Okay, but, you know, engineering companies-

Lex Fridman 27:05 Being nice is just...

Jim Keller 27:05 ... are full of adults who had all kinds of range of childhoods, and, you know-

Lex Fridman 27:09 I don't know.

Jim Keller 27:09 Most people had okay childhoods.

Lex Fridman 27:11 Well, I don't know if, uh...

Jim Keller 27:12 I know lots of people who only work for praise, which is weird.

Lex Fridman 27:15 You mean like everybody. (laughs) What?

Jim Keller 27:18 I'm not that interested in it, but...

Lex Fridman 27:20 Ah.

Lex Fridman 27:21 Well, you, you're, you're probably looking for somebody's approval.

Jim Keller 27:24 Mm.

Lex Fridman 27:25 Uh, i- i- even still.

Jim Keller 27:27 Yeah, maybe. I should think about that.

Lex Fridman 27:29 Maybe somebody who's no longer with us kind of thing.

Jim Keller 27:32 Mm.

Lex Fridman 27:33 I don't know.

Jim Keller 27:34 I used to call up my dad and tell him what I was doing. He was, he was very excited about engineering and stuff.

Lex Fridman 27:38 You got his approval?

Jim Keller 27:40 Uh, yeah, a lot. I was lucky. Like, he, he decided I was smart and unusual as a kid, and that was okay, when I was really young. So when I d- like did poorly in school, I was dyslexic, I didn't read until I was third or fourth grade and they didn't care. My parents were like, "Oh, he'll be fine." So-

Lex Fridman 28:00 Cool.

Jim Keller 28:00 ... I was lucky. That was cool.

Lex Fridman 28:02 Is he still with us?

Jim Keller 28:05 No.

Lex Fridman 28:05 You miss him?

Jim Keller 28:06 Mm-hmm. Sure, yeah. He had Parkinson's and then cancer. His last 10 years were tough. And I killed him. Killing a man like that's hard.

Lex Fridman 28:18 The mind?

Jim Keller 28:19 Well, it was pretty good. Um, Parkinson's caused the slow dementia. And, uh, the c- the chemotherapy, I think, accelerated it. But it was like hallucinogenic dementia. So he was clever and funny and interesting and was, it was pretty unusual.

Lex Fridman 28:37 Do you remember conversations, uh-

Jim Keller 28:40 Oh, yeah, of course.

Lex Fridman 28:40 ... from that time? Like what, do you have fond memories of the guy?

Jim Keller 28:44 Yeah. Oh, yeah.

Lex Fridman 28:45 Anything come to mind?

Jim Keller 28:46 Uh, a friend told me one time I could draw a computer on the whiteboard faster than anybody he'd ever met. And I said, "You should meet my dad." Like, when I was a kid, he'd come home and say, "I was driving by this bridge, and I was thinking about it." And he'd pull out a piece of paper and he'd draw the whole bridge.

Lex Fridman 29:00 (laughs)

Jim Keller 29:01 He was a mechanical engineer.

Lex Fridman 29:03 Yeah.

Jim Keller 29:03 And he would just draw the whole thing and then he would tell me about it and then tell me how he would've changed it. And he had this, you know, idea that he could understand and conceive anything. And I, I just grew up with that, so that was natural. So if, you know, like, when I interview people, I ask them to draw a picture of something they did on a whiteboard.

Lex Fridman 29:21 Mm-hmm.

Jim Keller 29:21 And it's really interesting. Like, some people draw a little box, you know, and then they'll say, "And then this talks to this, and..."

Lex Fridman 29:28 Yeah.

Jim Keller 29:28 And I'll be like, "Oh, this is frustrating."

Lex Fridman 29:30 (laughs)

Jim Keller 29:30 And then I had this other guy come in one time, he says, "Well, I designed the floating point in this chip, but I'd really like to tell you how the whole thing works and then tell you how the floating point works inside of it."

Lex Fridman 29:38 Yeah.

Jim Keller 29:38 "Do you mind if I do that?" And he covered two whiteboards in-

Lex Fridman 29:40 Yeah.

Jim Keller 29:40 ... like, 30 minutes.

Lex Fridman 29:41 Yeah.

Jim Keller 29:42 And I hired him. Like-

Lex Fridman 29:43 Yeah.

Jim Keller 29:43 ... he was great.

Lex Fridman 29:44 This craftsman. I mean, that's the craftsman shrimp to that.

Jim Keller 29:47 Yeah, but also the, the mental agility to understand the whole thing.

Lex Fridman 29:51 Right.

Jim Keller 29:51 Put the pieces in context. You know, c- you know, real view of the balance of how the design worked. Because i- if you don't understand it properly, when you start to draw it, you'll, you'll fill up half the whiteboard with, like, a little piece of it and, you know, like, your ability to lay it out in an understandable way takes a lot of understanding, so.

Lex Fridman 30:11 And be able to sort of zoom in to the detail and then zoom out to the big picture.

Jim Keller 30:15 And zoom out really fast and...

Lex Fridman 30:16 What about the impossible thing? You said your dad believed that, uh, you could do anything.That's a weird feature for a craftsman.

Jim Keller 30:25 Yeah.

Lex Fridman 30:26 It seems that that, uh, echoes in your own behavior.

Jim Keller 30:30 Yeah.

Lex Fridman 30:30 Like, that's, that's the-

Jim Keller 30:32 Well, it's not that anybody can do anything right now, right? It's that if you work at it, you can get better at it, and there might not be a limit. And he did funny things like, like he always wanted to play piano, so at the end of his life, he started playing the piano when he had Parkinson's.

Lex Fridman 30:49 Wow.

Jim Keller 30:49 And he was terrible.

Lex Fridman 30:51 (laughs)

Jim Keller 30:51 But he thought if he really worked at it in this life, maybe the next life, he'd be better at it.

Lex Fridman 30:56 He might be onto something. (laughs)

Jim Keller 30:57 Yeah, could be. He enjoyed doing it.

Lex Fridman 31:01 Yeah.

Jim Keller 31:01 So, it's pretty funny.

Lex Fridman 31:03 (laughs) Do you think the perfect is the enemy of the good in hardware and software engineering? It's like, we were talking about JavaScript a little bit, and the messiness of the 10-day building process.

Jim Keller 31:14 Yeah, it's- that's, you know, creative tension, right?

Lex Fridman 31:17 Hmm.

Jim Keller 31:18 Th- the, so creative tension is when you have two different ideas that you can't do both, right?

Lex Fridman 31:24 Right.

Jim Keller 31:24 And the- and, but the fact that you want to do both causes you to go try to solve that problem. That's the creative part. So, if you're building computers, like some people say, "We have this schedule, and anything that doesn't fit in the schedule, we can't do." Right? And so they, they throw out the perfect because they have a schedule. I hate that.

Lex Fridman 31:45 (laughs)

Jim Keller 31:45 Right? Then there's other people that say, "We need to get this perfectly right and no matter what." You know, more people, more money, right? And there's a really clear idea about what you want, and some people are really good at articulating it, right? So, so let's call that the perfect, yeah.

Lex Fridman 32:02 Yeah.

Jim Keller 32:03 All right, but that's also terrible because then you never ship anything and you never hit any goals. So, now you have the, now you have your framework.

Lex Fridman 32:10 Yes.

Jim Keller 32:10 You can't throw out stuff because you can't get it done today because maybe you'll get it done tomorrow with the next project, right? You can't... Y- so you have to... I worked with a guy that I really liked working with, but he over filters his ideas.

Lex Fridman 32:23 Over filters?

Jim Keller 32:24 He'd start thinking about something, and as soon as he figured out what was wrong with it, he'd throw it out.

Lex Fridman 32:29 Hmm.

Jim Keller 32:29 And then I start thinking about it, and like, you know, you come up with an idea, and then you find out what's wrong with it, and then you le- give it a little time to set because sometimes, you know, you figure out how to tweak it, or maybe that idea helps some other idea. So, idea generation is really funny. So, you have to give your ideas space, like spaciousness of mind is key, but you also have to execute programs and get shit done. And then it turns out computer engineering is fun because it takes, you know, 100 people to build a computer, 200 or 300, whatever the number is. And people are so variable about, you know, temperament and, you know, skill sets and stuff that in a, in a big organization, you find the, the people who love the perfect ideas and the people that want to get stuff done yesterday, and people like that- that come up with ideas, and people who like to, let's say, shoot down ideas. And it takes the whole... It takes a large group of people.

Lex Fridman 33:22 So, some are good at generating ideas, some are good at filtering ideas, and then all th- in that, uh, giant mess, you're somehow... I guess the goal is for that giant mess of people to, uh, find the perfect path through the-

Jim Keller 33:36 Mm-hmm.

Lex Fridman 33:36 ... the tension, the creative tension. But like, how do you know when... You said there's some people good at articulating what perfect looks like, what a good design is.

Jim Keller 33:44 Mm-hmm.

Lex Fridman 33:45 Like, if you're sitting in a, in a room, and, uh, you have a set of ideas about like how to design, uh, a better processor. How do you know this is, this is something special here, this is a good idea, let's try this?

Jim Keller 34:00 So, have you ever brainstormed an idea with a couple of people that were really smart? And you kind of go into it, and you, you don't quite understand it, and you're working on it, and then you start, you know, talking about it, putting it on the whiteboard. Maybe it takes days or weeks, and then your brains start to kind of synchronize. It's really weird.

Lex Fridman 34:19 (laughs) What's the-

Jim Keller 34:20 And like you start to see what each other is thinking.

Lex Fridman 34:26 Yeah.

Jim Keller 34:26 And, and it starts to work. Like, you can see it work. Like, my talent in computer design is I can, I can see how computers work in my head, like really well. And I know other people can do that too. And when you're working with people that can do that, like, it, it is kind of a, an amazing experience. And then... And every once in a while, you, you get to that place, and then you find the flaw in it, which is kind of funny because you, you can, you can fool yourself in, but-

Lex Fridman 34:53 The two of you kind of drifted a- a long, uh-

Jim Keller 34:56 Yeah, yeah, you got these-

Lex Fridman 34:56 ... into a direction that was useless. (laughs)

Jim Keller 34:58 Yeah, that happens too. Like, you have to... 'Cause, you know, the- it... Well, the nice thing about computer design, there's always reduction in practice. Like, you come up with your good ideas. And I know some architects who really love ideas, and then they work on them, and they put it on the shelf, then go work on the next idea, then put it on the shelf. They never reduce the practice. So, they find out what's good and bad, 'cause almost every time I've done something really new, by the time it's done, like, the good parts are good, but I know all the flaws, like...

Lex Fridman 35:27 Yeah. Would you say your career, just your own experience, is your career defined by mostly by flaws or by successes? Like, if-

Jim Keller 35:35 Oh, again, there's great tension between those. If you haven't tried hard-

Lex Fridman 35:41 Yeah.

Jim Keller 35:41 ... right, and done something new, right, then you're- you're not gonna be facing the challenges when you build it, then you find out all the problems with it. And-

Lex Fridman 35:52 But when you look back, do you see problems or?

Jim Keller 35:54 But

Lex Fridman 35:54 territory. Okay.

Jim Keller 35:56 Oh, when I look back, um...

Lex Fridman 35:58 What do you remember?

Jim Keller 35:58 I think earlier in my career-

Lex Fridman 36:00 Yeah.

Jim Keller 36:00 ... like, EV5 was the second alpha chip.

Lex Fridman 36:03 Mm-hmm.

Jim Keller 36:03 Uh, I was so embarrassed about the mistakes, I could barely talk about it. And it was in the Guinness Book of World Records, and it was the fastest processor on the planet.

Lex Fridman 36:12 Yeah.

Jim Keller 36:13 So, it was... And at some point, I realized that was really a bad mental framework to deal with, like doing something new. We did a bunch of new things, and some worked out great, and some were bad. And we learned a lot from it, and, and-... the next one, we learned a lot. That als- you know, EV6 also had some really cool things in it. I think the pro- portion of good stuff went up, but it had a couple fatal flaws in it that were painful. And then, uh, yeah.

Lex Fridman 36:41 Y- you, you learned to channel the pain into, like, pride.

Jim Keller 36:44 Not pride, really. You know, just, uh, realization about how the world works-

Lex Fridman 36:49 Okay.

Jim Keller 36:50 ... or how it, how that kind of idea set works.

Lex Fridman 36:52 Life is suffering. That's the reality. What, uh-

Jim Keller 36:55 No, it's not. (laughs)

Lex Fridman 36:57 Well-

Jim Keller 36:57 I know the Buddha said that, and a couple other people are, are stuck on it. No, it's, you know, there's this kinda weird combination of good and bad and, you know, light and darkness that you have to tolerate and, you know, deal with. Yeah, there's definitely lots of suffering in the world.

Lex Fridman 37:12 Depends on the perspective. It seems like there's way more darkness, but, uh, that makes the light part really nice.

Lex Fridman 37:18 What, uh, computing hardware or, um, just any kind, even software design are you, uh, do you find beautiful? From your own work, from o- o- o- other people's work that you're just, uh... We were just talking about the, the battleground of flaws and mistakes and errors, but things that were just beautifully done. Is there something that pops to mind?

Jim Keller 37:44 Well, when things are beautifully done, usually there's a well set, thought out set of abstraction layers. Like-

Lex Fridman 37:53 So, the whole thing works qu- like, in unison nicely.

Jim Keller 37:56 Yes. And, and when I, when I say abstraction layer, that means two different components, when they work together, they work independently. They don't have to know what the other one is doing.

Lex Fridman 38:07 Hmm. So, that decoupling.

Jim Keller 38:08 Yeah. So, the, the famous one was, uh, the network stack. Like, there's a seven-layer network stack.

Lex Fridman 38:12 Yep.

Jim Keller 38:13 You know, data transport and protocol and all the layers. And the innovation was, is when they really rote got that right, 'cause networks before that didn't define those very well. The layers could innovate independently, and occasionally the layer boundary would, would, you know, the interface would be upgraded. And that, that let, you know, the, the design space breathe.

Lex Fridman 38:34 Mm-hmm.

Jim Keller 38:35 And pe- you could do something new in layer seven without having to worry about how layer four worked.

Lex Fridman 38:40 Right.

Jim Keller 38:40 And so good design does that. And you see it in processor designs. When we did, um, the Zen design at AMD, we made several components very modular. And, you know, my insistence at the top was I wanted all the interfaces defined before we wrote the RTL for the pieces. One of the verification leads said, "If we do this right, I can test the pieces so well independently, when we put it together, we won't find all these interaction bugs 'cause the floating point knows how the cache works." And I was a little skeptical, but he was mostly right, that the, the modularity of the design greatly improved the quality.

Lex Fridman 39:19 Is that universally true in general? Would you say about good designs, the modularity is, uh, like usually modular?

Jim Keller 39:24 Well, we talked about this before. Humans are only so smart, like, like-

Lex Fridman 39:27 (laughs)

Jim Keller 39:27 ... and we're not getting any smarter, right? But the complexity of things is going up.

Lex Fridman 39:32 Yeah.

Jim Keller 39:32 So, you know, a, a beautiful design can't be bigger than the person doing it. It's just, you know, their piece of it. Like, the odds of you doing a really beautiful design of something that's way too hard for you is low, right? If it's way too simple for you, it's not that interesting. It's like, "Well, anybody could do that." But when you get the right match of your, your expertise and, you know, mental power to the right design size, that's cool, but that's not big enough to make a meaningful impact on the world. So now, you have to have some framework to design the pieces-

Lex Fridman 40:06 Yes.

Jim Keller 40:06 ... so that the whole thing is big and harmonious, but, you know, when you put it together, it's, you know, it's sufficently- sufficiently interesting to, to be used and, you know. So that's what a good, beautiful design is.

Lex Fridman 40:23 Matching the limits of that human cognitive capacity to, uh, to the modular you can create and creating a nice interface between those modules. And thereby, do you think there's a limit to the kind of beautiful complex systems we can build with this kind of modular design? It's like, uh, you know, if, if we build increasingly more complicated... You can think of, like, the internet. Okay, let's scale it down.

Jim Keller 40:50 Well-

Lex Fridman 40:50 Like, you can think of, like, social network, like Twitter-

Jim Keller 40:53 Mm-hmm.

Lex Fridman 40:53 ... as one computing system.

Jim Keller 40:56 Mm-hmm.

Lex Fridman 40:57 And, but those are little modules.

Jim Keller 40:59 Yeah.

Lex Fridman 40:59 Right? That's-

Jim Keller 41:00 But it's built on, it's built on so many components nobody at Twitter even understands.

Lex Fridman 41:06 Right. That's the-

Jim Keller 41:06 So, so, so if an alien showed up and looked at Twitter, he wouldn't just see Twitter as a beautiful, simple thing that everybody uses, which is really big. He would see the networks it runs on, the fiber optics, the data's transported, the computers. The whole thing is so bloody complicated, nobody at Twitter understands it. And so the-

Lex Fridman 41:24 You think that's what the alien (laughs) would sees? So yeah, if an alien showed up and looked at Twitter-

Jim Keller 41:28 Hmm.

Lex Fridman 41:28 ... or looked at the various different network systems that you can see on Earth...

Jim Keller 41:33 So, imagine they were really smart and they could comprehend the whole thing, and then they sort of, you know, evaluated a human and thought, "This is really interesting. No human on this planet comprehends the system they built."

Lex Fridman 41:45 No individual, or would they even see individual humans as an int- like, we humans are very human-centric, entity-centric, and so we think of us as the organ- as the central organism and the networks as just the connection of organisms. But from a perspective of an ali- from an outside perspective-

Jim Keller 42:04 Yeah.

Lex Fridman 42:04 ... it seems like we're just one organism-

Jim Keller 42:06 Yeah, we're, yeah, I get it. We're the ants, and they'd see the ant colony.

Lex Fridman 42:08 The ant colony, yeah.

Jim Keller 42:10 Yeah, yeah.

Lex Fridman 42:10 Or the result of production of the ant colony, which is like cities and-

Jim Keller 42:14 Yeah.

Lex Fridman 42:15 It's, it's, uh...

Jim Keller 42:17 Yeah.

Lex Fridman 42:18 In that sense, humans are pretty impressive, the modularity that we're able to... and the, and how robust we are to noise and mutation, all that kind of stuff.

Jim Keller 42:26 Well, that's 'cause it's stress tested all the time.

Lex Fridman 42:28 Yeah. (laughs)

Jim Keller 42:29 You know, you build all these cities with buildings and you get-

Lex Fridman 42:31 Yeah.

Jim Keller 42:31 ... earthquakes occasionally, and-

Lex Fridman 42:33 Wars.

Jim Keller 42:33 ... you know, some, you know, wars, earthquakes.

Lex Fridman 42:35 Viruses every once in a while. (laughs)

Jim Keller 42:37 ... you know-

Lex Fridman 42:37 (laughs)

Jim Keller 42:38 ... changes in business plans for, you know, like, shipping or something. Like, like, as long as it's all stress tested, then it kee- it keeps adapting to the, the situation. So the... Yeah, it's, it's a curious phenomena.

Lex Fridman 42:52 Well, let's go... Let's talk about Moore's Law a little bit. It's, uh-

Jim Keller 42:55 Mm-hmm.

Lex Fridman 42:56 Uh, at the broad view of Moore's Law was just exponential improvement of, uh, computing capability. Uh, like, OpenAI, for example, recently, uh, published this kind of... Papers looking at the exponential improvement in the training efficiency of neural networks.

Jim Keller 43:16 Mm-hmm.

Lex Fridman 43:17 For, like, ImageNet and all that kind of stuff, we just got better on this... And this is purely software side-

Jim Keller 43:22 Mm-hmm.

Lex Fridman 43:22 ... just figuring out better tricks and algorithms for training neural networks, and that seems to be improving, uh, significantly faster than the Moore's Law prediction, you know?

Jim Keller 43:32 Mm-hmm.

Lex Fridman 43:33 So that's in the software space. Like, what do you think... If Moore's Law continues or if the general version of Moore's Law continues, do you think that comes mostly from the hardware, from the software, some mix of the two? Some interesting totally, uh... So not, not the reduction of the size of the transistor kind of thing, but more in the, uh, uh, in the totally interesting kinds of innovations-

Jim Keller 43:59 Mm-hmm.

Lex Fridman 43:59 ... in the hardware space, all that kind of stuff?

Jim Keller 44:01 Well, there's, like, a half a dozen things going on in that graph. So one is, there's initial innovations that had a lot of hea- room to be exploited. So, you know, the efficiency of the networks has improved dramatically. And then the decomposability of those and the, the, the use go... You know, they started running on one computer, then multiple computers, then multiple GPUs, and then arrays of GPUs, and they're up to thousands. And at some point... So, so it's sort of like they were consume- they were going from, like, a single computer application to a thousand computer application. So that's not really a Moore's Law thing, that's an independent vector. How many computers can I put on this problem?

Lex Fridman 44:42 Yeah.

Jim Keller 44:42 'Cause the computers themselves are getting better on, like, a Moore's Law rate, but their ability to go from 1 to 10 to 100 to 1,000-

Lex Fridman 44:49 Yeah.

Jim Keller 44:50 ... you know, was something. And then multiplied by, you know, the amount of computes it took to resolve like AlexNet to ResNet to transformers. It's, it's been quite, you know, steady improvements.

Lex Fridman 45:01 But those are like S-curves, aren't they?

Jim Keller 45:03 Yeah.

Lex Fridman 45:03 That's the exactly kind of-

Jim Keller 45:04 Yeah.

Lex Fridman 45:04 ... S-curves that are underlying Moore's Law from the very beginning.

Jim Keller 45:06 Yeah, so-

Lex Fridman 45:08 So what, what's the biggest... What's the most, uh, productive, uh, rich source of S-curves in the, in the future do you think? Is this hardware? Is it software? Or is it's-

Jim Keller 45:18 So hardware is gonna move along relatively slowly, like, you know, double performance every two years.

Lex Fridman 45:26 (laughs)

Jim Keller 45:27 There, there's still-

Lex Fridman 45:28 I like how you call that slow.

Jim Keller 45:29 Yeah, it's the slow version. The snail's pace of Moore's Law. Maybe we should, we should, uh-

Lex Fridman 45:35 (laughs)

Jim Keller 45:35 ... we should, uh, trademark that one.

Lex Fridman 45:37 (laughs) That's-

Jim Keller 45:38 Where- whereas the scaling by number of computers, you know, can go much faster, you know? I'm sure at some point Google had a... You know, their initial search engine was running out of laptop, you know, like...

Lex Fridman 45:49 Yeah.

Jim Keller 45:50 And at some point they really worked on scaling that, and then they factored the, the indexer from, you know, this piece and this piece and this piece, and they spread the data on more and more things and... You know, they did a dozen innovations. But as they scaled up the number of computers on that, it kept breaking, finding new bottlenecks in their software and their schedulers and, and made 'em rethink. Like, it seems insane to do a scheduler across a thousand computers to schedule parts of it and then send the results to one computer. But if you wanna schedule a million searches, uh, that makes perfect sense.

Lex Fridman 46:23 But-

Jim Keller 46:23 So, so there's... The, the, the scaling by just quantity is probably the richest thing. But then as you scale quantity, like a network that was great on 100 computers may be completely the wrong one. You may pick a network that's 10 times slower on 10,000 computers, like per computer.

Lex Fridman 46:42 Mm-hmm.

Jim Keller 46:42 But if you go from 100 to 10,000, that's 100 times. So that's one of the things that happened when we did internet scaling, is the efficiency went down, not up. The future of computing is inefficiency, not efficiency.

Lex Fridman 46:55 But scales, inefficient scale. (laughs)

Jim Keller 46:57 But scale. It's, it's scaling faster than inefficiency bites you. And as long as there's, you know, dollar value there. Like, scaling costs lots of money.

Lex Fridman 47:05 Yeah.

Jim Keller 47:06 But Google showed, Facebook showed, everybody showed that the scale was where the money was at.

Lex Fridman 47:10 It was... And so it was, it was wo- worth it financially. Do you think... Is it possible that, like, basically the entirety of Earth will be like a computing surface? Like, this table will be doing computing. This hedgehog will be doing computing. Like, everything really inefficient down computing will be

Jim Keller 47:29 ... the science fiction books, they call it computronium.

Lex Fridman 47:31 Computronium?

Jim Keller 47:32 We, we turn everything into computing. Well, most of the elements aren't very good for anything.

Lex Fridman 47:37 (laughs)

Jim Keller 47:38 Like, you're not gonna make a computer out of iron. Like, you know, silicon and, and carbon have, like, nice structures, you know? Well, we'll see what, what you can do with the rest of it.

Lex Fridman 47:48 No, I just-

Jim Keller 47:48 Like, people talk about, "Well, maybe we can turn the sun into computer," but it's, it's hydrogen and hel- a little bit of helium, so.

Lex Fridman 47:55 What I, what I mean is more like actually just adding computers to everything.

Jim Keller 47:59 Oh, okay.

Lex Fridman 48:00 So-

Jim Keller 48:00 I thought you're just con- converting all the mass of the universe into computer.

Lex Fridman 48:04 No, no, no. So not using-

Jim Keller 48:05 It'd be ironic from the simulation point of view is like, (laughs) the simulator build mass to simulate, like...

Lex Fridman 48:12 Yeah, I mean, yeah. So, I mean, ultimately this is all heading towards a simulation yes, but-

Jim Keller 48:15 Yeah, well, I, I think I might have told you this story. At Tesla they were deciding... So they want to measure the current coming out of the battery, and they decide between putting a resistor in there and putting a computer on, with a sensor in there.

Lex Fridman 48:29 Mm-hmm.

Jim Keller 48:29 And the computer was faster than the computer I did- I worked on in 1982. And we, we chose the computer 'cause it was cheaper than the resistor.So, so, sure, this hedgehog, you know, it costs $13, and we can put a, you know, an AI that's as smart as you in there for five bucks. It'll have one. You know, so computers will be, you know, be everywhere.

Lex Fridman 48:51 I was hoping it wouldn't be smarter than me because-

Jim Keller 48:54 Well, everything is going to be smarter than you.

Lex Fridman 48:56 But you were saying it's inefficient. I thought it was better to have a lot of dumb things.

Jim Keller 49:00 Well, well, Moore's law will slowly compact that stuff.

Lex Fridman 49:02 So even the dumb things will be smarter than us.

Jim Keller 49:04 The dumb things are gonna be smart or they're gonna be smart enough to talk to something that's really smart.

Lex Fridman 49:10 (laughs)

Jim Keller 49:10 You know, it's like... Well, just remember, like, a big computer chip-

Lex Fridman 49:15 Yeah.

Jim Keller 49:15 ... you know, it's like an inch by an inch and, you know, 40 microns thick. It doesn't take very much, very many atoms to make a, you know, high-powered computer.

Lex Fridman 49:25 Yeah.

Jim Keller 49:25 And 10,000 of them can fit in a shoebox. But, you know, you have the, the cooling and power problems, but, you know, people are working on that.

Lex Fridman 49:33 But they still can't write, uh, compelling poetry or music or, uh, understand what love is or have a fear of mortality, so, so we're still winning.

Jim Keller 49:43 Neither can most of humanity, so. (laughs)

Lex Fridman 49:46 Well, they can write books about it. So, uh... (laughs)

Jim Keller 49:49 (laughs)

Lex Fridman 49:50 But, but speaking about this, uh...

Jim Keller 49:53 Yeah.

Lex Fridman 49:53 ... uh, this walk along the path of innovation towards, uh, the dumb things being smarter than humans, you are now-

Jim Keller 50:01 Mm-hmm.

Lex Fridman 50:01 ... the CTO of (laughs) of, uh, Tenstorrent.

Jim Keller 50:06 Mm-hmm.

Lex Fridman 50:06 Two- as of two months ago. They, uh, build hardware for deep learning.

Jim Keller 50:11 Mm-hmm.

Lex Fridman 50:13 Uh, how do you build scalable and efficient deep learning? This is such a fascinating space.

Jim Keller 50:17 Yeah, yeah. So it's interesting. So, um, up until recently, I thought there was two kinds of computers. There are serial computers that run like C programs, and then there's parallel computers. So the way I think about it is, you know, parallel computers y- have given parallelism. Like, GPUs are great because you have a million pixels.

Lex Fridman 50:34 Mm-hmm.

Jim Keller 50:34 And modern GPUs run a program on every pixel. They call it a shader program, right? So, or, like, finite element analysis. You, you build something, you know, you make this into little tiny chunks, you give each chunk to a computer, so you're given all these chunks, you have parallelism like that. But most C programs, you write this linear narrative, and you have to make it go fast. To make it go fast, you predict all the branches, all the data fetches, and you run that more in parallel, but that's found parallelism.

Lex Fridman 51:01 Mm-hmm.

Jim Keller 51:03 Um, AI is... I'm still trying to decide how fundamental this is. It's a given parallelism problem.

Lex Fridman 51:10 Mm-hmm.

Jim Keller 51:10 But the way people describe the neural networks and then how they write them in PyTorch, it makes graphs.

Lex Fridman 51:17 Yeah. That might be fundamentally different than the GPU kind of-

Jim Keller 51:21 Parallelism? Yeah, it might be. Because the, when you run the GPU program on all the pixels, you're running like, you know, depends, you know, this group of pixels, say it's background blue and it runs a really simple program. This pixel is, you know, some patch of your face, so you have some really interesting shader program to give you impression of translucency, but the pixels themselves don't talk to each other. There's no graph, right? So you, you do the image and then you do the next image and you do the next image and you run eight million pixels, eight million programs every time, and modern GPUs have like 6,000-

Lex Fridman 51:57 Mm-hmm.

Jim Keller 51:57 ... thread engines in 'em. So, you know, they got eight million pixels. Each one runs a program on, you know, 10 or 20 pixels, and that's how, uh, th- that's how they work. There's no graph.

Lex Fridman 52:09 But you think graph might be a totally, uh, new way to think about hardware?

Jim Keller 52:14 So, Raja Koduri and I have been having this good conversation about given versus found parallelism, and then the kind of walk as we got more transistors, like, you know, computers way back when did stuff on scale or data, then we did it on vector data, famous vector machines. Now we're making computers that operate on matrices, right? And then the, the ca- the category we, we said that was next was spatial. Like, imagine you have so much data that, you know, you want to do the compute on this data, and then when it's done, it says send the result of this pile of data, run some software on that.

Lex Fridman 52:49 Mm-hmm.

Jim Keller 52:49 And it's better to, to think about it spatially than to move all the data to a central processor and do all the work.

Lex Fridman 52:57 So spatially, you mean moving in the space of data as opposed to moving the data?

Jim Keller 53:02 Yeah. You know, you have a, you have a petabyte data space spread across some huge array of computers, and when you do a computation somewhere, you send the result of that computation or maybe a pointer to the next program to some other piece of data and do it. But I think the... a better word might be graph, and all the AI neural networks are graphs. Do some computations, send the result here, do another computation, do a data transformation, do a merging, do a pooling, do another computation.

Lex Fridman 53:30 Is it possible to compress and say how we make this thing efficient, this whole process efficient? There's different...

Jim Keller 53:37 So first, uh, the fundamental elements in the graphs are things like matrix multiplies, convolutions-

Lex Fridman 53:43 Okay.

Jim Keller 53:43 ... data manipulations, and data movements.

Lex Fridman 53:45 Yeah.

Jim Keller 53:46 So GPUs emulate those things with their-

Lex Fridman 53:48 Yes.

Jim Keller 53:48 ... little singles, you know, basically running a single threaded program.

Lex Fridman 53:52 Yeah.

Jim Keller 53:53 And then there's a, you know, NVIDIA calls it a warp where they group a bunch of programs that are similar together-

Lex Fridman 53:58 Mm-hmm.

Jim Keller 53:58 ... so for efficiency and instruction use. And then at a higher level, you kind of, you take this graph and you say, "This part of the graph is a matrix multiplier which runs on these thir- 32 threads." But the model at the bottom was built for running programs on pixels, not executing graphs.

Lex Fridman 54:17 Yeah, so it's emulation ultimately.

Jim Keller 54:19 Yes, yes. Yeah.

Lex Fridman 54:19 So is it possible to build something that natively runs graphs?

Jim Keller 54:23 Yes. So that's what Tenstorrent did. So-

Lex Fridman 54:27 Where are we on that? How, like, in the history of that effort, are we in the early days?

Jim Keller 54:32 Yeah, I think so. Tenstorrents started by a friend of mine, Lobisha Bajek, and I... I was his first investor. So I've been-

Lex Fridman 54:39 Uh-huh.

Jim Keller 54:39 ... you know, kind of following him and talking to him about it for years, and in the fall when I was considering things to do...... I decided, you know, the, the... We, we, we held a conference last year with a f- a friend organized it, and s- and we, we wanted to bring in thinkers, and two of the people were Andrej Karpathy and Chris Lattner. And Andrej gave this talk, it's on YouTube, called Software 2.0-

Lex Fridman 55:04 Mm-hmm.

Jim Keller 55:05 ... which I think is great. Which is... Where we went from programmed computers, where you write programs, to data program computers. You know, like the future is, you know, of software, is data programs, the, the networks, and I think that's true. And then, um, Chris has been work- he worked on LLVM, the Low-level Virtual Machine, which became the intermediate representation for all compilers. And now he's working on another project called MLIR, which is Mid-level Intermediate Representation, which is essentially under the graph about how do you represent that kind of computation and then coordinate large numbers of potentially heterogeneous computers. And, and I would say technically Tense Torrents', you know, two pillars are those, those, those two ideas, Software 2.0 and mid-level representation. But it's in service of executing graph programs. The hardware is designed to do that.

Lex Fridman 56:03 So it's including-

Jim Keller 56:04 Yeah.

Lex Fridman 56:04 ... the hardware piece.

Jim Keller 56:05 Yeah. And then the other cool thing is, for a relatively small amount of money, they did a test chip and two production chips. So it's like a super effective team, and, and unlike some AI startups where if you don't build the hardware to run the software that they really want to do, then you have to fix it by writing lots more software.

Lex Fridman 56:25 (laughs)

Jim Keller 56:26 So the hardware naturally does matrix multiply, convolution, the data manipulations, and the, uh, data movement between processing elements that, that you can see in the graph, which I think is all pretty clever. And, and that's, that's what I'm, I'm

Jim Keller 56:44 working on now.

Lex Fridman 56:45 So, uh, the... I think it's called the Grace Call Processor-

Jim Keller 56:48 Mm-hmm.

Lex Fridman 56:49 ... uh, introduced last year. It's, uh, you know, there's a bunch of measures of performance, we were talking about-

Jim Keller 56:54 Mm-hmm.

Lex Fridman 56:54 ... horses.

Jim Keller 56:55 Mm-hmm.

Lex Fridman 56:55 It seems to outperform 368 trillion operations per second, seems to out- outperformed NVIDIA's Tesla T4 system.

Jim Keller 57:02 Mm-hmm.

Lex Fridman 57:03 So these are just numbers.

Jim Keller 57:04 Mm-hmm.

Lex Fridman 57:04 What do they actually mean in real world perform- like what are the metrics for you that you're chasing? In, in your horse race, like what do you care about?

Jim Keller 57:13 Well, first the... So the, the native language of, you know, people who write AI network programs is PyTorch now, PyTorch, TensorFlow, there's a couple others. So-

Lex Fridman 57:24 Do you think PyTorch is won over TensorFlow, or is it just a-

Jim Keller 57:26 I'm not an expert on that.

Lex Fridman 57:27 Okay.

Jim Keller 57:28 I, I know many people who have switched from TensorFlow to PyTorch.

Lex Fridman 57:31 Yeah.

Jim Keller 57:31 And there's technical reasons for it, and openess-

Lex Fridman 57:33 I use both. Both are still awesome.

Jim Keller 57:35 Both are still awesome.

Lex Fridman 57:37 But the deepest love is for PyTorch currently.

Jim Keller 57:39 Yeah. There, there's more love for that. And that, that may change. So the first thing is when they write their programs, can the hardware execute it pretty much as it was written?

Lex Fridman 57:49 Mm-hmm.

Jim Keller 57:50 Right? So PyTorch turns into a graph, we have a graph compiler that makes that graph, then, like, it fractions the graph down, so if you have big matrix multiply, we turn it into right size chunks to run on the processing elements. It hooks all the graph up, it lays out all the data. There's a couple of mid-level in- representations of it that are also simulatable so that if you're writing the code you can see how it's gonna go through the machine, which is pretty cool. And then at the bottom it schedules kernels, like m- math, data manipulation, data movement kernels, which do this stuff. So we don't have to run, write a little program to do matrix multiply, 'cause we have a big matrix multiplier. Like there's no SIMD program for that. But, uh, there is scheduling for that, right? So the, the... One of the goals is, if you write a piece of PyTorch code that looks pretty reasonable, you should be able to compile it and run it on the hardware without having to tweak it and, and do all kinds of crazy things to get performance.

Lex Fridman 58:48 There's not a lot of intermediate steps.

Jim Keller 58:50 Right.

Lex Fridman 58:50 It's running directly as written.

Jim Keller 58:51 Like on a GPU, if you write a large matrix multiply naively, you'll get 5 to 10% of the peak performance of the GPU.

Lex Fridman 58:58 Hmm.

Jim Keller 58:58 Right? And then there's a bunch, there's a bunch of people who publish papers on this, and I read 'em about what steps do you have to do? And it goes from pretty reasonable, well, transpose one of the matrices, so you do-

Lex Fridman 59:09 Yeah.

Jim Keller 59:09 ... row order, not column order, you know, (clears throat) block it so that you can put a block of the matrix on different SMs, you know, groups of threads. But some of it gets into the little details, like you have to schedule it just so, so you don't have register conflicts. So the, the, the, the... They call 'em CUDA ninjas.

Lex Fridman 59:29 CUDA ninjas (laughs) . I love it.

Jim Keller 59:31 To get to the optimal point, you either write a pre-

Lex Fridman 59:33 Yeah.

Jim Keller 59:33 ... use a pre-written library, which is a good strategy for some things, or you have to be an expert in micro-architecture to program it. And so-

Lex Fridman 59:42 Right, so the optimization step is way more complicated with the GPU.

Jim Keller 59:45 So our, our goal is if you write PyTorch that's good PyTorch, you can do it. Now there's... As the networks are evolving, you know, they've changed from convolutional to matrix multiply, you know, people are talking about conditional graphs, they're talking about very large matrices, they're talking about sparsity, they're talking about problems that scale across many, many chips. So the, the native, you know, data item is a, is a packet. Like, so you send a packet to a processor, it gets processed, it does a bunch of work, and then it may send packets to other processors, and they, and they execute in, like, a data flow graph kind of methodology.

Lex Fridman 1:00:22 Got it.

Jim Keller 1:00:22 We have a big network on chip, and then 16... The next second chip has 16 ethernet ports to hook lots of them together. And it's the same graph compiler across multiple chips.

Lex Fridman 1:00:32 So that's where the scale comes in.

Jim Keller 1:00:33 So it's built to scale naturally. Now my experience with scaling is as you scale, you run into lots of interesting problems.

Lex Fridman 1:00:40 Yes.

Jim Keller 1:00:40 So scaling is a mountain to climb.

Lex Fridman 1:00:43 Yeah.

Jim Keller 1:00:43 So the hardware is built to do this, and then we're in the process of...

Lex Fridman 1:00:47 Is there a software part to this?

Jim Keller 1:00:49 Of course.

Lex Fridman 1:00:49 With, with ethernet and all that?

Jim Keller 1:00:51 Well, the, you know, the protocol at the bottom, you know, we send... You know, we, it's an ethernet PHY, but the protocol basically says, "Send the packet from here to there." It's all point-to-point.

Lex Fridman 1:01:02 Mm-hmm.

Jim Keller 1:01:03 The header bit says which processor to send it to. And we basically take a packet off our on-chip network, put an ethernet header on it, send it to the other end, strip the header off, and send it

Narrator 1:01:14 (laughs)

Jim Keller 1:01:14 ... to the local thing. It's pretty straightforward.

Lex Fridman 1:01:16 Human-to-human interaction is pretty straightforward too, but when you get a million of us, we could, we'd do-

Jim Keller 1:01:20 Okay, right.

Lex Fridman 1:01:20 ... some crazy stuff together.

Jim Keller 1:01:21 Yeah, could be fun.

Lex Fridman 1:01:22 (laughs) So, is that the goal, is scale? So, like for example, I have been recently doing a bunch of robots at home for my own-

Jim Keller 1:01:30 Mm-hmm.

Lex Fridman 1:01:30 ... personal pleasure. Uh, am I going to ever use Tenstorm, or is this more for-

Jim Keller 1:01:35 There's all kinds of problems. Like there's-

Lex Fridman 1:01:37 Mm-hmm.

Jim Keller 1:01:37 ... small inference problems, or small training problems, or big training problems.

Lex Fridman 1:01:41 What's the big goal? Is it the big inferen- uh, training problems, or the small training problems? What are some goals?

Jim Keller 1:01:46 Well, one of the goals is to scale from 100 milliwatts to a, you know, to a megawatt, you know, so like, really have some range on the problems. And the same kind of AI programs work at all different levels. So that's a goal.

Lex Fridman 1:02:00 Yes.

Jim Keller 1:02:00 The natural... Since the natural data item is a packet that we can move around, it's built to scale, but so many people have, you know, small problems.

Lex Fridman 1:02:11 Right.

Jim Keller 1:02:11 Right. But, but, uh, you know-

Lex Fridman 1:02:13 Like, like inside that phone is a small problem to solve, so do you see Tenstorm potentially being inside a phone?

Jim Keller 1:02:20 Well, th- the power efficiency of local memory, local computation in the way we built it is pretty good. And then there's a lot of efficiency on being able to do conditional graphs and sparsity.

Lex Fridman 1:02:31 Mm-hmm.

Jim Keller 1:02:31 I think it's, it's, it... For complicated networks that want to go into small factor, it's going to be quite good. Um, but we have to prove that. That's all. That's a fun problem.

Lex Fridman 1:02:40 And that's the early days of the company, right? It's a-

Jim Keller 1:02:42 Yeah.

Lex Fridman 1:02:42 ... couple years, you said?

Jim Keller 1:02:43 Mm-hmm.

Lex Fridman 1:02:44 But you think... You invested, you think they're legit-

Jim Keller 1:02:47 Yeah.

Lex Fridman 1:02:47 ... hence you joined.

Jim Keller 1:02:48 Yeah, I did.

Lex Fridman 1:02:48 Well, that's... (exhales deeply)

Jim Keller 1:02:50 Well, it's also, it's a really interesting place to be, like-

Lex Fridman 1:02:53 Right, absolutely.

Jim Keller 1:02:53 ... the AI world is exploding, you know, and I looked at some other opportunities, like build a faster processor, which people want-

Lex Fridman 1:03:01 Yes.

Jim Keller 1:03:01 ... but that's more on an incremental path than what's going to happen in AI in the next 10 years.

Lex Fridman 1:03:07 Yeah.

Jim Keller 1:03:07 So this is kind of, you know, an exciting place to be part of.

Lex Fridman 1:03:12 Yeah, the revolutions are, will be happening in the very space at Tenstorm is-

Jim Keller 1:03:15 And then lots of people are working on it, but there's lots of technical reasons why some of them, you know, aren't going to work out that well. And, and, you know, that's, that's interesting. And there's also the same problem about getting the basics right. Like, we've talked to customers about exciting features, and at some point ar- we realized that, Lishai and I were realizing they want to hear first about memory bandwidth, local bandwidth, compute intensity, programmability. They want to know the basics, power management, how the network ports work, what are the basics, do all the basics work? 'Cause it's easy to say, "We've got this great idea," you know, to crack GPT-3.

Lex Fridman 1:03:50 Mm-hmm.

Jim Keller 1:03:51 But the, the people we talk to want to say, "If I buy the c-" So we have a PCI Express card-

Lex Fridman 1:03:56 Mm-hmm.

Jim Keller 1:03:57 ... with our chip on it. If you buy the card, you plug it in your machine, you download the driver, how long does it take me to get my network to run?

Lex Fridman 1:04:04 (laughs) Right.

Jim Keller 1:04:05 Right. You know, that's a real question, so-

Lex Fridman 1:04:06 It's a very basic question.

Jim Keller 1:04:08 So-

Lex Fridman 1:04:09 Yeah. Is there an answer to that yet, or is it trying to get to that-

Jim Keller 1:04:11 Our goal is, like, an hour.

Lex Fridman 1:04:13 Okay. When can I buy a Tenstorm?

Jim Keller 1:04:16 Uh-

Lex Fridman 1:04:16 Um-

Jim Keller 1:04:16 Pretty soon.

Lex Fridman 1:04:17 ... for my, for the small case training and then-

Jim Keller 1:04:19 Yeah, pretty soon. Months.

Lex Fridman 1:04:22 Good. I love the idea of you inside a room with, uh, Karpathy, Andraj Karpathy and Chris Lattner.

Jim Keller 1:04:29 Mm-hmm.

Lex Fridman 1:04:29 Uh, v- very, um, very interesting, very brilliant people, very out of the box thinkers-

Jim Keller 1:04:37 Mm-hmm.

Lex Fridman 1:04:37 ... but also, like, first principles thinkers.

Jim Keller 1:04:40 Well, they both get stuff done. They only get stuff done to get their own projects done. They, they talk about it clearly, they educate large numbers of people, and they've created platforms for other people to go do their stuff on.

Lex Fridman 1:04:52 Yeah, the, the clear thinking that's able to be communicated-

Jim Keller 1:04:55 Yeah.

Lex Fridman 1:04:55 ... is kind of i- impressive.

Jim Keller 1:04:57 It's kind of remarkable, the... Yeah, I'm a fan.

Lex Fridman 1:05:00 Well, l- l- let me ask, 'cause, um, I, I talk to Chris actually a lot these days.

Jim Keller 1:05:05 Mm-hmm.

Lex Fridman 1:05:05 He's been, uh... One, one of the c- just to give him a shout out in, in this-

Jim Keller 1:05:08 Mm-hmm.

Lex Fridman 1:05:09 He's been so supportive as a human being. So everybody's quite different, like great engineers are different, but he's been, like, sensitive to the human element-

Jim Keller 1:05:20 Mm-hmm.

Lex Fridman 1:05:20 ... in a way that's been fascinating. Like, he was one of the early people on, on this stupid podcast that I do to say, like-

Jim Keller 1:05:27 Yeah.

Lex Fridman 1:05:28 ... "Don't quit this thing."

Jim Keller 1:05:29 Mm-hmm.

Lex Fridman 1:05:29 And also, "Talk to whoever the hell you want to talk to."

Jim Keller 1:05:33 Mm-hmm.

Lex Fridman 1:05:34 That kind of, from a legit engineer, to get, like, props-

Jim Keller 1:05:38 Mm-hmm.

Lex Fridman 1:05:38 ... and be like, "You can do this."

Jim Keller 1:05:39 Mm-hmm.

Lex Fridman 1:05:39 That was, I mean, that's what-

Jim Keller 1:05:41 That's good.

Lex Fridman 1:05:41 ... a good leader does, right? Is they just kinda-

Jim Keller 1:05:43 Mm-hmm.

Lex Fridman 1:05:43 ... let a little kid do his thing, like, "Go, go do it. Let's see, let's see, see what turns out." That, that's a, that's a pretty powerful thing. But what do you, um... What's your sense about... He used to be, he now, I think, stepped away from Google, right?

Jim Keller 1:05:57 Mm-hmm.

Lex Fridman 1:05:58 He's at PsiFi now, I think.

Jim Keller 1:06:01 (laughs)

Lex Fridman 1:06:01 Uh, what, what's really impressive to you about the things that Chris has worked on? 'Cause it's, uh... We mentioned the optimization, the compiler design stuff, the LLVM. Uh, then there's... He's also, at Google, worked at the TPU stuff.

Jim Keller 1:06:15 Mm-hmm.

Lex Fridman 1:06:15 Uh, he's obviously w- worked on Swift, so the programming language side. Talking about people that work in the entirety of the stack.

Jim Keller 1:06:23 Yeah, yeah.

Lex Fridman 1:06:24 Uh, what, uh... From your time interacting with Chris and knowing the guy, what, what's really impressive to you, that just inspires you?

Jim Keller 1:06:32 Well, well, like LLVM became, you know, the plat- the de facto platform for, you know, compilers. Like, it's, it's amazing. And, you know, it was good code quality, good design choices. He hit the right level of abstraction. There's a little bit of the, the right time and the right place.... and then he built a new programming language called Swift, which, you know, after, you know, let's say some adoption resistance, became very successful. I don't know that much about his work at Google, although I know that, you know, that was a typical... They started the TensorFlow stuff and they, you know, it was new, it was... You know, they, they wrote a lot of code and then at some point it needed to be refactored to be, you know... Because it, its development slowed down while Py- PyTorch started a little later and then passed it. So, he did a lot of work on that and then his idea about MLIR, which is... what people started to realize is the complexity of the software stack above the low level IR was getting so high that forcing the features of that into a lower level was, was putting too much of a burden on it. So, he's splitting that into multiple pieces, and that was one of the inspirations for our software stack, where we have several intermediate representations that are all executable, and you can look at them and do transformations on them before you, you lower the level. So that was... I think we started before MLIR really got, you know, far enough along to use.

Lex Fridman 1:08:01 Mm-hmm.

Jim Keller 1:08:01 Uh, but we're interested in that.

Lex Fridman 1:08:02 He's really excited about MLIR

Narrator 1:08:04 Yeah.

Lex Fridman 1:08:04 That's (laughs) that's his, like, little baby, so he, he-

Jim Keller 1:08:07 Yeah.

Lex Fridman 1:08:08 And th- there seems to be some profound ideas on that that are really useful.

Jim Keller 1:08:11 So, so each one of those things has been... As the world of software gets more and more complicated, how do we create the right abstraction levels to simplify it in a way that people can now work independently on different levels of it? So I would say all, all three of those projects, LLVM, Swift and, uh, MLIR did that successfully. So I'm interested what, what he's going to do next in the same kind of way.

Lex Fridman 1:08:34 Yes.

Jim Keller 1:08:35 So.

Lex Fridman 1:08:36 On the, either the TPU or maybe the NVIDIA GPU side, how does Tenstorrent, you think, or the ideas underlying it... It doesn't have to be Tenstorrent, just this kind of graph-focused, uh, graph-centric hardware, deep learning-centric hardware beat NVIDIA's? Do, do you think it's possible for it to basically overtake NVIDIA?

Jim Keller 1:09:02 Sure.

Lex Fridman 1:09:03 What's, what's that process look like? What's that, uh, journey look like, you think?

Jim Keller 1:09:08 Well, GPUs were built to run shader programs on millions of pixels, not to run graphs.

Lex Fridman 1:09:13 Yes.

Jim Keller 1:09:14 So there's a hypothesis that says the way the graphs, you know, are built is going to be really interesting to, to be inefficient on computing this. And then the, the primitives is not as simple program, it's matrix multiply convolution, and then the data manipulations are, are fairly extensive about... Like how do you do a fast transpose with a program? I don't know if you've ever written a transpose program. They're ugly and slow, but in hardware you can do really well. Like, I'll give you an example. So when GPU accelerators first started doing triangles, like if you have a triangle which maps on the set of pixels.

Lex Fridman 1:09:50 Mm-hmm.

Jim Keller 1:09:51 So you build... It's very easy, straightforward to build a hardware engine that'll find all those pixels.

Lex Fridman 1:09:55 Mm-hmm.

Jim Keller 1:09:55 And it's kind of weird because you walk along the triangle till you get to the edge, and then you have to go back down to the next row and walk along, and then you have to decide on the edge if the line of the triangle is like half on the pixel.

Lex Fridman 1:10:07 Mm-hmm.

Jim Keller 1:10:08 What's the pixel color? Because it's half of this pixel and half the next one. That's called rasterization.

Lex Fridman 1:10:13 Y- 'Cause... Y- you're saying that can be done in, uh, in hardware

Jim Keller 1:10:15 messages? No, I'm just... That's an example of that operation as a software program is really bad. I've written a program that did rasterization. The hardware that does it has actually less code than the software program that does it, and it's way faster, right? So there are certain times when the abstraction you have, rasterize a triangle-

Lex Fridman 1:10:36 Mm-hmm.

Jim Keller 1:10:37 ... you know, execute a graph, you know, components of a graph, the, the right thing to do in the hardware-software boundary is for the hardware to naturally do it.

Lex Fridman 1:10:45 So the GPU is really optimized for the rasterization of triangles? (laughs)

Jim Keller 1:10:50 Well, no, that's just... Well, like in a modern, you know... That's a small piece of modern GPUs.

Lex Fridman 1:10:56 Mm-hmm.

Jim Keller 1:10:57 What they did is... That... They still rasterize triangles when you're running the game, but for the most part, most of the computation area of the GPU is running shader programs, but they're single threaded programs on pixels, not graphs.

Lex Fridman 1:11:09 I have to be honest and say I don't actually know the, the math behind shader, uh, sh- shading and lighting and all that kind of stuff. I don't know what...

Jim Keller 1:11:17 They look like little simple floating point programs, or complicated ones. You can have 8,000 instructions in a shader program.

Lex Fridman 1:11:23 But I, I don't have a good intuition why it could be parallelized so easily.

Jim Keller 1:11:28 No, it's because you have eight million pixels in every single... So when you have a light, right?

Lex Fridman 1:11:32 Yeah.

Jim Keller 1:11:32 That comes down, the angle... You know, the amount of light... Like, like say this is a line of pixels across this table, right? The amount of light on each pixel is subtly different, right?

Lex Fridman 1:11:43 And each pixel is responsible for figuring out what

Jim Keller 1:11:45 ... figuring out. So that pixel says, "I'm this pixel. I know the angle of the light, I know the occlusion, I know the color I am."

Lex Fridman 1:11:51 Mm-hmm.

Jim Keller 1:11:52 Like every single pixel here is a different color, every single pixel gets a different amount of light, every single pixel has a subtly different translucency. So to make it look realistic, the solution was you run a separate program on every pixel.

Lex Fridman 1:12:05 See, but I thought there was like reflection-

Jim Keller 1:12:06 Wh- which is true.

Lex Fridman 1:12:06 ... from all over the place. Is every pixel

Jim Keller 1:12:08 Yeah, but there is. So, so you build a reflection map, which also has some pixelated thing.

Lex Fridman 1:12:13 Mm-hmm.

Jim Keller 1:12:14 And then when the pixel is looking at the reflection, the map has to calculate what the normal of the surface is.

Lex Fridman 1:12:18 Mm-hmm.

Jim Keller 1:12:19 And it does it per pixel. By the way, there's boatloads of hacks on that. You know, like you may have a lower resolution light map, you know, reflection map. There's all these, you know, hacks they do. But at the end of the day, it's per pixel computation.

Lex Fridman 1:12:32 And it so happened that you can map, uh, graph-like computation onto the, this pixel-centric computation?

Jim Keller 1:12:39 You could do floating point programs on convolution of the matrices.

Jim Keller 1:12:43 And, and NVIDIA invested for years in CUDA, first for HPC and then they got lucky with the AI trend.

Lex Fridman 1:12:50 But do you think they're going to essentially not be able to hardcore pivot out of their...

Jim Keller 1:12:55 We'll see.That's always interesting. How often do big companies hardcore pivot? Occasionally.

Lex Fridman 1:13:03 How much do you know about NVIDIA, folks?

Jim Keller 1:13:06 Some.

Lex Fridman 1:13:07 Some?

Jim Keller 1:13:07 Yeah.

Lex Fridman 1:13:08 Well, it's, um, I'm, I'm curious as well, who's ultimately... Is, uh-

Jim Keller 1:13:11 Oh, they've, they've innovated several times, but they've also worked really hard on mobile, they worked really hard on radios, you know. You know, they're fundamentally a GPU company.

Lex Fridman 1:13:20 Well, they tried to pivot. There's an in- interesting little, uh, game and play in autonomous vehicles, right, with, or, uh, semi-autonomous, like playing with Tesla and so on and seeing that's a, dipping a toe into that kind of pivot.

Jim Keller 1:13:35 They came out with this platform, which was interesting technically.

Lex Fridman 1:13:38 Yeah.

Jim Keller 1:13:39 But it was like a 3,000-watt, you know, it was 1,000-watt, three- $3,000, you know, GPU platform.

Lex Fridman 1:13:46 I don't know if it's interesting technically. It's interesting philosophically. I, I, technically, I don't know if it's the execution, the craftsmanship was there. I'm not sure. But I, I didn't get a sense-

Jim Keller 1:13:55 I think they were repurposing GPUs for an automotive solution.

Lex Fridman 1:13:59 Right, it's not a real pivot.

Jim Keller 1:14:00 They didn't, they didn't build a ground-up solution.

Lex Fridman 1:14:03 Right.

Jim Keller 1:14:03 Like the, the, like the chips inside Tesla are pretty cheap, like Mobileye's been doing this. They're, they're doing the classic work from the simplest thing.

Lex Fridman 1:14:10 Yeah.

Jim Keller 1:14:11 You know, they were building 40-mil- square-millimeter chips. In NVIDIA, their solution had two 800-millimeter chips and two 200-millimeter chips, and you know, like boatloads of really expensive DRAMs. And, and, you know, it's a really different approach.

Lex Fridman 1:14:27 And-

Jim Keller 1:14:27 So Mobileye fit the, let's say, automotive cost and form factor, and then they added features as it was economically viable and NVIDIA said, "Take the biggest thing, and l- we're gonna go make it work," you know. And, and that's also influenced, like Waymo, there's a whole bunch of autonomous startups where they have a 5,000-watt server in their trunk.

Lex Fridman 1:14:46 Mm-hmm.

Jim Keller 1:14:46 Right? And, but that's, that's 'cause they think, "Well, 5,000 watts and, you know, $10,000 is okay, 'cause it's replacing a driver." Elon's approach was, "That board has to be cheap enough to put it in every single Tesla, whether they turn on a- autonomous driving or not." Which, and Mobileye was like, "We need to fit in the BOM and, you know, cost structure that car companies do," so they may sell you a GPS for 1,500 bucks, but the BOM for that's like $25.

Lex Fridman 1:15:16 Well, and, uh, for Mobileye, it seems like neural networks were not first-class citizens, like the computation. They didn't start out as a-

Jim Keller 1:15:24 Yeah, it was a CV problem, you know, they-

Lex Fridman 1:15:26 Yeah. And-

Jim Keller 1:15:27 ... they did classic CV and found stoplights and lines, and they were really good at it.

Lex Fridman 1:15:31 Yeah, and they never, I mean, I don't know what's happening now, but they never fully pivoted. I mean, it's like, it's the NVIDIA thing. And then, as opposed to-

Jim Keller 1:15:39 Yeah.

Lex Fridman 1:15:39 So if you look at the new Tesla work, it's like neural networks from the ground up.

Jim Keller 1:15:44 Yep.

Lex Fridman 1:15:44 Right?

Jim Keller 1:15:45 Yeah, and even Tesla started with a lot of CV stuff in it, and Andrej's-

Lex Fridman 1:15:48 Yeah.

Jim Keller 1:15:48 ... basically been eliminating i- uh, you know, move, move everything into the network.

Lex Fridman 1:15:54 So, uh, without, this isn't like confidential stuff, but you sitting on a porch looking over the world, looking at the work that Andrej is doing, that Elon's doing with Tesla Autopilot, uh, do you like the trajectory of where things are going on the hardware side?

Jim Keller 1:16:09 Well, they're, they're making-

Lex Fridman 1:16:09 So-

Jim Keller 1:16:09 ... serious progress. I like the videos of people driving the beta stuff. Like, it's taken some pretty complicated intersections and all that, but it's, it's still an intervention per drive. I mean, I, I have autopilot, the current autopilot in my, my Tesla. I use it every day.

Lex Fridman 1:16:24 Do you have full self-driving beta or no?

Jim Keller 1:16:26 No.

Lex Fridman 1:16:26 So you like where this is going

Narrator 1:16:28 Yes.

Jim Keller 1:16:28 They're making progress. It's taking longer than anybody thought. You know, my, my wonder was, is, you know, hardware three, is it enough computing, off by two, off by five-

Lex Fridman 1:16:40 Wow.

Jim Keller 1:16:40 ... off by 10, off by 100?

Lex Fridman 1:16:42 Yeah.

Jim Keller 1:16:43 And, and I, I thought it probably wasn't enough, but they're doing pretty well with it now.

Lex Fridman 1:16:49 Yeah.

Jim Keller 1:16:50 And one thing is, the dataset gets bigger, the training gets better. And then there's this interesting thing is you sort of train and build an arbitrary-sized network that solves a problem. And then you refactor the network down to the thing that you can afford to, to ship.

Lex Fridman 1:17:05 Mm-hmm.

Jim Keller 1:17:06 Right? So the, the goal isn't to, to build a network that fits in the phone. It's to build something that actually works. And then, then how do you make that most effective on the hardware you have? And they seem to be doing that much better than a couple years ago.

Lex Fridman 1:17:23 Well, the one really important thing is also what they're doing well is how to iterate that quickly, which means like it's not just about one-time deployment, one building. It's constantly iterating the network and trying to automate as many steps as possible, right?

Jim Keller 1:17:36 Yeah.

Lex Fridman 1:17:37 And that's actually the principles of the Software 2.0, like you mentioned with Andrej, is, uh, it's not just... I mean, I don't know what the actual, his description of Software 2.0 is, if it's just high-level philosophical or there are specifics, but the interesting thing about what that actually looks in the real world is it's that, uh, what I think Andrej calls a data engine. It's like, it's the iterative improvement of the thing.

Jim Keller 1:18:06 Mm-hmm. Yeah.

Lex Fridman 1:18:06 You have a neural network that, uh, does stuff, fails on a bunch of things, and learns from it over and over and over. So you're constantly discovering edge cases.

Jim Keller 1:18:15 Mm-hmm.

Lex Fridman 1:18:15 So it's very much about, uh, like data engineering, like figuring out... It's, it's, it's kind of what you were talking about with Tenstorrent, is you have the data landscape, and you have to walk along that data landscape in a way that, uh, that's constantly improving the, the, the neural network, and that, that feels like that's the central piece that they can't solve.

Jim Keller 1:18:35 Yeah, so, and there's two pieces of it. Like, you, you find edge cases that don't work, and then you define something that goes get your data for that.

Lex Fridman 1:18:43 Mm-hmm.

Jim Keller 1:18:44 But then the other constraint is whether you have to label it or not. Like the, the, the amazing thing about like the GPT-3 stuff is it's