Chris Lattner: Compilers, LLVM, Swift, TPU, and ML Accelerators | Lex Fridman Podcast #21

1. 0:00 – 1:10:40

   Intro

   1. LFLex Fridman0:00

      The following is a conversation with Chris Lattner. Currently, he's a senior director at Google working on several projects, including CPU, GPU, TPU accelerators for TensorFlow, Swift for TensorFlow, and all kinds of machine learning compiler magic going on behind the scenes. He's one of the top experts in the world on compiler technologies, which means he deeply understands the intricacies of how hardware and software come together to create efficient code. He created the LLVM Compiler Infrastructure Project and the Clang compiler. He led major engineering efforts at Apple, including the creation of the Swift programming language. He also briefly spent time at Tesla as vice president of Autopilot Software during the transition from Autopilot Hardware 1.0 to Hardware 2.0, when Tesla essentially started from scratch to build an in-house software infrastructure for Autopilot. I could have easily talked to Chris for many more hours. Compiling code down across the levels of abstraction is one of the most fundamental and fascinating aspects of what computers do, and he is one of the world experts in this process. It's rigorous science, and it's messy, beautiful art. This conversation is part of The Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter @lexfridman, spelled F-R-I-D. And now here's my conversation with Chris Lattner. What was the first program you've ever written?

   2. CLChris Lattner1:33

      My first program-

   3. LFLex Fridman1:34

      Hmm, back. And when was it?

   4. CLChris Lattner1:35

      I think I started as a kid, and my parents got a BASIC programming book. And so when I started, it was typing out programs from a book, and seeing how they worked, and then typing them in wrong and trying to figure out why (laughs) they were not working right, that kinda stuff.

   5. LFLex Fridman1:53

      So BASIC. What was the first language that you remember yourself maybe falling in love with, like really connecting with?

   6. CLChris Lattner1:59

      Oh. I don't know, I mean, I feel like I've learned a lot along the way and, and each of them have a different special thing about them. So I started in BASIC and then went, like, GW-BASIC, which was the thing back in the DOS days, and then upgraded to QBasic and eventually QuickBASIC, which are all slightly more fancy versions of Microsoft BASIC. Um, made the jump to Pascal and started doing machine language programming in Assembly in Pascal, which was really cool. Turbo Pascal was amazing for its day. Um, eventually got into C, C++, and then kinda did lots of other weird things, so.

   7. LFLex Fridman2:33

      I feel like you took the dark path, which is the, uh... You coulda, you coulda gone Lisp.

   8. CLChris Lattner2:39

      Yeah, yeah.

   9. LFLex Fridman2:39

      You coulda gone higher level-

   10. CLChris Lattner2:41

       Yeah.

   11. LFLex Fridman2:41

       ... sort of functional, philosophical, hippie route.

   12. CLChris Lattner2:44

       Yeah.

   13. LFLex Fridman2:44

       Instead, you went into, like, the dark arts-

   14. CLChris Lattner2:47

       Yeah.

   15. LFLex Fridman2:47

       ... of the C++.

   16. CLChris Lattner2:47

       It was straight, straight into the machine, right?

   17. LFLex Fridman2:49

       Straight to the machine.

   18. CLChris Lattner2:50

       And so I s- so started with BASIC, Pascal, and then Assembly, and then wrote a lot of Assembly. And, um-

   19. LFLex Fridman2:56

       Why? How?

   20. CLChris Lattner2:57

       Eventually, eventually did Smalltalk and other things like that.

   21. LFLex Fridman3:00

       Yeah. (laughs)

   22. CLChris Lattner3:00

       But that was not the starting point.

   23. LFLex Fridman3:02

       But, so what, uh, what is this journey into C? Is that in high school? Is that in college?

   24. CLChris Lattner3:07

       That was in high school, yeah. So, and, and then that was, uh, it was really about trying to be able to do more powerful things than what Pascal could do and also to learn a different world. C was really confusing to me with the pointers and the syntax and everything, and it took a while. But Pascal's much more principled in various ways. C is more the, I mean, it has its historical roots, but it's, it's not as easy to learn.

   25. LFLex Fridman3:35

       With pointers, there's this memory management thing that you have to become conscious of. Is that the first time you start to understand that there's resources that you're supposed to manage?

   26. CLChris Lattner3:46

       Yeah. Uh, well, so you have that in, in Pascal as well, but in Pascal, they s- like the caret instead of the star and thing. There's some small differences like that, but it's not about, uh, pointer arithmetic. In p- in, in C, it, you end up thinking about how things get laid out in memory a lot more.

   27. LFLex Fridman4:00

       Mm-hmm.

   28. CLChris Lattner4:00

       And so in Pascal, you have allocating and deallocating and owning the, the memory, but just the programs are simpler and you don't have to... Well, for example, Pascal has a string type, and so you can think about a string instead of an array of characters which are consecutive in memory, yeah? So it's a little bit of a higher level abstraction.

   29. LFLex Fridman4:20

       So, uh, let's get into it. Let's talk about-

   30. CLChris Lattner4:23

       Yeah.
2. 1:10:40 – 1:13:11

   Dragons

   1. CLChris Lattner1:10:40

   2. LFLex Fridman1:10:40

      So last question. LLVM logo is a dragon.

   3. CLChris Lattner1:10:44

      Yeah.

   4. LFLex Fridman1:10:44

      You explain that this is because dragons have connotations of power, speed, intelligence. It can also be sleek, elegant, and modular, though you remove the, the modular part. What is your favorite dragon-related character from fiction, video, or movies?

   5. CLChris Lattner1:11:01

      So those are all very kind ways of explaining it. The... Do you wanna know the real reason it's a dragon?

   6. LFLex Fridman1:11:05

      Well, y- yeah. Yeah, let's... (laughs)

   7. CLChris Lattner1:11:07

      (laughs) Is that better? So there's a, a seminal book on compiler design called The Dragon Book. And so this is a really old now, uh, book on compilers. And so, um-

   8. LFLex Fridman1:11:17

      (laughs)

   9. CLChris Lattner1:11:18

      ... th- the, the dragon logo for LLVM came about because at Apple we kept talking about LLVM-related technologies and there was no logo to put on a slide.

   10. LFLex Fridman1:11:26

       Right. (laughs) Yeah.

   11. CLChris Lattner1:11:27

       And so we're like, "What, what do we do?" And somebody's like, "Well, what kind of logo should a compiler technology have?" And I'm like, "I don't know. I mean, the dragons are... a dragon is the best thing that, that we've got." And, you know, Apple somehow magically came up with the logo and-

   12. LFLex Fridman1:11:41

       (laughs)

   13. CLChris Lattner1:11:41

       ... and it was a great thing and the whole community rallied around it. And, uh, and then it got better as other graphic designers got involved. But that's, that's originally where it came from.

   14. LFLex Fridman1:11:49

       Good story.

   15. CLChris Lattner1:11:50

       Yeah.

   16. LFLex Fridman1:11:50

       Is there dragons from fiction that you connect with for, uh, that, uh-

   17. CLChris Lattner1:11:54

       Yeah.

   18. LFLex Fridman1:11:55

       ... Game of Thrones, Lord of the Rings, that kind of thing?

   19. CLChris Lattner1:11:58

       Uh, Lord of the Rings is great. I also like role-playing games and things like inpu- computer role-playing games. And so dragons often show up in there. But, but really it comes back to, uh, to...

   20. LFLex Fridman1:12:06

       To the book.

   21. CLChris Lattner1:12:07

       Oh no, we need, we need a thing. What do we do?

   22. LFLex Fridman1:12:09

       Yeah.

   23. CLChris Lattner1:12:10

       And, and hilariously, one of the, one of the, the funny things about LLVM is that my wife, who's amazing, runs the, the LLVM Foundation, and she goes to Grace Hopper and is trying to get more women involved in the... She's also a compiler engineer, so she's trying to get other, other women to get interested in compilers and things like this. And so she hands out the stickers and people like the LLVM sticker because of Game of Thrones.

   24. LFLex Fridman1:12:33

       (laughs)

   25. CLChris Lattner1:12:34

       And so sometimes culture has this-

   26. LFLex Fridman1:12:36

       Yeah.

   27. CLChris Lattner1:12:36

       ... helpful effect to, like, get the next generation of compiler engineers engaged f- with the cause.

   28. LFLex Fridman1:12:42

       (laughs) Okay. Awesome. Chris, thanks so much for talking to me today.

   29. CLChris Lattner1:12:44

       Yeah, it's been great talking with you.

Episode duration: 1:13:05