Chris Lattner: Future of Programming and AI | Lex Fridman Podcast #381

1. 0:00 – 2:20

   Introduction

   1. CLChris Lattner0:00

      ... on one axis, you have more hardware coming in. On the other hand, you have an explosion of innovation in AI.

   2. LFLex Fridman0:05

      Mm-hmm.

   3. CLChris Lattner0:05

      And so what happened with both Tensorflow and Pytorch is that the explosion of innovation in AI has led to... It's not just about matrix multiplication and convolution. These things have now, like, 2,000 different operators.

   4. LFLex Fridman0:16

      Mm-hmm.

   5. CLChris Lattner0:17

      And on the other hand, you have I don't know how many pieces of hardware there are out there. It's a lot. Part of my thesis, part of my belief of where computing goes if you look out 10 years from now is it's not gonna get simpler. Physics isn't going back to where we came from.

   6. LFLex Fridman0:30

      Mm-hmm.

   7. CLChris Lattner0:30

      It's only gonna get weirder from here on out, right? And so to me, the exciting part about what we're building is it's about building that universal platform, which world can continue to get weird, 'cause again, I don't think it's avoidable. It's physics. But we can help lift people, scale, do things with it, and they don't have to rewrite their code every time a new device comes out. And I think that's pretty cool.

   8. LFLex Fridman0:54

      The following is a conversation with Chris Lattner, his third time on this podcast. As I've said many times before, he's one of the most brilliant engineers in modern computing, having created LLM Compiler Infrastructure Project, the Clang compiler, the Swift programming language, a lot of key contributions to Tensorflow and TPUs as part of Google. He served as vice president of Autopilot software at Tesla, was a software innovator and leader at Apple, and now he co-created a new full stack AI infrastructure for distributed training, inference, and deployment on all kinds of hardware called Modular, and a new programming language called Mojo that is a superset of Python, giving you all the usability of Python, but with the performance of C, C++. In many cases, Mojo Code has demonstrated over 30,000X speedup over Python. If you love machine learning, if you love Python, you should definitely give Mojo a try. This programming language, this new AI framework and infrastructure, and this conversation with Chris is mind-blowing. I love it. It gets pretty technical at times, so I hope you hang on for the ride. This is the Lex Fridman Podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Chris Lattner.

2. 2:20 – 12:37

   Mojo programming language

   1. LFLex Fridman2:21

      It's been, I think, two years since we last talked, and then in that time, you somehow went and co-created a new programming language called Mojo. Uh, so it's optimized for AI. It's a superset of Python. Let's look at the big picture. What is the vision, uh, for Mojo?

   2. CLChris Lattner2:37

      For Mojo? Well, so I mean, I think you have to zoom out. So I've been working on a lot of related technologies for many, many years. So I've worked on LLVM and a lot of things in mobile and servers and things like this. But the world's changing, and what's happened with AI is we have new GPUs and new machine learning accelerators and other ASICs and things like that that make AI go real fast. At Google, I worked on TPUs. That's one of the biggest larger-scale deployed systems that exist for AI.

   3. LFLex Fridman3:06

      Mm-hmm.

   4. CLChris Lattner3:06

      And really, what you see is if you look across all of the things that are happening in the industry, there's this new compute platform coming, and it's not just about CPUs or GPUs or TPUs or NPUs or IPUs or whatever, all the PUs.

   5. LFLex Fridman3:19

      Mm-hmm.

   6. CLChris Lattner3:19

      (laughs) Right? It's about, how do we program these things, right? And so for software folks like us, right now, it doesn't do us any good if there's this amazing hardware that we can't use. And one of the things that you find out really quick is that having the theoretical capability of programming something, and then having the world's power and the innovation of all the, all the smart people in the world get unleashed on something can be quite different. And so really, where Mojo came from was starting from a problem of, we need to be able to take machine learning, take the infrastructure underneath it, and make it way more accessible, way more usable, way more understandable by normal people and researchers and other folks that are not themselves, like, experts in GPUs and things like this. And then, uh, through that journey, we realized, hey, we need syntax for this. We need to do a programming language.

   7. LFLex Fridman4:07

      Mm-hmm. So one of, one of the, the main features of the language, I say so fully in jest, is that, uh, it allows you to have the file extension to be, uh, an emoji or the fire emoji, which is one of the first emojis used as a file extension I've ever seen in my life. And then you ask yourself the question, "Why in the 21st century are we not using Unicode for file extensions?" Because, I mean, it's an epic decision. I think clearly the most important decision you made in the most... But, but you could also just use M-O-J-O as the file extension.

   8. CLChris Lattner4:42

      Also... Okay, so take-

   9. LFLex Fridman4:43

      (laughs)

   10. CLChris Lattner4:43

       ... a step back. I mean, come on, Lex. Do you think that the world's ready for this? This is a big moment in the world, right?

   11. LFLex Fridman4:47

       This is, we're releasing this onto the world. (laughs)

   12. CLChris Lattner4:49

       This is innovation. (laughs)

   13. LFLex Fridman4:51

       (laughs) I mean, it really is kinda brilliant. Emojis are such a big part of our daily lives. Why is it not in programming?

   14. CLChris Lattner5:00

       Well, and, and, like, you take a step back and look, look at what file extensions are, right? They're basically metadata, right? And so why are we spending all the screen space on them and all that stuff? Also, you know, you have them stacked up next to text files and PDF files and whatever else. Like, if you're gonna do something cool, you want it to stand out, right? And emojis are colorful. They're visual.

   15. LFLex Fridman5:18

       (laughs)

   16. CLChris Lattner5:18

       And they're, they're beautiful.

   17. LFLex Fridman5:19

       Yeah.

   18. CLChris Lattner5:19

       Right?

   19. LFLex Fridman5:19

       What's been the response so far from, uh... Is, is there a support on, like, Windows on the operating system-

   20. CLChris Lattner5:25

       Yeah.

   21. LFLex Fridman5:25

       ... in displaying, like, File Explorer?

   22. CLChris Lattner5:27

       Yeah, yeah.

   23. LFLex Fridman5:27

       Can they-

   24. CLChris Lattner5:27

       The, the one problem I've seen is that Git doesn't escape it right.

   25. LFLex Fridman5:31

       Uh-huh.

   26. CLChris Lattner5:31

       And so it thinks that the fire emoji is unprintable, and so it, like, prints out weird hex things if you use the command line Git tool.

   27. LFLex Fridman5:37

       Ah.

   28. CLChris Lattner5:37

       But everything else, as far as I'm aware, works fine, and I, I have faith that Git can be improved. So I'm not, I'm not worried.

   29. LFLex Fridman5:42

       So GitHub is fine?

   30. CLChris Lattner5:44

       GitHub is fine, yeah. GitHub is fine, Visual Studio Code, Windows, like, all the stuff, totally ready, because people have internationalization-
3. 12:37 – 21:04

   Code indentation

   1. LFLex Fridman12:37

      uh, tabs is an interesting design decision. And so you've really written a new programming language here. Yes, it is a super set of Python, but you can make a bunch of different interesting decisions here.

   2. CLChris Lattner12:48

      Totally, yeah.

   3. LFLex Fridman12:48

      And you chose actually to stick with Python as, um, in terms of some of the syntax.

   4. CLChris Lattner12:55

      Well, so, so let me explain why, right? So I mean, you can explain this in many rational ways.

   5. LFLex Fridman13:01

      Mm-hmm.

   6. CLChris Lattner13:03

      I think that the indentation is beautiful, but that's not a rational explanation, right? So, but I can defend it rationally, right? So first of all, Python 1 has millions of programmers.

   7. LFLex Fridman13:12

      Yeah.

   8. CLChris Lattner13:12

      It's huge. It's everywhere. It owns machine learning, right? So factually, it is the thing, right? Second of all, if you look at it, C code, C++ code, Java, whatever, Swift, curly brace languages also run through formatting tools and get indented.

   9. LFLex Fridman13:28

      Mm-hmm.

   10. CLChris Lattner13:29

       And so if they're not indented correctly, first of all, it will twist your brain around. (laughs) It can lead to bugs. There's notorious bugs that have happened across time where the, the indentation was wrong or misleading and it wasn't formatted right, and so it turned into an issue, right? And so what ends up happening in modern large scale code bases is people run automatic formatters. So now what you end up with is indentation and curly braces. Well, if you're gonna have, you know, the notion of grouping-

   11. LFLex Fridman13:58

       Mm-hmm.

   12. CLChris Lattner13:58

       ... why not have one thing, right? And get rid of all the clutter and have a more beautiful thing, right? Also, you look at many of these languages, it's like, okay, well, you can have curly braces or you can omit them if there's one statement, or you just like enter this entire world of complicated design space that objectively you don't need if you have Python style indentation. So...

   13. LFLex Fridman14:15

       Yeah, I would love to actually see statistics on errors made because of indentation. Like how many errors are made in Python versus in C++ that have to do with basic formatting, all that kind of stuff.

   14. CLChris Lattner14:26

       Yeah.

   15. LFLex Fridman14:26

       I would love to see.

   16. CLChris Lattner14:27

       I think it's probably pretty minor because once you get, like you use VS Code, I do too.

   17. LFLex Fridman14:32

       Mm-hmm.

   18. CLChris Lattner14:32

       So if you get VS Code set up, it does the indentation for you generally.

   19. LFLex Fridman14:34

       Yep.

   20. CLChris Lattner14:35

       Right? And so you don't, you know, it's actually really nice to not have to fight it. And then what you can see is the editor is telling you how your code will work by indenting it, which I think is pretty cool.

   21. LFLex Fridman14:45

       I honestly don't think I've ever, I don't remember having an error in Python because I indented stuff wrong.

   22. CLChris Lattner14:53

       Yeah. So, I mean, I think that there's, again, this is a religious thing, and so I can joke about it and I love, I love to kind of, you know, I realize that this is such a polarizing thing and everybody wants to argue about it. And so I like poking at the bear a little bit, right?

   23. LFLex Fridman15:06

       (laughs)

   24. CLChris Lattner15:06

       But, but frankly, right, come back to the first point, Python won.

   25. LFLex Fridman15:10

       Yeah, yeah.

   26. CLChris Lattner15:10

       Like it's huge. It's an AI. It's the right thing for us. Like we see Mojo as being an incredible part of the Python ecosystem. We're not looking to break Python or change it or "fix it".

   27. LFLex Fridman15:20

       Mm-hmm.

   28. CLChris Lattner15:21

       We love Python for what it is. Our view is that Python is just not done yet. And so if you look at, you know, you mentioned Python being slow. Well, there's a couple of different things that go into that, which we can talk about if you want. But one of them is it just doesn't have those features that you would use to do C-like programming.

   29. LFLex Fridman15:37

       Mm-hmm.

   30. CLChris Lattner15:37

       And so if you say, "Okay, well, I'm forced out of Python into C for certain use cases," well then what we're doing is we're saying, "Okay, well why, why is that? Can we just add those features that are missing from Python back up to Mojo?" And then you can have everything that's great about Python, all the things you're talking about that you love.
4. 21:04 – 30:54

   The power of autotuning

   1. CLChris Lattner21:04

      cool.

   2. LFLex Fridman21:04

      Yeah. And, and for machine learning, I think meta-programming, I think we could generally say is extremely useful. And so you, you get features, I mean, that th- I'll jump around, but there's ... the feature of auto-tuning-

   3. CLChris Lattner21:17

      Yeah.

   4. LFLex Fridman21:17

      ... and adaptive compilation just blows my mind.

   5. CLChris Lattner21:20

      Yeah. Well, so, okay, so let's come back to that.

   6. LFLex Fridman21:22

      Sure, all right.

   7. CLChris Lattner21:22

      So, so, what, what, what is, what is, what is machine learning? Like, what, or what is a machine learning model? Like, you take a PyTorch model off the internet, right?

   8. LFLex Fridman21:28

      Yeah.

   9. CLChris Lattner21:29

      Um, it's really interesting to me because what a Py- what PyTorch and what TensorFlow and all these frameworks are kind of pushing compute into is they're pushing into, like, this abstract specification of a compute problem, which then gets mapped in a whole bunch of different ways.

   10. LFLex Fridman21:42

       Mm-hmm.

   11. CLChris Lattner21:43

       Right? And so this is why it became a meta-programming problem, is that you wanna be able to say, "Cool, I have, I have this neural net. Now run it with batch size 1,000," (laughs) right? Do a, do a, do a, do a mapping across batch. Or, "Okay, I wanna take this problem, now run it across 1,000 CPUs-"

   12. LFLex Fridman21:58

       Mm-hmm.

   13. CLChris Lattner21:58

       "... or GPUs." Right? And so, like, this, this problem of, like, des- describe the compute and then map it and do things and transform it are, are, like, actually, it's very profound, and that's one of the things that makes machine learning systems really special.

   14. LFLex Fridman22:11

       Uh, maybe can you describe auto-tuning and how do you pull off ... I mean, I guess adaptive compilation is what we're talking about as meta-programming.

   15. CLChris Lattner22:19

       Yeah.

   16. LFLex Fridman22:19

       How do you pull off auto-tune? I mean, is that, is that as profound as I think it is? It just seems like a really, like, uh, you know, we'll mention list comprehensions. To me, from a quick glance at Mojo, uh, which by the way, I have to absolutely, like, dive in, uh, th- as I realize how amazing this is, I, I absolutely must dive in. Uh, it, that looks like just an incredible feature for machine learning people.

   17. CLChris Lattner22:43

       Yeah. Well, so, so what is auto-tuning? So take a step back. Auto-tuning is a feature in Mojo. It's not ... So ve- very little of what we're doing is actually research. Like, many of these ideas have existed in other systems and other places, and so what we're doing is we're pulling together good ideas, remixing them, and making them into a, hopefully a beautiful system, right? And so auto-tuning, the observation is that turns out hardware systems, algorithms are really complicated. Turns out maybe you don't actually want to know how the hardware works (laughs) , right? A lot of people don't, right? And so there are lots of really smart hardware people, I know a lot of them, uh, where they know everything about, okay, the, the cache size is this and the number of registers is that, and if you use this l- length of vector, it's gonna be super efficient because it maps directly onto what it can do, and like all this kinda stuff. Or the GPU has SMs and it has a warp size of whatever, right? All this stuff that goes into these things, where the tile size of a TPU is 128, like these, these factoids, right? My belief is that most normal people, and I love hardware people also, I'm not trying to offend literally everybody on the internet, um, but, uh, most programmers actually don't wanna know this stuff.

   18. LFLex Fridman23:51

       Mm-hmm.

   19. CLChris Lattner23:51

       Right? And so if you come at it from the perspective of how do we allow people to build both more abstracted but also more portable code, because, you know, could be that the vector length changes or the cache size changes, or it could be that the tile size of your matrix changes, or the number, you know, an A100 versus an H100 versus a Volta versus a whatever GPU have different characteristics, right? A lot of the algorithms that you run are actually the same, but the parameters, these magic numbers you have to fill in, end up being really fiddly numbers that an expert has to go figure out. And so what auto-tuning does is it says, "Okay, well, guess what? There's a lot of compute out there." Right? So instead of having humans go randomly try all the things or do a grid search or go search some complicated multidimensional space-... how about we have computers do that?

   20. LFLex Fridman24:35

       Mm-hmm.

   21. CLChris Lattner24:36

       Right? And so what auto-tuning does is you can say, "Hey, here's my algorithm." If it's a, a matrix operation or something like that, you can say, "Okay, I'm gonna carve it up into blocks. I'm gonna do those blocks in parallel, and I wanna, this, this, with 128 things that I'm running on, I wanna cut it this way or that way," or whatever. And you can say, "Hey, go see which one's actually empirically better on this system." Mm-hmm.

   22. LFLex Fridman24:57

       And then the result of that, you cache for that system.

   23. CLChris Lattner25:00

       Yep.

   24. LFLex Fridman25:00

       You save it.

   25. CLChris Lattner25:01

       And so come back to twisting your compiler brain, right? So not only does the compiler have an interpreter that's used to do meta-programming, that compiler, that interpreter, that meta-programming now has to actually take your code and go run it on a target machine.

   26. LFLex Fridman25:16

       Mm-hmm.

   27. CLChris Lattner25:16

       (laughs) See, see which one it likes the best, and then stitch it in and then keep going, right?

   28. LFLex Fridman25:21

       So part of the compilation is machine-specific.

   29. CLChris Lattner25:23

       Yeah. Well, so I mean, this is an optional feature, right? So you don't have to use it for everything.

   30. LFLex Fridman25:26

       Yes.
5. 30:54 – 47:38

   Typed programming languages

   1. CLChris Lattner30:54

   2. LFLex Fridman30:54

      Before I talk about some of the interesting stuff with parallelization, all that, let's- let- let's first talk about, like, the basics. We talked indentation, right? So this thing looks like Python. It's sexy and beautiful like Python, as we mentioned.

   3. CLChris Lattner31:07

      Yep.

   4. LFLex Fridman31:07

      Uh, is it a typed language? So what's the role of types?

   5. CLChris Lattner31:10

      Yeah. Good question. So Python has types. It has strings, it has integers, it has dictionaries, and like all that stuff, but they all live at runtime.

   6. LFLex Fridman31:19

      Mm-hmm.

   7. CLChris Lattner31:20

      Right? And so because all those types live at runtime in Python, you never s- or you don't have to spell them. (laughs) Python also has, like, this whole typing thing going on now, and a lot of people use it.

   8. LFLex Fridman31:30

      Yeah.

   9. CLChris Lattner31:30

      I'm not talking about that. That's, that's kind of a different thing. We can come back to that if you want. But- but typically, the, um, you know, you just say, "I take... I have a def, and my def takes two parameters. I'm gonna call them A and B," and I don't have to write a type. Okay? So that is great, but what that does is that forces what's called a consistent representation. So these things have to be a pointer to an object with the object header, and they all have to look the same.

   10. LFLex Fridman31:54

       Mm-hmm.

   11. CLChris Lattner31:54

       And then when you dispatch a method, you go through all the same different paths no matter what the, the receiver, whatever that type is. So what Mojo does is, is it allows you to have more than one kind of type. And so what it does is it allows you to say, "Okay, cool. I have, I have an object," and objects behave like Python does, and so it's fully dynamic, and that's all great. And for many things, classes, like, that's all very powerful and very important. But if you wanna say, "Hey, it's an integer, and it's 32 bits or 64 bits," or whatever it is, or it's a floating point value and it's 64 bits, well, then the compiler can take that, and it can use that to do way better optimization, and it turns out, again, getting rid of the indirections. That's huge. It means you can get better code completion because you have, um, 'cause the compiler knows what the type is, and so it knows what operations work on it. And so that's actually pretty huge. And so what Mojo does is ha- allows you to progressively adopt types into your program. And so you can start, again, it's compatible with Python, and so then you can add however many types you want wherever you want them. And if you don't wanna deal with it, you don't have to deal with it.

   12. LFLex Fridman32:54

       Right.

   13. CLChris Lattner32:55

       And so one of- one of, you know, our opinions on this (laughs) is that it's not that types are the right thing or the wrong thing. It's that they're a useful thing.

   14. LFLex Fridman33:05

       Well, so it's kind of optional. It's not strict typing. Like. you don't have to specify a type.

   15. CLChris Lattner33:09

       Exactly.

   16. LFLex Fridman33:10

       Okay, so it's starting from the thing that Python's kinda reaching towards right now with trying to inject types into it, but it's doing-

   17. CLChris Lattner33:17

       Yeah, with a very different approach, but yes.

   18. LFLex Fridman33:19

       Okay.

   19. CLChris Lattner33:19

       Yeah, so-

   20. LFLex Fridman33:20

       What's the- what's the different approach? I- I'm actually one of the people (sighs) that have not been using types very much in Python. So I haven't-

   21. CLChris Lattner33:27

       That's okay. Why- why did you say?

   22. LFLex Fridman33:29

       It just... Well, because I- I know the importance. It's like adults use-

   23. CLChris Lattner33:33

       (laughs)

   24. LFLex Fridman33:34

       ... strict typing. And so I'm- I refuse to grow up in that sense. It's a- it's a kind of rebellion. But I- I- I just know that, um, it probably reduces the amount of errors, even if it's for... Forget about performance improvements. It probably reduces-

   25. CLChris Lattner33:47

       Well-

   26. LFLex Fridman33:47

       ... errors when you do strict typing.

   27. CLChris Lattner33:49

       Yeah, so I mean, I think it's- it's interesting if you look at that, right? And the reason I- I'm giving you a hard time, man-

   28. LFLex Fridman33:53

       Yes.

   29. CLChris Lattner33:53

       ... is that there- there's this- this cultural norm, this pressure, this like-

   30. LFLex Fridman33:58

       Mm-hmm.
6. 47:38 – 59:56

   Immutability

   1. LFLex Fridman47:38

      Yeah. Anyway, so there's a lot of amazing features on the roadmap and those already implemented. It'd be awesome-

   2. CLChris Lattner47:43

      Yeah.

   3. LFLex Fridman47:43

      ... I could just ask you a few things.

   4. CLChris Lattner47:44

      Yeah. Go for it.

   5. LFLex Fridman47:45

      So, uh, the other performance improvement comes from immutability. So what's the, what's this var and this let thing that we got going on? What's-

   6. CLChris Lattner47:55

      Well, so-

   7. LFLex Fridman47:55

      ... what's immutability?

   8. CLChris Lattner47:57

      Yeah. So one of the things that is useful, and it's not always required but it's useful, is knowing whether something can change out from underneath you. Right? And so in Python you have a pointer to an array, right? And so you pass that pointer to an array around to things. If you pass into a function, they may take that and scroll away in some other data structure.

   9. LFLex Fridman48:17

      Mm-hmm.

   10. CLChris Lattner48:18

       And so you get your array back and you go to use it. Now somebody else is like putting stuff in your array.

   11. LFLex Fridman48:22

       Yeah.

   12. CLChris Lattner48:23

       How do you reason about that? It gets to be very complicated and leads to lots of bugs, right?

   13. LFLex Fridman48:28

       Mm-hmm.

   14. CLChris Lattner48:28

       And so one of the things that, you know, again, this is not something Mojo forces on you, but something that Mojo enables is a thing called value semantics. And what value semantics do is they take collections like arrays, like dictionaries, also tensors and strings and things like this that are much higher level and make them behave like proper values. And so it makes it look like if you pass these things around, you get a logical copy of all the data.

   15. LFLex Fridman48:54

       Mm-hmm.

   16. CLChris Lattner48:55

       And so if I pass you an array, it's your array. You can go do what you want to it. You're not gonna hurt my array.... now, that is an interesting and very powerful design prin- principle. It defines away a ton of bugs. You have to be careful to implement it in an efficient way. (laughs)

   17. LFLex Fridman49:08

       I- yes, is there a performance hit that's s- uh, significant?

   18. CLChris Lattner49:12

       Uh, generally not, if you implement it the right way.

   19. LFLex Fridman49:15

       Interesting.

   20. CLChris Lattner49:15

       But it requires a lot of very low level, uh, getting the language right bits.

   21. LFLex Fridman49:20

       I- I assumed that'd be a huge performance hit 'cause it's a really n- the benefit is really nice 'cause you don't get into the-

   22. CLChris Lattner49:25

       Absolutely. Well, well, the-

   23. LFLex Fridman49:27

       ... complex ...

   24. CLChris Lattner49:27

       ... the trick is, is you can't do, you can't do copies, so you have to provide the behavior of copying without doing the copy. (laughs)

   25. LFLex Fridman49:35

       Yeah. How do you do that? Is it ... How do you do that?

   26. CLChris Lattner49:39

       It's not magic.

   27. LFLex Fridman49:40

       Okay.

   28. CLChris Lattner49:40

       It's just ... It's- it's actually pretty cool. Well, so first, before we talk about how that works, let's talk about how it works in Python, right? So in Python, you define a person class, or maybe a person class is a bad idea. The ... You define a database class, right?

   29. LFLex Fridman49:52

       Mm-hmm.

   30. CLChris Lattner49:52

       And a database class has an array of records, something like that, right? And so what the problem is, is that if you pass in a record or a class instance into the database, it'll take ahold of that object, and then it assumes it has it.
7. 59:56 – 1:34:23

   Distributed deployment

   1. CLChris Lattner59:56

   2. LFLex Fridman59:56

      What about the, uh, the deployment and the execution across multiple machines?

   3. CLChris Lattner1:00:00

      Yeah.

   4. LFLex Fridman1:00:00

      So, uh, you write that, "The modular compute platform dynamically partitions models with billions of parameters and distributes their execution across multiple machines, uh, enabling unparalleled efficiency." By the way, the, the use of "unparalleled" in that sentence... anyway-

   5. CLChris Lattner1:00:18

      (laughs)

   6. LFLex Fridman1:00:18

      "... enabling unparalleled efficiency, scale, and reliability for the largest workloads." So how, how do you do this, um, abstraction of, um, distributed deployment o- of, of, of large models?

   7. CLChris Lattner1:00:31

      Yeah, so one, one of the really interesting, um, tensions... So there's a whole bunch of stuff that goes into that. I'll pick a random walk through it.

   8. LFLex Fridman1:00:38

      Mm-hmm.

   9. CLChris Lattner1:00:38

      Uh, if you, if you go back and replay the history of machine learning, right? I mean, the brief, the brief most recent history of machine learning, 'cause this is ver- as you know, very deep.

   10. LFLex Fridman1:00:46

       Yeah.

   11. CLChris Lattner1:00:48

       I, uh, I knew Lex when he had an AI podcast, yes. (laughs)

   12. LFLex Fridman1:00:52

       Right? Yep.

   13. CLChris Lattner1:00:54

       (laughs)

   14. LFLex Fridman1:00:54

       Yeah.

   15. CLChris Lattner1:00:54

       Um, so, uh, so if you look at just TensorFlow and PyTorch, which is pretty recent history in the big picture, right? But TensorFlow is all about graphs. PyTorch, I think, pretty unarguably ended up winning. And why did it win? Mostly because of usability.

   16. LFLex Fridman1:01:09

       Mm-hmm.

   17. CLChris Lattner1:01:09

       Right, and the usability of PyTorch is, I think, huge. And I think, again, that's a huge testament to the power of taking abstract, theoretical, technical concepts and bringing it to the masses, right? Now, the challenge with what the TensorFlow versus the PyTorch n- design points was that TensorFlow's e- kind of difficult to use for researchers, but it was actually pretty good for deployment. PyTorch is really good for researchers, it kind of's not super great for deployment, right? And so I think that we as an industry have been struggling. And if you look at what deploying a machine learning model today means is that you'll have researchers who are, I mean, wicked smart, of course, but they're wicked smart at model architecture and data and calculus (laughs) and like all, uh, like, they're wicked smart in various domains. They don't wanna know anything about the hardware or deployment or C++ or things like this, right? And so what's happened is you get people who train the model, they throw over, throw it over the fence, and then you have people that try to deploy the model. Well, every time you have a team A does X, they throw it over the fence, and team Y does some, t-team B does Y, like, you have a problem, because of course it never works the first time. And so you throw it over the fence, they figure out, "Okay, it's too slow, it won't fit, doesn't use the right operator, m- the tool crashes," whate- whatever the problem is. Then they have to throw it back over the fence. And every time you throw a thing over a fence, it takes three weeks of project managers and meetings and things like this. And so, uh, what we've seen today is that getting models into production can take weeks or months.

   18. LFLex Fridman1:02:43

       Mm-hmm.

   19. CLChris Lattner1:02:43

       Like, it's not atypical, right? I talk to lots of people, and you talk about, like, VP of software at some internet company trying to deploy a model, and they're like, "Why do I need a team of 45 people?" (laughs) Like, "I, it's so easy to train a model. Why, why can't I deploy it," right? And if you dig into this, every layer is problematic. So if you look at the language piece, I mean, this is tip of the iceberg. It's a very exciting tip of the iceberg for folks, but you've got Python on one side and C++ on the other side. Python doesn't really deploy. I mean, it can theoretically, technically, in some cases, but often a lot of production teams will want to get things out of Python because they get better performance and control and whatever else. So Mojo can help with that. If you look at serving, so you talk about gigantic models. Well, a gigantic model won't fit on one machine (laughs) , right? And so now you have this model, it's written in Python, it has to be rewritten in C++. Now it also has to be carved up so that half of it runs on one machine, half of it runs on another machine.

   20. LFLex Fridman1:03:40

       Mm-hmm.

   21. CLChris Lattner1:03:40

       Or maybe it runs on 10 machines. Well, so now suddenly the complexity is exploding, right? And the reason for this is that if you, if you look into TensorFlow, PyTorch, these systems, they weren't really designed for this world, right? They were designed for, you know, back in the day when we were n- starting and doing things, where it was a different, much simpler world. Like, you wanted to run ResNet-50 or some ancient model architecture like this, it was just a, it was a completely different world than-

   22. LFLex Fridman1:04:08

       Train on one GPU-

   23. CLChris Lattner1:04:09

       Exactly.

   24. LFLex Fridman1:04:10

       ... do inference on one GP (laughs) .

   25. CLChris Lattner1:04:11

       AlexNet, yeah, AlexNet, right? The major breakthrough.

   26. LFLex Fridman1:04:14

       (laughs)

   27. CLChris Lattner1:04:14

       And, um, and the world has changed, right? And so now the challenge is, is that TensorFlow, PyTorch, these systems, they weren't actually designed for LLMs. Like, that, that was not, that was not a thing. And so what, where TensorFlow actually has amazing power in terms of scale and deployment and things like that, and I think Google is, I mean, maybe not unmatched, but they're, like, incredible in terms of their capabilities and, and gigantic scale. Um, m-many researchers using PyTorch, right? And so PyTorch doesn't have those same capabilities. And so what Modular can do is it can help with that. Now if you take a step back and you say, like, "What is Modular doing," right? So Modular has, like, a, a bitter enemy that we're fighting against in the industry, and it's one of these things where everybody knows it, but nobody is usually willing to talk about it.

   28. LFLex Fridman1:05:00

       The bitter enemy.

   29. CLChris Lattner1:05:02

       The bitter thing that we have to destroy...

   30. LFLex Fridman1:05:04

       Yeah.

Episode duration: 3:34:03