Chris Lattner: Future of Programming and AI | Lex Fridman Podcast #381

Chris Lattner 0:00 ... on one axis, you have more hardware coming in. On the other hand, you have an explosion of innovation in AI.

Lex Fridman 0:05 Mm-hmm.

Chris Lattner 0:05 And so what happened with both Tensorflow and Pytorch is that the explosion of innovation in AI has led to... It's not just about matrix multiplication and convolution. These things have now, like, 2,000 different operators.

Lex Fridman 0:16 Mm-hmm.

Chris Lattner 0:17 And on the other hand, you have I don't know how many pieces of hardware there are out there. It's a lot. Part of my thesis, part of my belief of where computing goes if you look out 10 years from now is it's not gonna get simpler. Physics isn't going back to where we came from.

Lex Fridman 0:30 Mm-hmm.

Chris Lattner 0:30 It's only gonna get weirder from here on out, right? And so to me, the exciting part about what we're building is it's about building that universal platform, which world can continue to get weird, 'cause again, I don't think it's avoidable. It's physics. But we can help lift people, scale, do things with it, and they don't have to rewrite their code every time a new device comes out. And I think that's pretty cool.

Lex Fridman 0:54 The following is a conversation with Chris Lattner, his third time on this podcast. As I've said many times before, he's one of the most brilliant engineers in modern computing, having created LLM Compiler Infrastructure Project, the Clang compiler, the Swift programming language, a lot of key contributions to Tensorflow and TPUs as part of Google. He served as vice president of Autopilot software at Tesla, was a software innovator and leader at Apple, and now he co-created a new full stack AI infrastructure for distributed training, inference, and deployment on all kinds of hardware called Modular, and a new programming language called Mojo that is a superset of Python, giving you all the usability of Python, but with the performance of C, C++. In many cases, Mojo Code has demonstrated over 30,000X speedup over Python. If you love machine learning, if you love Python, you should definitely give Mojo a try. This programming language, this new AI framework and infrastructure, and this conversation with Chris is mind-blowing. I love it. It gets pretty technical at times, so I hope you hang on for the ride. This is the Lex Fridman Podcast. To support it, please check out our sponsors in the description. And now, dear friends, here's Chris Lattner.

Lex Fridman 2:21 It's been, I think, two years since we last talked, and then in that time, you somehow went and co-created a new programming language called Mojo. Uh, so it's optimized for AI. It's a superset of Python. Let's look at the big picture. What is the vision, uh, for Mojo?

Chris Lattner 2:37 For Mojo? Well, so I mean, I think you have to zoom out. So I've been working on a lot of related technologies for many, many years. So I've worked on LLVM and a lot of things in mobile and servers and things like this. But the world's changing, and what's happened with AI is we have new GPUs and new machine learning accelerators and other ASICs and things like that that make AI go real fast. At Google, I worked on TPUs. That's one of the biggest larger-scale deployed systems that exist for AI.

Lex Fridman 3:06 Mm-hmm.

Chris Lattner 3:06 And really, what you see is if you look across all of the things that are happening in the industry, there's this new compute platform coming, and it's not just about CPUs or GPUs or TPUs or NPUs or IPUs or whatever, all the PUs.

Lex Fridman 3:19 Mm-hmm.

Chris Lattner 3:19 (laughs) Right? It's about, how do we program these things, right? And so for software folks like us, right now, it doesn't do us any good if there's this amazing hardware that we can't use. And one of the things that you find out really quick is that having the theoretical capability of programming something, and then having the world's power and the innovation of all the, all the smart people in the world get unleashed on something can be quite different. And so really, where Mojo came from was starting from a problem of, we need to be able to take machine learning, take the infrastructure underneath it, and make it way more accessible, way more usable, way more understandable by normal people and researchers and other folks that are not themselves, like, experts in GPUs and things like this. And then, uh, through that journey, we realized, hey, we need syntax for this. We need to do a programming language.

Lex Fridman 4:07 Mm-hmm. So one of, one of the, the main features of the language, I say so fully in jest, is that, uh, it allows you to have the file extension to be, uh, an emoji or the fire emoji, which is one of the first emojis used as a file extension I've ever seen in my life. And then you ask yourself the question, "Why in the 21st century are we not using Unicode for file extensions?" Because, I mean, it's an epic decision. I think clearly the most important decision you made in the most... But, but you could also just use M-O-J-O as the file extension.

Chris Lattner 4:42 Also... Okay, so take-

Lex Fridman 4:43 (laughs)

Chris Lattner 4:43 ... a step back. I mean, come on, Lex. Do you think that the world's ready for this? This is a big moment in the world, right?

Lex Fridman 4:47 This is, we're releasing this onto the world. (laughs)

Chris Lattner 4:49 This is innovation. (laughs)

Lex Fridman 4:51 (laughs) I mean, it really is kinda brilliant. Emojis are such a big part of our daily lives. Why is it not in programming?

Chris Lattner 5:00 Well, and, and, like, you take a step back and look, look at what file extensions are, right? They're basically metadata, right? And so why are we spending all the screen space on them and all that stuff? Also, you know, you have them stacked up next to text files and PDF files and whatever else. Like, if you're gonna do something cool, you want it to stand out, right? And emojis are colorful. They're visual.

Lex Fridman 5:18 (laughs)

Chris Lattner 5:18 And they're, they're beautiful.

Lex Fridman 5:19 Yeah.

Chris Lattner 5:19 Right?

Lex Fridman 5:19 What's been the response so far from, uh... Is, is there a support on, like, Windows on the operating system-

Chris Lattner 5:25 Yeah.

Lex Fridman 5:25 ... in displaying, like, File Explorer?

Chris Lattner 5:27 Yeah, yeah.

Lex Fridman 5:27 Can they-

Chris Lattner 5:27 The, the one problem I've seen is that Git doesn't escape it right.

Lex Fridman 5:31 Uh-huh.

Chris Lattner 5:31 And so it thinks that the fire emoji is unprintable, and so it, like, prints out weird hex things if you use the command line Git tool.

Lex Fridman 5:37 Ah.

Chris Lattner 5:37 But everything else, as far as I'm aware, works fine, and I, I have faith that Git can be improved. So I'm not, I'm not worried.

Lex Fridman 5:42 So GitHub is fine?

Chris Lattner 5:44 GitHub is fine, yeah. GitHub is fine, Visual Studio Code, Windows, like, all the stuff, totally ready, because people have internationalization-

Lex Fridman 5:51 Yeah.

Chris Lattner 5:51 ... in their normal part of their paths. So this is just, like, taking the next step, right?

Lex Fridman 5:56 (laughs)

Chris Lattner 5:56 Somewhere between, "Oh, wow, that makes sense. Cool. I like new things," to...... "Oh my God, you're killing my baby." Like, "What are you talking about? This can never be. Like, I can never hand this. How am I gonna type this?" Like all these things. And so this is something where, uh, I think that the world will get there. We don't have to bet o- the whole farm on this. I think we can provide both paths.

Lex Fridman 6:16 Mm-hmm.

Chris Lattner 6:17 But I think it'll be great.

Lex Fridman 6:18 Uh, when can we have emojis as part of the code? I wonder.

Chris Lattner 6:22 Uh, yeah. So I mean, lots of languages provide that, so, um, I think that we have partial support for that. It's probably not fully done yet, but, but yeah, you can, you can do that. For example, in Swift, you can do that for sure. So a, a example we give, gave at Apple was the do- the dog cow.

Lex Fridman 6:37 Yeah.

Chris Lattner 6:37 So that's a classical Mac heritage thing, and so you use the dog and the cow emoji together, and that could be your variable name. But of course, the internet went and made pile of poop for everything.

Lex Fridman 6:46 Yeah.

Chris Lattner 6:46 So, you know-

Lex Fridman 6:47 Of course.

Chris Lattner 6:47 ... if you wanna name your function pile of poop, then you can totally go to town and see how that gets through code review.

Lex Fridman 6:52 (laughs) Okay, so, uh, let me just ask a bunch of random questions. Uh, so is Mojo primarily designed for AIs or is it a general-purpose programming language?

Chris Lattner 7:03 Yeah, good question. So it's AI first, and so AI is driving a lot of the requirements. And so, um, Modular is building and designing and driving Mojo forward, and it's not because it's an interesting project theoretically to build, it's because we need it. And so at Modular, we're really tackling the AI infrastructure landscape and the big problems in AI and the reasons that it is so difficult to use and scale and adopt and deploy, and like all these big problems in AI. And so we're coming at it from that perspective. Now, when you do that, when you start tackling these problems, you realize that the, um, solution to these problems isn't actually an AI-specific solution. And so while we're doing this, we're building Mojo to be a fully general programming language, and that means that you can, uh, obviously tackle GPUs and CPUs and like these AI things, but it's also a really great way to build NumPy and other things like that. Or, you know, just if you look at what many Python libraries are today, often they're a layer of Python for the API, and they end up being C and C++ code underneath them. That's very true in AI. That's true in lots of other domains as well. And so anytime you see this pattern, that's an opportunity for Mojo to help simplify the world and help people have one thing.

Lex Fridman 8:15 Mm-hmm. So optimize through simplification by having one thing. So you mentioned Modular. Mojo is the programming language. Modular is the whole software stack.

Chris Lattner 8:25 So just over a year ago, we started this company called Modular.

Lex Fridman 8:28 Yeah.

Chris Lattner 8:28 Yeah, what Modular is about is it's about taking AI and upleveling it into the next generation, right? And so if you take a step back, what's gone on in the last five, six, seven, eight years is that we've had things like TensorFlow and PyTorch and these other systems come in. You've used them, you know this. And what's happened is these things have grown like crazy and they get tons of users. It's in production deployment scenarios. It's being used to power so many systems. I mean, AI is all around us now, now. It used to be controversial years ago, but now it's (laughs) it's a thing. Um, but the challenge with these systems is that they haven't always been, um, thought out with current demands in mind.

Lex Fridman 9:08 Mm-hmm.

Chris Lattner 9:09 And so you think about it, when, where were LLMs eight years ago? (laughs) Well, they didn't exist, right? AI has changed so much, and a lot of what people are doing today are very different than when these systems were built. And meanwhile, the hardware side of this has gotten into a huge mess. There's tons of new chips and accelerators, and every, every big company is announcing a new chip every day it feels like. And so between that, you have like this moving system on one side, moving system on the other side, and it just turns into this gigantic mess, which makes it very difficult for people to actually use AI, particularly in production deployment scenarios.

Lex Fridman 9:42 Mm-hmm.

Chris Lattner 9:42 And so what Modular is doing is we're helping build out that software stack to help solve some of those problems, so then people can be more productive and get more AI research into production. Now, what Mojo does is it's a really, really, really important piece of that. And so that is, you know, part of that engine and part of the technology that allows us to solve these problems.

Lex Fridman 10:00 So Mojo is a programming language that allows you to do the high level programming, the low level programming, like do all kinds of programming in that spectrum that gets you closer and closer to the hardware.

Chris Lattner 10:14 So take a step back. So Lex, what do you love about Python?

Lex Fridman 10:17 Oh boy, where do I begin? Um, what is love? What do I love about Python?

Chris Lattner 10:23 You're, you're a guy who knows love. I know this.

Lex Fridman 10:25 Yes. Um, how intuitive it is, how it feels like I'm writing natural language English.

Chris Lattner 10:33 Yeah.

Lex Fridman 10:34 Uh, how when I can not just write but read other people's code, somehow I can understand it faster. It's more, uh, condensed than other languages, like ones I'm really familiar with, like C++ and C.

Chris Lattner 10:50 Mm-hmm.

Lex Fridman 10:50 Uh, there's a bunch of sexy little features.

Chris Lattner 10:53 Yeah.

Lex Fridman 10:54 Uh, we'll probably talk about some of them, but list comprehensions and stuff like this. Uh...

Chris Lattner 10:59 Well, so Py- and, and don't forget the, the entire ecosystem of all the packages, right?

Lex Fridman 11:03 Oh yeah, there's probably huge-

Chris Lattner 11:04 Because there's always something. If you wanna do anything, there's always a package.

Lex Fridman 11:07 Yeah, so it's not just, uh, the ecosystem of the packages and the ecosystem of the humans that do it. That, that, that's a really... That's an interesting dynamic.

Chris Lattner 11:17 That's huge, yeah.

Lex Fridman 11:18 Because I think something, uh, about the, the usability and the ecosystem makes the thing viral and it grows and then it's a virtuous cycle, I think.

Chris Lattner 11:27 Well, and there, there's many things that went into that. Like, so I think that ML was very good for Python, and so I think that TensorFlow and PyTorch and these systems embracing Python really took and helped Python grow. But I think that the major thing underlying it is that Python's like the universal connector.

Lex Fridman 11:43 Mm-hmm.

Chris Lattner 11:43 Right? It really helps bring together lots of different systems so you can compose them and build out larger systems without having to understand how it works. But then what is the problem with Python? (laughs)

Lex Fridman 11:53 Well, I guess you could say several things, but probably that it's slow.

Chris Lattner 11:57 I think that's usually what people complain about, right? And so slow... I mean, other people complain about tabs and spaces versus curly braces or whatever, but, I mean, th- those people are just wrong 'cause it is actually just better to use indentation.

Lex Fridman 12:11 (laughs) Wow, strong words.

Chris Lattner 12:12 (laughs)

Lex Fridman 12:12 So actually, on a small tangent, let's actually take that. Let's take all kinds of tangents.

Chris Lattner 12:17 Oh, come on, Lex. You can push me. I know, I can take it.

Lex Fridman 12:19 Design, design. Listen, I've recently left Emacs for VS Code, and-

Chris Lattner 12:24 Okay.

Lex Fridman 12:24 ... the kind of hate mail I had to receive, because on the way to doing that, I also said I've considered Vim.

Chris Lattner 12:30 Yep.

Lex Fridman 12:30 And, uh, chose not to and went with VS Code and just-

Chris Lattner 12:33 You're touching on deep religions, right?

Lex Fridman 12:35 (laughs) Anyway,

Lex Fridman 12:37 uh, tabs is an interesting design decision. And so you've really written a new programming language here. Yes, it is a super set of Python, but you can make a bunch of different interesting decisions here.

Chris Lattner 12:48 Totally, yeah.

Lex Fridman 12:48 And you chose actually to stick with Python as, um, in terms of some of the syntax.

Chris Lattner 12:55 Well, so, so let me explain why, right? So I mean, you can explain this in many rational ways.

Lex Fridman 13:01 Mm-hmm.

Chris Lattner 13:03 I think that the indentation is beautiful, but that's not a rational explanation, right? So, but I can defend it rationally, right? So first of all, Python 1 has millions of programmers.

Lex Fridman 13:12 Yeah.

Chris Lattner 13:12 It's huge. It's everywhere. It owns machine learning, right? So factually, it is the thing, right? Second of all, if you look at it, C code, C++ code, Java, whatever, Swift, curly brace languages also run through formatting tools and get indented.

Lex Fridman 13:28 Mm-hmm.

Chris Lattner 13:29 And so if they're not indented correctly, first of all, it will twist your brain around. (laughs) It can lead to bugs. There's notorious bugs that have happened across time where the, the indentation was wrong or misleading and it wasn't formatted right, and so it turned into an issue, right? And so what ends up happening in modern large scale code bases is people run automatic formatters. So now what you end up with is indentation and curly braces. Well, if you're gonna have, you know, the notion of grouping-

Lex Fridman 13:58 Mm-hmm.

Chris Lattner 13:58 ... why not have one thing, right? And get rid of all the clutter and have a more beautiful thing, right? Also, you look at many of these languages, it's like, okay, well, you can have curly braces or you can omit them if there's one statement, or you just like enter this entire world of complicated design space that objectively you don't need if you have Python style indentation. So...

Lex Fridman 14:15 Yeah, I would love to actually see statistics on errors made because of indentation. Like how many errors are made in Python versus in C++ that have to do with basic formatting, all that kind of stuff.

Chris Lattner 14:26 Yeah.

Lex Fridman 14:26 I would love to see.

Chris Lattner 14:27 I think it's probably pretty minor because once you get, like you use VS Code, I do too.

Lex Fridman 14:32 Mm-hmm.

Chris Lattner 14:32 So if you get VS Code set up, it does the indentation for you generally.

Lex Fridman 14:34 Yep.

Chris Lattner 14:35 Right? And so you don't, you know, it's actually really nice to not have to fight it. And then what you can see is the editor is telling you how your code will work by indenting it, which I think is pretty cool.

Lex Fridman 14:45 I honestly don't think I've ever, I don't remember having an error in Python because I indented stuff wrong.

Chris Lattner 14:53 Yeah. So, I mean, I think that there's, again, this is a religious thing, and so I can joke about it and I love, I love to kind of, you know, I realize that this is such a polarizing thing and everybody wants to argue about it. And so I like poking at the bear a little bit, right?

Lex Fridman 15:06 (laughs)

Chris Lattner 15:06 But, but frankly, right, come back to the first point, Python won.

Lex Fridman 15:10 Yeah, yeah.

Chris Lattner 15:10 Like it's huge. It's an AI. It's the right thing for us. Like we see Mojo as being an incredible part of the Python ecosystem. We're not looking to break Python or change it or "fix it".

Lex Fridman 15:20 Mm-hmm.

Chris Lattner 15:21 We love Python for what it is. Our view is that Python is just not done yet. And so if you look at, you know, you mentioned Python being slow. Well, there's a couple of different things that go into that, which we can talk about if you want. But one of them is it just doesn't have those features that you would use to do C-like programming.

Lex Fridman 15:37 Mm-hmm.

Chris Lattner 15:37 And so if you say, "Okay, well, I'm forced out of Python into C for certain use cases," well then what we're doing is we're saying, "Okay, well why, why is that? Can we just add those features that are missing from Python back up to Mojo?" And then you can have everything that's great about Python, all the things you're talking about that you love.

Lex Fridman 15:53 Mm-hmm.

Chris Lattner 15:54 Plus not be forced out of it when you do something a little bit more computationally intense or weird or hardwarey or whatever it is that you're doing.

Lex Fridman 16:03 Well, a million questions I want to ask, but...

Chris Lattner 16:05 Yeah.

Lex Fridman 16:05 High level again, is it compiled or is an interpreted language? So Python is just in time compilation. What's, what's Mojo?

Chris Lattner 16:14 So Mojo, a complicated answer, does all the things.

Lex Fridman 16:16 Mm-hmm.

Chris Lattner 16:16 So it's interpreted, it's JIT compiled, and it's statically compiled. (laughs) And so this is for a variety of reasons. Um, so one of the things that makes Python beautiful is that it's very dynamic.

Lex Fridman 16:29 Mm-hmm.

Chris Lattner 16:29 And because it's dynamic, one of the things they added is that it has this powerful meta programming feature. And so if you look at something like PyTorch or TensorFlow or, or um, I mean even a simple, simple use case, like you define a class that has the plus method, right?

Lex Fridman 16:45 Mm-hmm.

Chris Lattner 16:45 You can overload the dunder methods, like Dunder add, for example, and then the plus method works on your class. And so it has very nice and very expressive dynamic meta programming features. In Mojo, we want all those features come in. Like, we don't want to break Python, we want it all to work. But the problem is you can't run those super dynamic features on an embedded processor or on a GPU, right? Or, or if you could, you probably don't want to just because of the performance. And so we entered this question of saying, "Okay, how do you get the power of this dynamic meta programming into a language that has to be super efficient in specific cases?" And so what we did was we said, "Okay, well take that interpreter." Python has an interpreter in it, right?

Lex Fridman 17:28 Mm-hmm.

Chris Lattner 17:28 Take that interpreter and allow it to run at compile time.

Lex Fridman 17:30 Mm-hmm.

Chris Lattner 17:31 And so now what you get is you get compiled time meta programming.

Lex Fridman 17:34 Mm-hmm.

Chris Lattner 17:34 And so this is super interesting and super powerful because one of the big advantages you get is you get python style expressive APIs. You get the ability to have overloaded operators. And if you look at what happens inside of like PyTorch, for example, with automatic differentiation and eager mode and like all these things, they're using these really dynamic and powerful features at runtime. But we can take those features and lift them so that they run at compile time.

Lex Fridman 17:58 So you're... Because C++ has meta programming with, with templates.

Chris Lattner 18:03 Yep.

Lex Fridman 18:04 But it's really messy.

Chris Lattner 18:05 It's super messy. It's always, um, it was accidentally... I mean, different people have different interpretations. My interpretation is that it was made accidentally powerful. It was not designed to be Turing complete, for example. But that was discovered kind of along the way accidentally. Um, and so there have been a number of languages in the space, and so they usually have templates or code instantiation, code copying features of various sorts.Um, th- some more modern languages, or some more newe- newer languages, let's say, like, you know, they're fairly, um, unknown, like Zig, for example, um, says, "Okay, well, let's take all of those types that you can run at, all those things you can do at runtime, and allow them to happen at compile time." And so one of the problems with C++, I mean, th- which is, one of, one of the problems with C++ (laughs) -

Lex Fridman 18:54 There we go.

Chris Lattner 18:54 ... is-

Lex Fridman 18:55 Wrong words.

Chris Lattner 18:55 ... is-

Lex Fridman 18:55 We're gonna offend everybody today.

Chris Lattner 18:57 Oh, that's okay. I mean, everybody hates me for a variety of reasons anyways, I'm sure, right? (laughs) I've written enough-

Lex Fridman 19:02 It's just the way they show love is through their hate.

Chris Lattner 19:03 I, I have written enough C++ code to earn a little bit of grumpiness-

Lex Fridman 19:06 Yes.

Chris Lattner 19:06 ... with C++. But, um, but one of the problems with it is that the meta-programming system templates, it's just a completely different universe from the normal runtime programming world. And so if you do meta-programming and programming, it's just like a different universe, different syntax, different concepts, different stuff going on. And so, again, one of our goals with Mojo is to make things really easy to use, easy to learn, and so there's a natural stepping stone. And so as you do this, you say, "Okay, well, I have to do programming at runtime, I have to do programming at compile time, why are these different things?" (laughs)

Lex Fridman 19:41 How hard is that to pull it off? 'Cause that sounds, to me as a fan of meta-programming in C++ even, how, how hard is it to pull that off? That sounds really, really exciting 'cause you can do the same style programming at compile time and at runtime. That's really, really exciting.

Chris Lattner 19:55 Yep, yep. And so, I mean, in terms of the compiler implementation details, it's hard. (laughs) I won't, I won't be shy about that. It's super hard. It requires, uh, I mean, what Mojo has underneath the covers is a completely new approach to the design of the compiler itself.

Lex Fridman 20:09 Mm-hmm.

Chris Lattner 20:09 And so this builds on these technologies like MLIR that you mentioned, but it also includes other, like caching and other, uh, interpreters and JIT compilers and other stuff like that that run-

Lex Fridman 20:19 So you have, like, an interpreter inside the comp-

Chris Lattner 20:20 Within the compiler, yes.

Lex Fridman 20:22 Oh, man.

Chris Lattner 20:23 And so it really takes the standard model of programming languages and kind of twists it and unifies it with the runtime model.

Lex Fridman 20:32 Right.

Chris Lattner 20:32 Which I think is really cool. And to me, the value of that is that, again, many of these languages have meta-programming features, like they grow macros or something, right? Even Lisp, right?

Lex Fridman 20:41 Mm-hmm. Yes.

Chris Lattner 20:42 I, I know your roots, right?

Lex Fridman 20:43 (laughs)

Chris Lattner 20:43 Um, you know, and, and this is a powerful thing, right? And so, you know, if you go back to Lisp, one of the most powerful things about, about it is that it said that the meta-programming and the programming are the same, right? And so that made it way simpler, way more consistent, way easier to understand and reason about, and it made it more composable, so if you build a library, you can use it both at runtime and compile time, which is pretty

Chris Lattner 21:04 cool.

Lex Fridman 21:04 Yeah. And, and for machine learning, I think meta-programming, I think we could generally say is extremely useful. And so you, you get features, I mean, that th- I'll jump around, but there's ... the feature of auto-tuning-

Chris Lattner 21:17 Yeah.

Lex Fridman 21:17 ... and adaptive compilation just blows my mind.

Chris Lattner 21:20 Yeah. Well, so, okay, so let's come back to that.

Lex Fridman 21:22 Sure, all right.

Chris Lattner 21:22 So, so, what, what, what is, what is, what is machine learning? Like, what, or what is a machine learning model? Like, you take a PyTorch model off the internet, right?

Lex Fridman 21:28 Yeah.

Chris Lattner 21:29 Um, it's really interesting to me because what a Py- what PyTorch and what TensorFlow and all these frameworks are kind of pushing compute into is they're pushing into, like, this abstract specification of a compute problem, which then gets mapped in a whole bunch of different ways.

Lex Fridman 21:42 Mm-hmm.

Chris Lattner 21:43 Right? And so this is why it became a meta-programming problem, is that you wanna be able to say, "Cool, I have, I have this neural net. Now run it with batch size 1,000," (laughs) right? Do a, do a, do a, do a mapping across batch. Or, "Okay, I wanna take this problem, now run it across 1,000 CPUs-"

Lex Fridman 21:58 Mm-hmm.

Chris Lattner 21:58 "... or GPUs." Right? And so, like, this, this problem of, like, des- describe the compute and then map it and do things and transform it are, are, like, actually, it's very profound, and that's one of the things that makes machine learning systems really special.

Lex Fridman 22:11 Uh, maybe can you describe auto-tuning and how do you pull off ... I mean, I guess adaptive compilation is what we're talking about as meta-programming.

Chris Lattner 22:19 Yeah.

Lex Fridman 22:19 How do you pull off auto-tune? I mean, is that, is that as profound as I think it is? It just seems like a really, like, uh, you know, we'll mention list comprehensions. To me, from a quick glance at Mojo, uh, which by the way, I have to absolutely, like, dive in, uh, th- as I realize how amazing this is, I, I absolutely must dive in. Uh, it, that looks like just an incredible feature for machine learning people.

Chris Lattner 22:43 Yeah. Well, so, so what is auto-tuning? So take a step back. Auto-tuning is a feature in Mojo. It's not ... So ve- very little of what we're doing is actually research. Like, many of these ideas have existed in other systems and other places, and so what we're doing is we're pulling together good ideas, remixing them, and making them into a, hopefully a beautiful system, right? And so auto-tuning, the observation is that turns out hardware systems, algorithms are really complicated. Turns out maybe you don't actually want to know how the hardware works (laughs) , right? A lot of people don't, right? And so there are lots of really smart hardware people, I know a lot of them, uh, where they know everything about, okay, the, the cache size is this and the number of registers is that, and if you use this l- length of vector, it's gonna be super efficient because it maps directly onto what it can do, and like all this kinda stuff. Or the GPU has SMs and it has a warp size of whatever, right? All this stuff that goes into these things, where the tile size of a TPU is 128, like these, these factoids, right? My belief is that most normal people, and I love hardware people also, I'm not trying to offend literally everybody on the internet, um, but, uh, most programmers actually don't wanna know this stuff.

Lex Fridman 23:51 Mm-hmm.

Chris Lattner 23:51 Right? And so if you come at it from the perspective of how do we allow people to build both more abstracted but also more portable code, because, you know, could be that the vector length changes or the cache size changes, or it could be that the tile size of your matrix changes, or the number, you know, an A100 versus an H100 versus a Volta versus a whatever GPU have different characteristics, right? A lot of the algorithms that you run are actually the same, but the parameters, these magic numbers you have to fill in, end up being really fiddly numbers that an expert has to go figure out. And so what auto-tuning does is it says, "Okay, well, guess what? There's a lot of compute out there." Right? So instead of having humans go randomly try all the things or do a grid search or go search some complicated multidimensional space-... how about we have computers do that?

Lex Fridman 24:35 Mm-hmm.

Chris Lattner 24:36 Right? And so what auto-tuning does is you can say, "Hey, here's my algorithm." If it's a, a matrix operation or something like that, you can say, "Okay, I'm gonna carve it up into blocks. I'm gonna do those blocks in parallel, and I wanna, this, this, with 128 things that I'm running on, I wanna cut it this way or that way," or whatever. And you can say, "Hey, go see which one's actually empirically better on this system." Mm-hmm.

Lex Fridman 24:57 And then the result of that, you cache for that system.

Chris Lattner 25:00 Yep.

Lex Fridman 25:00 You save it.

Chris Lattner 25:01 And so come back to twisting your compiler brain, right? So not only does the compiler have an interpreter that's used to do meta-programming, that compiler, that interpreter, that meta-programming now has to actually take your code and go run it on a target machine.

Lex Fridman 25:16 Mm-hmm.

Chris Lattner 25:16 (laughs) See, see which one it likes the best, and then stitch it in and then keep going, right?

Lex Fridman 25:21 So part of the compilation is machine-specific.

Chris Lattner 25:23 Yeah. Well, so I mean, this is an optional feature, right? So you don't have to use it for everything.

Lex Fridman 25:26 Yes.

Chris Lattner 25:26 But yeah, if you, if you're ... So one of, one of the things that we're in the quest of is ultimate performance.

Lex Fridman 25:33 Yes.

Chris Lattner 25:34 Right? And ultimate performance is important for a couple of reasons, right? So if you're an enterprise, you're looking to save cost and compute and things like this, ultimate performance translates to, you know, fewer servers. Like, if you care about the environment, hey, better performance leads to more efficiency, right? I mean, you could joke and say like, you know, "Python's bad for the environment." (laughs) Right? And so if you move to Mojo, it's like at least 10X better just out of the box.

Lex Fridman 25:57 Yeah.

Chris Lattner 25:57 (laughs) And keep going, right?

Lex Fridman 25:59 That's good.

Chris Lattner 25:59 Um, uh, but, but performance is also interesting because it leads to better products.

Lex Fridman 26:03 Yeah.

Chris Lattner 26:03 And so in the space of machine learning, right, if you reduce the latency of a model so that it runs faster, so every time you query the server running the model, it takes less time, well, then the product team can go and make the model bigger.

Lex Fridman 26:15 Mm-hmm.

Chris Lattner 26:15 Well, that's actually makes it so you have a better experience as a customer. And so a lot of people care about that.

Lex Fridman 26:21 So for auto-tuning, for like tile size, you mentioned 128 for GPU, you would specify, like, a bunch of options to try?

Chris Lattner 26:27 Yeah.

Lex Fridman 26:28 Just in the code?

Chris Lattner 26:29 Yep.

Lex Fridman 26:29 Just simple statement?

Chris Lattner 26:30 Yep.

Lex Fridman 26:31 And then you can just set and forget and know-

Chris Lattner 26:33 Yeah.

Lex Fridman 26:33 ... depending what order it compiles, it'll actually be the fastest for that case?

Chris Lattner 26:36 And, and, yeah, exactly. And the beauty of this is that it helps you in a whole bunch of different ways, right? So if you're building ... So s- often what'll happen is that, you know, you've written a bunch of software yourself, right? You, you wake up one day, you say, "I have an idea, I'm gonna go code up some code." I get to work. I forget about it and move on with life. I come back six months or a year or two years or three years later, you dust it off, and you go use it again in a new environment.

Lex Fridman 26:58 Mm-hmm.

Chris Lattner 26:59 And maybe your GP is different. Maybe you're running on a server instead of a laptop. Maybe you're whatever, right?

Lex Fridman 27:04 Mm-hmm.

Chris Lattner 27:04 And so the problem now is you say, "Okay, well ... " I mean, again, not everybody cares about performance, but if you do, you say, "Okay, well, I wanna take advantage of all these new features. I don't wanna break the old thing though." Right?

Lex Fridman 27:15 Mm-hmm.

Chris Lattner 27:15 And so the typical way of handling this kind of stuff before is, you know, if you're talking about C++ templates, or you're talking about C with macros, you end up with IFDEFs. You get, like, all these weird things get layered in, make the code super complicated, and then how do you test it?

Lex Fridman 27:30 Mm-hmm.

Chris Lattner 27:30 Right? It becomes this, this crazy complexity, multidimensional space that you have to worry about. And, you know, that just doesn't scale very well.

Lex Fridman 27:39 Actually, let me just jump around before I go into some specific features.

Chris Lattner 27:42 (laughs)

Lex Fridman 27:42 Like, the increase in performance here that we're talking about-

Chris Lattner 27:45 Yeah.

Lex Fridman 27:45 ... can be just insane. Uh, you write that Mojo can provide a 35,000X speed up over Python. Uh, how does it do that?

Chris Lattner 27:57 Yeah. So, it can even do more, but, uh, we'll get to that.

Lex Fridman 28:00 (laughs)

Chris Lattner 28:00 So, uh, so first of all, when we say that we're talking about what's called CPython. It's the default Python that everybody uses. When you type Python 3, that's like-

Lex Fridman 28:10 Mm-hmm.

Chris Lattner 28:11 ... typically the one you use, right? CPython is an interpreter. And so interpreters, they have an extra layer of, like, byte codes and things like this that they have to go read, parse, interpret, and it makes them kind of slow from that perspective. And so one of the first things we do is we move to a compiler. And so just moving to a compiler, getting the interpreter out of the loop, is 2 to 5 to 10X speed up, depending on the code. So just out of the gate, just using more modern techniques, right?

Lex Fridman 28:38 Mm-hmm.

Chris Lattner 28:39 Now, if you do that, one of the things you can do is you can start to look at how CPython started to lay out data.

Lex Fridman 28:45 Mm-hmm.

Chris Lattner 28:46 And so one of the things that, that, um, CPython did, and this isn't part of the Python spec necessarily, but this is just s- sets of decisions, is that if you take an integer, for example, it'll put it in an object.

Lex Fridman 28:58 Mm-hmm.

Chris Lattner 28:58 'Cause in Python, everything's an object. And so they do the very logical thing of keeping the memory representation of all objects the same. So all objects have a header. They have, like, payload data. They ... And, and what this means is that every time you pass around an object, you're passing around a pointer to the data. Well, this has overhead, right? It turns out that modern computers don't like chasing pointers very much and things like this. It means that you have to allocate the data. It means you have to reference count it, which is another way of, that Python uses to keep track of memory. And so this has a lot of overhead.

Lex Fridman 29:29 (laughs)

Chris Lattner 29:29 And so if you say, "Okay, let's try to get that out of the heap, out of a box, out of an indirection and into the registers."

Lex Fridman 29:39 Mm-hmm. So-

Chris Lattner 29:40 That's, that's another 10X or more.

Lex Fridman 29:42 So it adds up if you, if you're reference counting every single f-

Chris Lattner 29:45 Absolutely.

Lex Fridman 29:45 ... every, every single thing you create, that adds up.

Chris Lattner 29:48 Yeah. And if you look at, you know, people complain about the Python GIL. This is one of the things that hurts parallelism. Um, that's because of the reference counting. Right? And so the GIL and reference counting are very tightly intertwined in Python. It's not the only thing, but it's very tightly intertwined. And so then you lean into this and you say, "Okay, cool. Well, modern computers, they can do more than one operation at a time." So they have vectors. What is a vector? Well, a vector allows you to take one ... Instead of taking one piece of data, doing an add or a multiply, and then picking up the next one, you can now do four or eight or 16 or 32 at a time. Right? Well, Python doesn't expose that because of reasons. And so now you can say, "Okay, well, you can adopt that." Now you have threads. Now you have, like, additional things, like you can control memory hierarchy. And so what Mojo allows you to do is it allows you to start taking advantage of all these powerful things that have been built into the hardware over time, and it gives, the library gives, um, very nice-... features, so you can say, "Just parallelize this, do this in parallel." Right? So it's very, um, very powerful weapons against (laughs) slowness, which is why people have been, uh, I think, having fun like just taking code and making it go fast, because it's just kind of an adrenaline rush to see like how fast you can get things.

Lex Fridman 30:54 Before I talk about some of the interesting stuff with parallelization, all that, let's- let- let's first talk about, like, the basics. We talked indentation, right? So this thing looks like Python. It's sexy and beautiful like Python, as we mentioned.

Chris Lattner 31:07 Yep.

Lex Fridman 31:07 Uh, is it a typed language? So what's the role of types?

Chris Lattner 31:10 Yeah. Good question. So Python has types. It has strings, it has integers, it has dictionaries, and like all that stuff, but they all live at runtime.

Lex Fridman 31:19 Mm-hmm.

Chris Lattner 31:20 Right? And so because all those types live at runtime in Python, you never s- or you don't have to spell them. (laughs) Python also has, like, this whole typing thing going on now, and a lot of people use it.

Lex Fridman 31:30 Yeah.

Chris Lattner 31:30 I'm not talking about that. That's, that's kind of a different thing. We can come back to that if you want. But- but typically, the, um, you know, you just say, "I take... I have a def, and my def takes two parameters. I'm gonna call them A and B," and I don't have to write a type. Okay? So that is great, but what that does is that forces what's called a consistent representation. So these things have to be a pointer to an object with the object header, and they all have to look the same.

Lex Fridman 31:54 Mm-hmm.

Chris Lattner 31:54 And then when you dispatch a method, you go through all the same different paths no matter what the, the receiver, whatever that type is. So what Mojo does is, is it allows you to have more than one kind of type. And so what it does is it allows you to say, "Okay, cool. I have, I have an object," and objects behave like Python does, and so it's fully dynamic, and that's all great. And for many things, classes, like, that's all very powerful and very important. But if you wanna say, "Hey, it's an integer, and it's 32 bits or 64 bits," or whatever it is, or it's a floating point value and it's 64 bits, well, then the compiler can take that, and it can use that to do way better optimization, and it turns out, again, getting rid of the indirections. That's huge. It means you can get better code completion because you have, um, 'cause the compiler knows what the type is, and so it knows what operations work on it. And so that's actually pretty huge. And so what Mojo does is ha- allows you to progressively adopt types into your program. And so you can start, again, it's compatible with Python, and so then you can add however many types you want wherever you want them. And if you don't wanna deal with it, you don't have to deal with it.

Lex Fridman 32:54 Right.

Chris Lattner 32:55 And so one of- one of, you know, our opinions on this (laughs) is that it's not that types are the right thing or the wrong thing. It's that they're a useful thing.

Lex Fridman 33:05 Well, so it's kind of optional. It's not strict typing. Like. you don't have to specify a type.

Chris Lattner 33:09 Exactly.

Lex Fridman 33:10 Okay, so it's starting from the thing that Python's kinda reaching towards right now with trying to inject types into it, but it's doing-

Chris Lattner 33:17 Yeah, with a very different approach, but yes.

Lex Fridman 33:19 Okay.

Chris Lattner 33:19 Yeah, so-

Lex Fridman 33:20 What's the- what's the different approach? I- I'm actually one of the people (sighs) that have not been using types very much in Python. So I haven't-

Chris Lattner 33:27 That's okay. Why- why did you say?

Lex Fridman 33:29 It just... Well, because I- I know the importance. It's like adults use-

Chris Lattner 33:33 (laughs)

Lex Fridman 33:34 ... strict typing. And so I'm- I refuse to grow up in that sense. It's a- it's a kind of rebellion. But I- I- I just know that, um, it probably reduces the amount of errors, even if it's for... Forget about performance improvements. It probably reduces-

Chris Lattner 33:47 Well-

Lex Fridman 33:47 ... errors when you do strict typing.

Chris Lattner 33:49 Yeah, so I mean, I think it's- it's interesting if you look at that, right? And the reason I- I'm giving you a hard time, man-

Lex Fridman 33:53 Yes.

Chris Lattner 33:53 ... is that there- there's this- this cultural norm, this pressure, this like-

Lex Fridman 33:58 Mm-hmm.

Chris Lattner 33:58 ... there has to be a right way to do things, like-

Lex Fridman 34:00 Yes.

Chris Lattner 34:00 ... you know, only- grown-ups only do it one way, and if you don't do that-

Lex Fridman 34:02 Yes.

Chris Lattner 34:02 ... you should feel bad.

Lex Fridman 34:03 Yes.

Chris Lattner 34:03 Right? Like, some people feel like Python's a guilty pleasure or something.

Lex Fridman 34:06 Mm-hmm.

Chris Lattner 34:07 And that's like, "When it gets serious, I need to go rewrite it," right?

Lex Fridman 34:09 Yeah (laughs) , exactly.

Chris Lattner 34:10 Well, I mean, cool. I understand history, and I understand kind of where this comes from, but I don't think it has to be a guilty pleasure.

Lex Fridman 34:16 Yeah (laughs) .

Chris Lattner 34:17 Right? (laughs)

Lex Fridman 34:17 Yeah.

Chris Lattner 34:17 And so if you look at that, you say, "Why do you have to rewrite it?" Well, you have to rewrite it to deploy.

Lex Fridman 34:21 Mm-hmm.

Chris Lattner 34:22 Well, why do you wanna deploy? Well, you care about performance, or you care about predictability, or you want, you know, a tiny thing on the server that has no dependencies, or, you know, you have objectives that you're trying to attain.

Lex Fridman 34:33 Mm-hmm.

Chris Lattner 34:34 So what if Python can achieve those objectives? So if you want types, well, maybe you want types because you wanna make sure you're passing on the right thing.

Lex Fridman 34:41 Mm-hmm.

Chris Lattner 34:41 Sure, you can add a type. If you don't care, you're prototyping some stuff, you're hacking some things out, you're like pulling some random code off the internet, it should just work (laughs) , right?

Lex Fridman 34:51 Mm-hmm.

Chris Lattner 34:51 And you shouldn't be, like, pressured. You shouldn't feel bad about doing the right thing or the thing that feels good. Now, if you're in a team, right? You're working at some massive internet company, and you have 400 million lines of Python code, well, they- they may have a house rule that you use types.

Lex Fridman 35:06 Yeah.

Chris Lattner 35:06 Right? Because it makes it easier for different humans to talk to each other and understand what's going on, and bugs at scale, right? And so there- there are lots of good reasons why you might wanna use types, but this doesn't mean that everybody should use them all the time, right?

Lex Fridman 35:18 Yeah.

Chris Lattner 35:18 So what Mojo does is it says, "Cool. Well, allow people to use types, and if you use types, you get nice things out of it," right? You get better performance and things like this, right? But Mojo is a full compatible superset of Python.

Lex Fridman 35:32 Mm-hmm.

Chris Lattner 35:33 Right? And so that means it has to work without types (laughs) . It has to support all the dynamic things. It has to support all the packages. It has to support, uh, uh, for comprehension, list comprehensions, and things like this, right? And so that- that- that starting point, I think, is really important. And I think that, again, you can look at why I care so much about this, and there- there's many different aspects of that, one of which is the world went through a very challenging migration from Python 2 to Python 3, right?

Lex Fridman 35:59 Yes.

Chris Lattner 35:59 And this- this migration took many years.

Lex Fridman 36:01 Okay.

Chris Lattner 36:02 And it was very painful for many teams, right?

Lex Fridman 36:03 Yeah.

Chris Lattner 36:04 And there's a lot of- a lot of things that went on in that. Um, I'm not an expert on all the details, and I honestly don't wanna be.

Lex Fridman 36:10 Mm-hmm.

Chris Lattner 36:10 I don't want the world to have to go through that (laughs) .

Lex Fridman 36:12 Yeah.

Chris Lattner 36:13 Right? And you know, people can ignore Mojo, and if it's not their thing, that's- that's cool. But if they wanna use Mojo, I don't want them to have to rewrite all their code.

Lex Fridman 36:20 Yeah, I mean, this... Okay, the superset part is- is just... I mean, there's so much brilliant stuff here that definitely is- is- is incredible. Um, uh, we'll talk about that.

Chris Lattner 36:29 Yeah, well-

Lex Fridman 36:30 But first of all, how's the typing-

Chris Lattner 36:31 Yeah.

Lex Fridman 36:31 ... implemented differently in, uh, in Python versus, uh, Mojo?

Chris Lattner 36:36 Yeah.

Lex Fridman 36:36 So th- this heterogeneous flexibility-

Chris Lattner 36:39 Yeah.

Lex Fridman 36:39 ... you said, is differently implemented.

Chris Lattner 36:41 Yeah. So I'm- I'm not a full expert (laughs) in the- in the whole backstory on types in Python. So I'll give you- I'll give you that. I can give you my understanding.

Lex Fridman 36:47 Mm-hmm.

Chris Lattner 36:48 Um, my understanding is, basically, like many dynamic languages, the ecosystem went through a phase where there-... people went from writing scripts to writing large-scale, huge codebases in Python, and at scale. It kind of helps to have types.

Lex Fridman 37:03 Yeah.

Chris Lattner 37:04 People want to be able to reason about interfaces, what, what, do you expect a string or an int, or like, what, what, these basic things, right? And so what the Python community started doing is it started saying, "Okay, let's have tools on the side, checker tools, right, that go and, like, enforce invariants, check for bugs, try to identify things." These are called static analysis tools, generally. And so these tools run over your code and try to look for bugs.

Lex Fridman 37:27 Mm-hmm.

Chris Lattner 37:28 What ended up happening is there's so many of these things, so many different weird patterns and different approaches on specifying the types and different things going on, that the Python community realized and recognized, "Hey, hey, hey, there's a thing here." (laughs)

Lex Fridman 37:38 Mm-hmm.

Chris Lattner 37:38 And so what they started to do is they started to standardize the syntax for adding types to Python. Now, one of the challenges that they had is that they're coming from kind of this fragmented world where there's lots of different tools, they have different trade-offs and interpretations and the types mean different things. And so if you look at types in Python, according to the Python spec, the types are ignored.

Lex Fridman 37:58 Mm-hmm.

Chris Lattner 37:59 Right, so according to the Python spec, you can write pretty much anything (laughs) in- in a type position, okay? And, um, uh, you can, technically, you can write any expression. Okay?

Lex Fridman 38:09 Hmm.

Chris Lattner 38:10 Now, that's beautiful because you can extend it, you can do cool things, you can write, build your own tools, you can build your own house linter or something like that, right? But it's also a problem because any existing Python program may be using different tools.

Lex Fridman 38:24 Mm-hmm.

Chris Lattner 38:24 And they have different interpretations. And so if you adopt somebody's package into your ecosystem, try to run the tool you prefer, it may throw out tons of weird errors and warnings and problems just because it's incompatible with how these things work.

Lex Fridman 38:35 Mm-hmm.

Chris Lattner 38:35 Also because they're added late and they're not checked by the Python interpreter, it's always kind of more of a hint than it is a requirement.

Lex Fridman 38:41 Mm-hmm.

Chris Lattner 38:42 Also, uh, the CPython implementation can't use them for performance, and so it's really-

Lex Fridman 38:47 Well, I mean, that's a big one, right? So is, you can't utilize the, for the compilation, for the just-in-time compilation.

Chris Lattner 38:52 Yep.

Lex Fridman 38:52 Okay.

Chris Lattner 38:52 Yup, exactly. And this, this all comes back to the design principle of it's, it's kind, they're kind of hints. They're kind of the definition's a little bit murky. It's unclear exactly the interpretation in a bunch of cases. And so because of that, you can't actually, um, even if you want to, it's really difficult to use them to say like, "It is going to be an int, and if it's not, it's a problem."

Lex Fridman 39:10 Mm-hmm.

Chris Lattner 39:10 Right? A lot of code would break if you did that. So, so in Mojo, right, so you can still use those kind of type annotations, it's fine. But in Mojo, if you declare a type and you use it, then it means it is going to be that type.

Lex Fridman 39:22 Mm-hmm.

Chris Lattner 39:23 And the compiler helps you check that and enforce it and it's safe. Um, and it's not, it's not a, like, best effort hint kind of a thing.

Lex Fridman 39:30 So if you try to shove a string-type thing into a integer...

Chris Lattner 39:34 You get an error from the compiler.

Lex Fridman 39:36 You get an, from the compiler, compile time.

Chris Lattner 39:39 Yeah.

Lex Fridman 39:39 Nice. Okay. What kind of basic types are there?

Chris Lattner 39:41 Yeah, so, uh, Mojo is, um, pretty hardcore in terms of what it tries to do in the language.

Lex Fridman 39:49 Mm-hmm.

Chris Lattner 39:49 Which is, the philosophy there is that we, um ... Again, if you, if you look at Python, right? Python's a beautiful language because it's so extensible, right? And so all of the different things in Python, like for loops and plus and like all these things can be accessed through these underbar underbar methods.

Lex Fridman 40:07 Mm-hmm.

Chris Lattner 40:07 Okay? So you have to say, "Okay, if I make something that is super fast, I can go all the way down to the metal, why do I need to have integers built into the language?"

Lex Fridman 40:16 Mm-hmm.

Chris Lattner 40:17 Right? And so what Mojo does is it says, "Okay, well, we can have this notion of structs." So you have classes in Python.

Lex Fridman 40:22 Mm-hmm.

Chris Lattner 40:22 Now you can have structs.

Lex Fridman 40:23 Mm-hmm.

Chris Lattner 40:24 Classes are dynamic. Structs are static. Cool, we can get high performance, we can write C++ kind of code with structs if you want.

Lex Fridman 40:31 Mm-hmm.

Chris Lattner 40:31 These things mix and work beautifully together. Um, but what that means is that you can go and implement strings and ints and floats and arrays and all that kind of stuff in the language, right? And so that's really cool because, you know, to, to me as a, uh, i- ideal- i- idealizing compi- compiler language type of person, what, what I wanna do is I wanna get magic out of the compiler-

Lex Fridman 40:52 Mm-hmm.

Chris Lattner 40:53 ... and put it in the libraries. Because if somebody can, you know, if, if we can build an integer that's beautiful and it has an amazing API and it does all the things you'd expect an integer to do, but you don't like it, maybe you want a big integer, maybe you want a, like, sideways integer, I don't know, like, what, what all the space of integers are, um-

Lex Fridman 41:09 (laughs)

Chris Lattner 41:09 ... then, uh, then you can do that, and it's not a second-class citizen. And so if you look at certain other languages, like C++, one I also love and use a lot, um, int is hardcoded in the language, but complex is not.

Lex Fridman 41:26 Mm-hmm.

Chris Lattner 41:27 And so isn't it kind of weird that, you know, you have this STD complex class-

Lex Fridman 41:30 Mm-hmm.

Chris Lattner 41:31 ... but you have int, and complex tries to look like a natural numeric type and things like this, but integers and floating point have these, like, special promotion rules and other things like that that are magic and they're hacked into the compiler. And because of that, you can't actually make something that works like the built-in types.

Lex Fridman 41:46 Mm-hmm.

Chris Lattner 41:47 And so...

Lex Fridman 41:47 Is there something provided as a standard because, uh, you know, b- because it's AI first, you know, numerical types are so important here.

Chris Lattner 41:56 Mm-hmm.

Lex Fridman 41:56 So is there something com- like a nice standard implementation of integer and floats?

Chris Lattner 42:01 Yeah, so, so we're still building all that stuff out. So we provide integers and floats and all that kind of stuff. We also provide like buffers and tensors and things like that that you'd expect in an ML context. Honestly, we need to keep designing and redesigning and working with the community to build that out and make that better. That's not our strength right now. (laughs) Give us six months or a year and I think it'll be way better. But, um, but the power of putting it in the library means we can have teams of experts that aren't compiler engineers that can help us design and refine and drive this forward.

Lex Fridman 42:28 So, uh, one of the exciting things we should mention here is that this is, uh, this is new and fresh. This cake is unbaked. It's almost baked. You can tell it's delicious, but it's not fully ready to be consumed.

Chris Lattner 42:42 Yup, that's very fair. It is very useful, but it's very useful if you're a super low level programmer right now. And what we're doing is we're working our way up the stack. And so the way I would look at Mojo today, in May in 2023, um, is that it's like a 0.1.... so I think that, you know, a year from now, it's gonna be way more interesting to a variety of people. But what we're doing is we're, we decided to release it early so that people can get access to it and play with it, and we can build it with the community. We, um, have a big roadmap, fully published, being transparent about this, and a lot of people are involved in this stuff. And so what we're doing is we're really optimizing for building this thing the right way.

Lex Fridman 43:21 Mm-hmm.

Chris Lattner 43:21 And building it the right way is kind of interesting working with the community, because everybody wants it yesterday, (laughs) and so it's, it's, sometimes it's kind of, you know, there's some dynamics there, but I-

Lex Fridman 43:32 Yeah.

Chris Lattner 43:32 ... I think it's a good, it's the right thing.

Lex Fridman 43:34 So there's a Discord also. So the dynamics is pretty interesting.

Chris Lattner 43:37 Yeah.

Lex Fridman 43:37 Sometimes the community probably can be very chaotic and, uh, introduce a lot of stress. Guido famously quit over the stress of the walrus operator. I mean, it's-

Chris Lattner 43:48 Yeah.

Lex Fridman 43:48 ... you know, it broke... (laughs)

Chris Lattner 43:50 Broke, it's the straw that broke the camel's back.

Lex Fridman 43:52 Yeah, exactly. And so like, it could be very stressful as you, to develop. But can you just, a tangent upon a tangent, is it stressful to, to, uh, to work through the design of various features here, given that the community is so richly involved?

Chris Lattner 44:07 Well, so, um, so I've been doing open development and community stuff for decades now, (laughs) somehow this has happened to me. Um, so I've, I've learned some tricks. But the, the thing that always gets me is I want to make people happy, right? And so this, this is maybe not all people all happy all the time, but-

Lex Fridman 44:23 Yeah.

Chris Lattner 44:23 ... generally I want, I want people to be happy, right? And so the challenge is that, again, we're tapping into some long, some deep seeded, long tensions and pressures both in the, the Python world, but also in the AI world, in the hardware world and things like this. And so people just want us to move faster, right? And so again, our decision was let's release this early. Let's get people used to it or access to it and play with it. And like, let's, let's build in the open, which we could have, you know, had the, the, uh, language monk sitting in the cloister up on the mo- hil- hilltop, like beavering away trying to build something. But in my experience, you get something that's way better if you work with the community.

Lex Fridman 45:03 Right. Uh-

Chris Lattner 45:04 And so yes, it can be frustrating, it can be challenging for lots of people involved. And you know, if you, I mean if you mentioned our Discord, we have over 10,000 people on the Discord, 11,000 people or something. Keep in mind we released Mojo like two weeks ago.

Lex Fridman 45:15 Yeah.

Chris Lattner 45:15 (laughs) So, um-

Lex Fridman 45:17 It's very active.

Chris Lattner 45:18 So it's very cool, um, but what that means is that, um, you know, 10, 11,000 people all will want something different.

Lex Fridman 45:25 Right.

Chris Lattner 45:25 And so what we've done is we've tried to say, okay, cool, here's our roadmap. Here, here, and the roadmap isn't completely arbitrary. It's based on here's the logical order in which to build these features or add, add these capabilities and things like that. And what we've done is we've spun really fast on like bug fixes. And so I, we actually have very few bugs, which is cool, I mean, actually for a project in this state. But then what we're doing is we're dropping in features very deliberately.

Lex Fridman 45:51 I mean, this is fun to watch 'cause you got the two gigantic communities of like hardware, like systems engineers, and then you have the machine learning Python people that are like higher level.

Chris Lattner 46:03 Yeah.

Lex Fridman 46:03 And it's just two, like four, like army, like, uh-

Chris Lattner 46:07 They've all, they've been at war. Yeah.

Lex Fridman 46:09 (laughs)

Chris Lattner 46:09 They've been at war. Right? And so, so here, here's, here's a test-

Lex Fridman 46:12 Like a Tolkien novel or something. Okay.

Chris Lattner 46:13 So here's the test. Again, like it's, it's super funny for, for something that's only been out for two weeks, right? People are so impatient, right? But okay, cool, let's fast forward a year. Like in a year's time, Mojo will be actually quite amazing and solve tons of problems and be very good. Um, people still have these problems.

Lex Fridman 46:30 Mm-hmm.

Chris Lattner 46:31 Right? And so you, you look at this and you say, and the way I look at this at least is to say, okay, well, we're solving big, long-standing problems. To me, I, again working on many different problems, I want to make sure we do it right. Right? There's like a responsibility you feel because if you mess it up, (laughs) right, there's very few opportunities to do projects like this and have them really have impact on the world. If we do it right, then maybe we can take those feuding armies and actually heal some of those wounds.

Lex Fridman 46:59 Yeah.

Chris Lattner 46:59 Right?

Lex Fridman 46:59 This is like, this feels, this feels like a speech by George Washington or Abraham Lincoln or something. Uh... (laughs)

Chris Lattner 47:05 (laughs) And you look at this and it's like, okay, well how different are we?

Lex Fridman 47:07 Yeah.

Chris Lattner 47:08 We all want beautiful things. We all want something that's nice. We all want to be able to work together. We all want our stuff to be used, right? And so if we can help heal that, now I'm not optimistic that all people will use Mojo and they'll stop using C++. Like I, that's not my goal, right? But, um, but if we can heal some of that, I think that'd be pretty cool.

Lex Fridman 47:24 Yeah.

Chris Lattner 47:24 That's nice.

Lex Fridman 47:24 And we start by putting the people who like braces into the gulag. No. Uh-

Chris Lattner 47:30 So, so there are proposals for adding braces to Mojo, and we just-

Lex Fridman 47:33 Oh, interesting.

Chris Lattner 47:33 ... we tell them no.

Lex Fridman 47:35 Okay. (laughs)

Chris Lattner 47:36 (laughs)

Lex Fridman 47:37 Politely.

Lex Fridman 47:38 Yeah. Anyway, so there's a lot of amazing features on the roadmap and those already implemented. It'd be awesome-

Chris Lattner 47:43 Yeah.

Lex Fridman 47:43 ... I could just ask you a few things.

Chris Lattner 47:44 Yeah. Go for it.

Lex Fridman 47:45 So, uh, the other performance improvement comes from immutability. So what's the, what's this var and this let thing that we got going on? What's-

Chris Lattner 47:55 Well, so-

Lex Fridman 47:55 ... what's immutability?

Chris Lattner 47:57 Yeah. So one of the things that is useful, and it's not always required but it's useful, is knowing whether something can change out from underneath you. Right? And so in Python you have a pointer to an array, right? And so you pass that pointer to an array around to things. If you pass into a function, they may take that and scroll away in some other data structure.

Lex Fridman 48:17 Mm-hmm.

Chris Lattner 48:18 And so you get your array back and you go to use it. Now somebody else is like putting stuff in your array.

Lex Fridman 48:22 Yeah.

Chris Lattner 48:23 How do you reason about that? It gets to be very complicated and leads to lots of bugs, right?

Lex Fridman 48:28 Mm-hmm.

Chris Lattner 48:28 And so one of the things that, you know, again, this is not something Mojo forces on you, but something that Mojo enables is a thing called value semantics. And what value semantics do is they take collections like arrays, like dictionaries, also tensors and strings and things like this that are much higher level and make them behave like proper values. And so it makes it look like if you pass these things around, you get a logical copy of all the data.

Lex Fridman 48:54 Mm-hmm.

Chris Lattner 48:55 And so if I pass you an array, it's your array. You can go do what you want to it. You're not gonna hurt my array.... now, that is an interesting and very powerful design prin- principle. It defines away a ton of bugs. You have to be careful to implement it in an efficient way. (laughs)

Lex Fridman 49:08 I- yes, is there a performance hit that's s- uh, significant?

Chris Lattner 49:12 Uh, generally not, if you implement it the right way.

Lex Fridman 49:15 Interesting.

Chris Lattner 49:15 But it requires a lot of very low level, uh, getting the language right bits.

Lex Fridman 49:20 I- I assumed that'd be a huge performance hit 'cause it's a really n- the benefit is really nice 'cause you don't get into the-

Chris Lattner 49:25 Absolutely. Well, well, the-

Lex Fridman 49:27 ... complex ...

Chris Lattner 49:27 ... the trick is, is you can't do, you can't do copies, so you have to provide the behavior of copying without doing the copy. (laughs)

Lex Fridman 49:35 Yeah. How do you do that? Is it ... How do you do that?

Chris Lattner 49:39 It's not magic.

Lex Fridman 49:40 Okay.

Chris Lattner 49:40 It's just ... It's- it's actually pretty cool. Well, so first, before we talk about how that works, let's talk about how it works in Python, right? So in Python, you define a person class, or maybe a person class is a bad idea. The ... You define a database class, right?

Lex Fridman 49:52 Mm-hmm.

Chris Lattner 49:52 And a database class has an array of records, something like that, right? And so what the problem is, is that if you pass in a record or a class instance into the database, it'll take ahold of that object, and then it assumes it has it.

Lex Fridman 50:04 Mm-hmm.

Chris Lattner 50:05 And if you're passing an object in, you have to know that that database is gonna take- take it, and therefore, you shouldn't change it after you put it in the database, right? This is- this is the problem.

Lex Fridman 50:12 You just kinda have to know that.

Chris Lattner 50:13 You just have to kinda know that, right? And so, you roll out version one of the database. You just kinda have to know that. Of course, Lex uses his own database, right?

Lex Fridman 50:21 Yeah.

Chris Lattner 50:22 Right? 'Cause you built it. You understand how this works, right?

Lex Fridman 50:24 Mm-hmm.

Chris Lattner 50:25 Somebody else joins the team. They don't know this.

Lex Fridman 50:26 Yes.

Chris Lattner 50:27 Right? And so now they suddenly get bugs. You're having to maintain the database. You shake your fist.

Lex Fridman 50:32 Mm-hmm.

Chris Lattner 50:32 You argue. The 10th time this happens you're like, "Okay, we have to do something different," right?

Lex Fridman 50:36 Mm-hmm.

Chris Lattner 50:36 And so what you do is you go change your Python code and you change your database class to copy the record-

Lex Fridman 50:42 Mm-hmm.

Chris Lattner 50:42 ... every time you add it. And so what ends up happening is you say, "Okay, I will do what's called a defensive copy inside the database, and then that way, if somebody passes something in, I will have my own copy of it, and they can go do whatever and they're not gonna break my thing." (laughs) Okay, this is usually the- the two design patterns. If you look in PyTorch, for example, this is cloning a tensor.

Lex Fridman 51:03 Mm.

Chris Lattner 51:03 Like, there's a specific thing, and you have to know where to call it, and if you don't call it in the right place, you get these bugs, and-

Lex Fridman 51:07 Mm-hmm.

Chris Lattner 51:08 ... this is state of the art, right? So a different approach, so it's used in many languages, so I've worked with it in Swift, um, is you say, "Okay, well, let's provide value semantics." And so we wanna provide the view that you get a logically independent copy.

Lex Fridman 51:22 Mm-hmm.

Chris Lattner 51:23 But we wanna do that lazily. And so what- what we do is we say, "Okay, if you pass something into a function, it doesn't actually make a copy." What it actually does is it just increments a reference to it, and if you pass it around, you stick it in your database, it can go in the database, you own it, and then you come back out of the stack, nobody's copied anything. You come back out of the stack, and then the caller lets go of it. Well, then, you've just handed it off to the database, you've transferred it, and there's no copies made. Now on the other hand, if, you know, your coworker goes and hands you a record and you pass it in, you stick it in the database, and then you go to town and you start modifying it, what happens is you get a copy lazily on demand. And so what this does is gives you copies only when you need them, and it also ... So it defines away the bugs, but it also re- generally reduces the number of copies in practice, and so it's-

Lex Fridman 52:12 But the implementation details are tricky here? I assume it doesn't-

Chris Lattner 52:16 Yeah, so this is ... Yes.

Lex Fridman 52:17 Something with reference counting, but to make it performant, uh, across a number of different kinds of objects?

Chris Lattner 52:26 Yeah. Well, so you need a couple of things. So there's many ... So this concept has existed in many different worlds, and so it's a ... Again, it's not novel research at all, right? The magic is getting the design right so that you can do this in a reasonable way, right? And so there's a number of components that go into this. One is, when you're passing around ... So you ... We're talking about Python and reference counting and the expense of doing that. When you're passing values around, you don't wanna do extra reference counting for no good reason.

Lex Fridman 52:51 Mm-hmm.

Chris Lattner 52:52 And so you have to make sure that you're efficient and you transfer ownership instead of duplicating references and things like that, which is a very low level problem. Um, you also have to adopt this, and you have to build these data structures. And so if you say, um, you know, Mojo has to be compatible with Python, so of course, the default list is a reference semantic list that works the way you'd expect in Python.

Lex Fridman 53:13 Mm-hmm.

Chris Lattner 53:14 But then you have to design a value semantic list, and so you just have to implement that, and then you implement the logic within. And so the- the role of the language here is to provide all the low level hooks that allow the author of the type to be able to get and express this behavior without forcing it into all cases or hard coding this into the language itself.

Lex Fridman 53:32 But there's, uh, ownership, so you, you're- you're constantly transfer- ... You're tracking who owns the thing.

Chris Lattner 53:37 Yes. And so there's a whole system called ownership, and so this is related to work done in the Rust community.

Lex Fridman 53:43 Mm-hmm.

Chris Lattner 53:43 Um, also the Swift community has done a bunch of work, and there's a bunch of different other languages that have all kind of ... C++ actually has copy constructors and destructors and things like that, and so, um ... And m- ... I mean, C++ has everything, so it has move constructors and it has, like, all ... this whole world of things, and so this is, uh, this is a body of work that's kinda been developing for many, many years now. And so Mojo takes some of the best ideas out of all of these systems and then remixes it in a nice way so that, um, you get the power of something like the Rust programming language, but you don't have to deal with it when you don't want to, which is a major thing in terms of teaching and learning and being able to use and scale these systems.

Lex Fridman 54:20 Uh, well, how does that play with argument conventions? What are they? Why are they important? How does the value of semantics, how does the transfer of ownership, uh, work with- with the arguments when they're passed into functions?

Chris Lattner 54:30 Yeah. So- so if you go deep into systems programming land, so this isn't ... Again, this is not something for everybody, but if you go deep into systems programming land, what you encounter is you encounter these types that get weird. (laughs) So if you're used to Python and you think about everything, I could just copy it around, I can go change it and mutate it and do these things and it's all cool. Um, if you get into systems programming land, you get into these things like, I have an atomic number, or I have a mutex, or I have a, um, uniquely owned database handle.

Lex Fridman 55:02 Mm-hmm.

Chris Lattner 55:02 Things like this, right? So these types, you can't necessarily copy. Sometimes you can't necessarily even move them to a different address.... and so what Mojo allows you to do is it allows you to express, "Hey, I-I don't wanna get a copy of this thing. I wanna actually just get a reference to it." And by doing that, what you can say is you can say, "Okay, if I'm defining something weird, like a, you know, atomic number or something," it's like, it has to be... So an a- so an atomic number is a, uh, an area in memory that multiple threads can access at a time without synchronous- without, without locks, right? And so, uh, and so, like, the definition of an atomic number is multiple different things have to be poking it, that therefore they have to agree on where it is, (laughs) right? And so you can't just, like, move it out from underneath one, because it kind of breaks what it, what it means. And so that's, that's an example of a type that you can't even... you can't copy, you can't move. It, like... once you create it, it has to be where it was, right? Now, if you look at, um, many other examples, like a database handle, right? So, okay, well what happens? How do you copy a database handle? Do you copy the whole database? That's not something you necessarily want to do. Um, the... there's a lot of types like that th- where you want to be able to say that they are uniquely owned.

Lex Fridman 56:11 Mm-hmm.

Chris Lattner 56:11 So there's always one of this thing, and... or if, if I create a thing, I d- I don't copy it.

Lex Fridman 56:19 Mm-hmm.

Chris Lattner 56:19 And so what Mojo allows you to do is it allows you to say, "Hey, I wanna pass around a reference to this thing without copying it." And so it has borrowed conventions, so you can say, "You can use it, but you don't get to change it. You can pass it by mutable reference." And so if you do that, then you can... you get a reference to it, but you can change it. And so it manages all that kind of stuff.

Lex Fridman 56:39 So it's, uh, it's just a really nice implementation of... like, uh, C++ has-

Chris Lattner 56:44 Yeah.

Lex Fridman 56:44 ... uh, you know-

Chris Lattner 56:46 References.

Lex Fridman 56:46 ... different kinds of pointers-

Chris Lattner 56:47 Yeah, has pointers.

Lex Fridman 56:48 ... uh, smart, smart... Different, different kinds of implementations and smart pointers-

Chris Lattner 56:50 Yep.

Lex Fridman 56:50 ... that you can, uh, explicitly define this s- allows you... But you're saying that's more like, um, the weird case versus the common case.

Chris Lattner 56:57 Well, it s- it depends on where... I mean, uh, I mean-

Lex Fridman 57:00 Huh, yes.

Chris Lattner 57:00 ... I don't th- I don't think I'm a normal person, so-

Lex Fridman 57:01 Yes.

Chris Lattner 57:02 ... I mean, I'm not one to call other people weird.

Lex Fridman 57:03 Yeah.

Chris Lattner 57:04 Um, (laughs) but the, uh, uh... but, you know, if you talk to a normal Python... a typical Python programmer-

Lex Fridman 57:10 Yeah.

Chris Lattner 57:10 ... you're typically not thinking about this, right? This is a lower level of abstraction. Now, if you talk to a C++ programmer, certainly if you talk to a Rust programmer, again, they're not weird, they're delightful. Like, the- these are all good people, right? Um, those, those folks will think about it all the time, right? And so y- I, I look at this as there's a spectrum between very deep low-level systems, "I'm gonna go poke the bits and care about how they're laid out in memory," all the way up to application and scripting and other things like this. And so it's not that anybody's right or wrong. It's about, how do we build one system that scales?

Lex Fridman 57:41 By the way, the, the idea of a, a- an atomic number has been something that always brought me deep happiness, because the flip side of that, the, the idea that threads can just modify stuff, um, asynchronously, just the whole idea of concurrent programming is a source of infinite stress for me.

Chris Lattner 58:05 Well, so this is where you jump into... Um, uh, you know, again, you zoom out and, and get out of programming languages or compilers and you just look at what the industry has done.

Lex Fridman 58:13 Mm-hmm.

Chris Lattner 58:14 My, my mind is constantly blown by this, right? And you look at what... you know, Moore's law.

Lex Fridman 58:20 (laughs)

Chris Lattner 58:20 Moore's law is this idea (laughs) that, like, computers, for a long time, single thread performance just got faster and faster and faster and faster for free, but then physics and other things intervened, and power consumption, like, other things started to matter. And so what ended up happening is we went from single-core computers to multi-core, then we went to accelerators, right? And this, this trend towards specialization of hardware is only gonna continue. And so, for years, us programming language nerds and compiler people have been saying, "Okay, well how do we ta- tackle multi-core?" Right? For a while it was like, "Multi-core's the future, we have to get on top of this thing." Then it was, "Multi-core is the default, what are we doing with this thing?" And then it was like, "There's chips with hundreds of cores in them." (laughs) What, what would happen, right?

Lex Fridman 59:02 Yeah.

Chris Lattner 59:03 And so I'm super inspired by the fact that, you know, in the face of this, you know, those machine learning people invented this idea of a tensor.

Lex Fridman 59:12 Mm-hmm.

Chris Lattner 59:12 Right? And what is a tensor? A tensor is an... like, an arithmetic, an algebraic concept. It's like an abstraction around a gigantic parallelizable data set, right?

Lex Fridman 59:23 Hmm.

Chris Lattner 59:24 And because of that, and because of things like TensorFlow and PyTorch, we're able to say, "Okay, well, express the math of the system."

Lex Fridman 59:31 Mm-hmm.

Chris Lattner 59:31 This enables you to do automatic differentiations, enables you to do, like, all these cool things. Um, and, and it's, it's an abstract representation. Well, because you have that abstract representation, you can now map it onto these parallel machines without having to, um, control, "Okay, put that byte here, put that byte there, put that byte there." And this has enabled an explosion in terms of AI compute, accelerators, like all the stuff. And so that's super, super exciting.

Lex Fridman 59:56 What about the, uh, the deployment and the execution across multiple machines?

Chris Lattner 1:00:00 Yeah.

Lex Fridman 1:00:00 So, uh, you write that, "The modular compute platform dynamically partitions models with billions of parameters and distributes their execution across multiple machines, uh, enabling unparalleled efficiency." By the way, the, the use of "unparalleled" in that sentence... anyway-

Chris Lattner 1:00:18 (laughs)

Lex Fridman 1:00:18 "... enabling unparalleled efficiency, scale, and reliability for the largest workloads." So how, how do you do this, um, abstraction of, um, distributed deployment o- of, of, of large models?

Chris Lattner 1:00:31 Yeah, so one, one of the really interesting, um, tensions... So there's a whole bunch of stuff that goes into that. I'll pick a random walk through it.

Lex Fridman 1:00:38 Mm-hmm.

Chris Lattner 1:00:38 Uh, if you, if you go back and replay the history of machine learning, right? I mean, the brief, the brief most recent history of machine learning, 'cause this is ver- as you know, very deep.

Lex Fridman 1:00:46 Yeah.

Chris Lattner 1:00:48 I, uh, I knew Lex when he had an AI podcast, yes. (laughs)

Lex Fridman 1:00:52 Right? Yep.

Chris Lattner 1:00:54 (laughs)

Lex Fridman 1:00:54 Yeah.

Chris Lattner 1:00:54 Um, so, uh, so if you look at just TensorFlow and PyTorch, which is pretty recent history in the big picture, right? But TensorFlow is all about graphs. PyTorch, I think, pretty unarguably ended up winning. And why did it win? Mostly because of usability.

Lex Fridman 1:01:09 Mm-hmm.

Chris Lattner 1:01:09 Right, and the usability of PyTorch is, I think, huge. And I think, again, that's a huge testament to the power of taking abstract, theoretical, technical concepts and bringing it to the masses, right? Now, the challenge with what the TensorFlow versus the PyTorch n- design points was that TensorFlow's e- kind of difficult to use for researchers, but it was actually pretty good for deployment. PyTorch is really good for researchers, it kind of's not super great for deployment, right? And so I think that we as an industry have been struggling. And if you look at what deploying a machine learning model today means is that you'll have researchers who are, I mean, wicked smart, of course, but they're wicked smart at model architecture and data and calculus (laughs) and like all, uh, like, they're wicked smart in various domains. They don't wanna know anything about the hardware or deployment or C++ or things like this, right? And so what's happened is you get people who train the model, they throw over, throw it over the fence, and then you have people that try to deploy the model. Well, every time you have a team A does X, they throw it over the fence, and team Y does some, t-team B does Y, like, you have a problem, because of course it never works the first time. And so you throw it over the fence, they figure out, "Okay, it's too slow, it won't fit, doesn't use the right operator, m- the tool crashes," whate- whatever the problem is. Then they have to throw it back over the fence. And every time you throw a thing over a fence, it takes three weeks of project managers and meetings and things like this. And so, uh, what we've seen today is that getting models into production can take weeks or months.

Lex Fridman 1:02:43 Mm-hmm.

Chris Lattner 1:02:43 Like, it's not atypical, right? I talk to lots of people, and you talk about, like, VP of software at some internet company trying to deploy a model, and they're like, "Why do I need a team of 45 people?" (laughs) Like, "I, it's so easy to train a model. Why, why can't I deploy it," right? And if you dig into this, every layer is problematic. So if you look at the language piece, I mean, this is tip of the iceberg. It's a very exciting tip of the iceberg for folks, but you've got Python on one side and C++ on the other side. Python doesn't really deploy. I mean, it can theoretically, technically, in some cases, but often a lot of production teams will want to get things out of Python because they get better performance and control and whatever else. So Mojo can help with that. If you look at serving, so you talk about gigantic models. Well, a gigantic model won't fit on one machine (laughs) , right? And so now you have this model, it's written in Python, it has to be rewritten in C++. Now it also has to be carved up so that half of it runs on one machine, half of it runs on another machine.

Lex Fridman 1:03:40 Mm-hmm.

Chris Lattner 1:03:40 Or maybe it runs on 10 machines. Well, so now suddenly the complexity is exploding, right? And the reason for this is that if you, if you look into TensorFlow, PyTorch, these systems, they weren't really designed for this world, right? They were designed for, you know, back in the day when we were n- starting and doing things, where it was a different, much simpler world. Like, you wanted to run ResNet-50 or some ancient model architecture like this, it was just a, it was a completely different world than-

Lex Fridman 1:04:08 Train on one GPU-

Chris Lattner 1:04:09 Exactly.

Lex Fridman 1:04:10 ... do inference on one GP (laughs) .

Chris Lattner 1:04:11 AlexNet, yeah, AlexNet, right? The major breakthrough.

Lex Fridman 1:04:14 (laughs)

Chris Lattner 1:04:14 And, um, and the world has changed, right? And so now the challenge is, is that TensorFlow, PyTorch, these systems, they weren't actually designed for LLMs. Like, that, that was not, that was not a thing. And so what, where TensorFlow actually has amazing power in terms of scale and deployment and things like that, and I think Google is, I mean, maybe not unmatched, but they're, like, incredible in terms of their capabilities and, and gigantic scale. Um, m-many researchers using PyTorch, right? And so PyTorch doesn't have those same capabilities. And so what Modular can do is it can help with that. Now if you take a step back and you say, like, "What is Modular doing," right? So Modular has, like, a, a bitter enemy that we're fighting against in the industry, and it's one of these things where everybody knows it, but nobody is usually willing to talk about it.

Lex Fridman 1:05:00 The bitter enemy.

Chris Lattner 1:05:02 The bitter thing that we have to destroy...

Lex Fridman 1:05:04 Yeah.

Chris Lattner 1:05:04 ... that we're all struggling with, and it's, like, all ar- it's like fish can't see water-

Lex Fridman 1:05:07 Mm-hmm.

Chris Lattner 1:05:08 ... is complexity.

Lex Fridman 1:05:09 Sure, yes.

Chris Lattner 1:05:10 (laughs)

Lex Fridman 1:05:10 Is complexity.

Chris Lattner 1:05:12 Right.

Lex Fridman 1:05:12 That was very philosophical of you-

Chris Lattner 1:05:13 (laughs)

Lex Fridman 1:05:13 ... to say that.

Chris Lattner 1:05:15 And so if you look at it, yes, it is on the hardware side.

Lex Fridman 1:05:18 Yes.

Chris Lattner 1:05:18 All these, all these accelerators, all these software stacks that go with the accelerator, all these, like there's massive complexity over there. You look at what's hap- happening on the modeling side, massive amount of complexity, like things are changing all the time. People are inventing... Turns out the research is not done (laughs) , right? And so people want to be able to move fast. Transformers are amazing, but there's a ton of diversity even within transformers, and what's the next transformer, right? And you look into serving.

Lex Fridman 1:05:44 Mm-hmm.

Chris Lattner 1:05:45 Also huge amounts of complexity. It turns out that all the cloud providers, right, have all their very weird but very cool hardware for networking and all this kind of stuff, and it's all very complicated. People aren't using that. You look at classical serving, right? There, there's this whole world of people who know how to write high performance servers with zero-copy networking and, like, all, all this fancy, uh, asynchronous I/O and, like, all these fancy things in the, in, in the serving community. Very little of that has pervaded into the machine learning world.

Lex Fridman 1:06:14 Mm-hmm.

Chris Lattner 1:06:14 Right? And why is that? Well, it's because, again, these systems have been built up over many years. They, they haven't been rethought. There hasn't been a first principles approach to this.

Lex Fridman 1:06:23 Mm-hmm.

Chris Lattner 1:06:23 And so what Modular is doing is we're saying, okay, we've built many of these things, right? So I've worked on TensorFlows and TPUs and things like that. Other folks on our team have, like, worked on PyTorch Core, we've worked on ONNX Runtime, we've worked on many of these other systems, and so built systems like the Apple accelerators and all that kind of stuff. Like, our team is quite amazing. And so one of the things that roughly everybody at Modular is grumpy about is that when you're working on one of these projects, you have a first order goal.

Lex Fridman 1:06:53 Mm-hmm.

Chris Lattner 1:06:54 Get the hardware to work, get the system to enable one more model, get this product out the door, enable this specific workload or make it, uh, solve this problem for this, this product team, right?

Lex Fridman 1:07:05 Mm-hmm.

Chris Lattner 1:07:05 And nobody's been given a chance to actually do that step back. And so we as an industry, we didn't take two steps forward, we took, like, 18 steps forward-

Lex Fridman 1:07:12 (laughs)

Chris Lattner 1:07:12 ... in terms of all this really cool technology across compilers and systems and run times and heterogeneous computing, like all this kind of stuff, and, like, all this technology has been...... you know, I wouldn't say, uh, beautifully designed, but it's been proven in different quadrants.

Lex Fridman 1:07:25 Mm-hmm.

Chris Lattner 1:07:25 Like, you know, you look at Google with TPUs. Massive, huge exaflops of compute strapped together in- into machines that researchers are programming in Python in a notebook. That's huge. That's amazing.

Lex Fridman 1:07:38 That's amazing. It's incredible.

Chris Lattner 1:07:39 Right? It's incredible. And so you look at the technology that goes into that, and the- the algorithms are actually quite general. And so lots of other hardware out there that- and lots of other teams out there don't have the sophistication or the, maybe the- the years working on it, or the- the budget, or whatever that Google does, right? And so they should be getting access to the same algorithms, but they just don't have that, right? That's what Modular is doing, is we're saying, "Cool, this is not research anymore." Like, we've- we've built auto-tuning in many systems. We've built programming languages, right? (laughs) And some, like, have- have, you know, implemented C++, have implemented Swift, have implemented many of these things. And so, you know, th- this, i- i- it's hard, but it's not research.

Lex Fridman 1:08:19 Mm-hmm.

Chris Lattner 1:08:19 And you look at accelerators. Well, we know there's a bunch of different weird kind of accelerators, but they actually cluster together, right? And you look at GPUs. Well, there's a couple of major vendors of GPUs, and they maybe don't always get along, but their architectures are very similar. You look at CPUs. CPUs are still super important for the deployment side of things, and you're seeing new- new architectures coming out from all the cloud providers and things like this, and they're all super important to the world, right? But they don't have the 30 years of development that the entrenched people do, right? And so what Modular can do is we're saying, "Okay, all this complexity, like, it's not- it's not bad complexity, it's actually innovation," (laughs) right? And so it's innovation that's happening, and it's for good reasons, but I have sympathy for the poor software people, right? I mean, again, I'm a- generally a software person too, I love hardware. But software people wanna build applications and products and solutions that scale over many years. They don't wanna build a solution for one p- generation of hardware with one vendor's tools, right? And because of this, they need something that scales with them, they need something that works on cloud and mobile, right? Because, you know, their product manager said, "Hey, I want it to be- h- have lower latency, and it's better for personalization," or whatever they decide, right? Products evolve. And so the challenge with the machine learning technology and the infrastructure we have today in the industry is that it's all these point solutions.

Lex Fridman 1:09:44 Mm-hmm.

Chris Lattner 1:09:44 And because they're all these point solutions, it means that as your product evolves, you have to, like, switch to different technology stacks or switch to a different vendor. And what that does is that slows down progress.

Lex Fridman 1:09:54 So basically, uh, a lot of the things we've developed in those little, uh, silos for machine learning tasks, you want to make that the first-class citizen of a general-purpose programming language that can then be compiled across all these kinds of hardware.

Chris Lattner 1:10:08 Well, so it's- it's not really about a programming language. I mean, the programming language is a component of the mission, right?

Lex Fridman 1:10:13 Hm.

Chris Lattner 1:10:13 And the mission is, our not literal but our joking mission is to save the world from terrible AI software.

Lex Fridman 1:10:19 (laughs) Excellent. I love it.

Chris Lattner 1:10:20 Okay? (laughs) So- so, you know, if you look at this mission, you need a syntax.

Lex Fridman 1:10:26 Mm-hmm.

Chris Lattner 1:10:27 So that's th- so yes, you need a programming language, right? And- and, like, we wouldn't have to build the programming language if one existed, right? So if Python was already good enough, then cool, we would've just used it, right? We're not just doing very large-scale expensive engineering projects for