DeepSeek, China, OpenAI, NVIDIA, xAI, TSMC, Stargate, and AI Megaclusters | Lex Fridman Podcast #459

Lex Fridman 0:00 The following is a conversation with Dylan Patel and Nathan Lambert. Dylan runs SemiAnalysis, a well-respected research and analysis company that specializes in semiconductors, GPUs, CPUs, and AI hardware in general. Nathan is a research scientist at the Allen Institute for AI, and is the author of the amazing blog on AI called Interconnects. They are both highly respected, read, and listened to by the experts, researchers, and engineers in the field of AI. And personally, I'm just a fan of the two of them. So, I used the DeepSeek moment that shook the AI world a bit as an opportunity to sit down with them and lay it all out. From DeepSeek, OpenAI, Google, xAI, Meta Anthropic, to NVIDIA and TSMC, and to US, China, Taiwan relations, and everything else that is happening at the cutting edge of AI. This conversation is a deep dive into many critical aspects of the AI industry. While it does get super technical, we tried to make sure that it's still accessible to folks outside of the AI field by defining terms, stating important concepts explicitly, spelling out acronyms, and in general, always moving across the several layers of abstraction and levels of detail. There is a lot of hype in the media about what AI is and isn't. The purpose of this podcast, in part, is to cut through the hype, through the bullshit, and the low resolution analysis, and to discuss in detail how stuff works and what the implications are. Let me also, if I may, comment on the new OpenAI o3 mini reasoning model, the release of which we were anticipating during the conversation, and it did indeed come out right after. Its capabilities and cost are on par with our expectations, as we stated. OpenAI o3 mini is indeed a great model, but it should be stated that, uh, DeepSeek R1 has similar performance on benchmarks, is still cheaper, and it reveals its chain of thought reasoning, which o3 mini does not. It only shows a summary of the reasoning. Plus, R1 is open weight and, uh, o3 mini is not. By the way, I got a chance to play with, uh, o3 mini, and uh, anecdotal vibe check-wise, I felt that o3 mini, specifically o3 mini high, is, uh, better than R1. Still, for me personally, I find that Claude Sonnet 3.5 is the best model for programming, except for tricky cases where I will use o1 pro to brainstorm. Either way, many more better AI models will come, including reasoning models, both from American and Chinese companies. They will continue to shift the cost curve. But the "DeepSeek moment" is indeed real. I think it will still be remembered five years from now as a pivotal event in tech history, due in part to the geopolitical implications, but for other reasons too, as we discuss in detail from many perspectives in this conversation. This is the Lex Fridman Podcast. To support it, please check out our sponsors in the description, and now, dear friends, here's Dylan Patel and Nathan Lambert.

Lex Fridman 3:33 A lot of people are curious to understand China's DeepSeek AI models, so let's lay it out. Nathan, can you describe what DeepSeek V3 and DeepSeek R1 are, how they work, how they're trained? Let's, uh, look at the big picture and then we'll zoom in on the details.

Nathan Lambert 3:49 Yeah, so DeepSeek V3 is a new mixture of experts transformer language model from DeepSeek, who is based in China. They have some new specifics in the model that we'll get into. Largely this is a open weight model, and it's a instruction model like what you would use in ChatGPT. Um, they also released what is called the base model, which is before these techniques of post-training. Uh, most people use instruction models today, and those are what's served in all sorts of applications. This was released on, I believe, December 26th or that week, and then weeks later on January 20th, DeepSeek released DeepSeek R1, which is a reasoning model, which really accelerated a lot of this discussion. This reasoning model i- has a lot of overlapping training steps to DeepSeek V3, and it's confusing that you have a base model called V3 that you do something to to get a chat model, and then you do some different things to get a reasoning model. I think a lot of the AI industry is going through this challenge of communications right now where OpenAI makes fun of their own naming schemes. They have GPT-4o, they have OpenAI o1, and there's a lot of types of models, so we're gonna break down what each of them are. There's a lot of s- technical specifics on training and go through them high level to specific and kind of go through each of them.

Lex Fridman 5:14 There's so many places we can go here, but maybe let's go to open weights first. What does it mean for a model to be open weights, and what are the different flavors of open source in general?

Nathan Lambert 5:23 Yeah, so this discussion has been going on for a long time in AI, it became more important since ChatGPT, or more focal since ChatGPT at the end of 2022. Open weights is the accepted term for, um, when model weights of a language model are available on the internet for people to download. Those weights can have different licenses, which is the essen- effectively the terms by which you can use the model. There are licenses that come from history in open source software. There are licenses that are designed by companies specifically. Um, all of LLaMA, DeepSeek, Qwen, Mistral, these popular names in open weight models have some of their own licenses. It's complicated 'cause not all the s- same models have the same terms.The big debate is on what makes a model open weight. It's, like, why are we saying this term? It's kind of a mouthful. It sounds close to open source, but it's not the same. There's still a lot of debate on the definition and soul of open source AI. Open source software has a rich history on freedom to modify, freedom to take on your own, freedom from any restrictions on how you would use the software, and what that means for AI is still being defined. So, uh, for what I do, I work at the Allen Institute for AI. We're a nonprofit. We want to make AI open for everybody, and we try to lead on what we think is truly open source. There's not full agreement in the community, but for us, that means releasing the training data, releasing the training code, and then also having open weights like this. And we'll get into the details of the models and... Again and again, as we try to get deeper into how the models will train- were trained, we will say things like, "The data processing, data filtering, data quality is the number one determinant of the model quality," and then a lot of the training code is the determinant on how long it takes to train and how fast your experimentation is. So without fully open source models where you have access to this data, it is hard to know, or it's harder to replicate. So we'll get into cost numbers for DeepSeek V3 on mostly GPU hours and how much you could pay to rent those yourselves. But without the data, the replication cost is going to be far, far higher. And same goes for the code.

Lex Fridman 7:37 We should also say that this is probably one of the more open models out of the frontier models.

Nathan Lambert 7:43 Yeah.

Lex Fridman 7:43 So, like, in this full spectrum, we're probably the fullest open source, like you said. Open code, open data, open weights. This is not open code. This is probably not open data. And this is open weights, and the licensing is, uh, MIT license or it's... Uh, I mean, there's some nuance in the different models, but it's towards the free... In terms of the open source movement, these are the, kind of the good guys.

Nathan Lambert 8:13 Yeah. DeepSeek is doing fantastic work for disseminating understanding of AI. Uh, their papers are extremely detailed in what they do, and for other teams around the world, they're very actionable in terms of improving your own training techniques. Uh, and we'll talk about licenses more. The DeepSeek R1 model has a very permissive license. It's called the MIT license. That effectively means there's no downstream res- restrictions on commercial use. There's no use case restrictions. You can use the outputs from the models to create synthetic data. And this is all fantastic. I think the closest peer is something like LLaMA, where you have the weights and you have a technical report, and the technical report is very good for LLaMA. One of the most read PDFs of the year last year is the LLaMA 3 paper. But it... In some ways, it's slightly less actionable. It has less details on the training specifics, I think less plots, um, and so on. And the LLaMA 3 license is more restrictive than MIT, and then between the DeepSeek custom license and the LLaMA license, we could get into this whole rabbit hole, I think. We'll, we'll, we'll make sure we want to go down the license rabbit hole before we do specifics.

Lex Fridman 9:21 Yeah, and I mean... So it should be stated that one of the implications of DeepSeek, it puts pressure on LLaMA and everybody else on OpenAI to push towards, uh, open source. And that's the other side of open source that, uh, you mentioned, is how much is published in detail about it. So how open are you with the sort of the insights behind the code? So like, how good is the technical reports? Are they hand wavy or is there actual, uh, details in there? And that's one of the things that DeepSeek did well, is they published a lot of the details.

Nathan Lambert 9:52 Yeah. Especially in the DeepSeek V3, which is their pre-training paper. They were very clear that they are doing interventions on the technical stack that go at many different levels. For example, on their... To get highly efficient training, they're making modifications at or below the CUDA layer for NVIDIA chips. I have never worked there myself, and there are a few people in the world that do that very well, and some of them are at DeepSeek. And these types of people are at DeepSeek and leading American frontier labs, but there are not many places.

Lex Fridman 10:25 To help people understand w- the other implication of open weights, just, you know, there's, um, a topic w- we'll return to often here. So, there's a, uh, fear that China, the nation, might have interest in, um, stealing American data, violating privacy of American citizens. What can we say about open weights to help us understand what, what the weights are able to do-

Nathan Lambert 10:52 Yeah.

Lex Fridman 10:52 ... in terms of stealing people's data?

Nathan Lambert 10:54 Yeah. So these weights that you can download from Hugging Face or other platforms are very big matrices of numbers. You can download them to a computer in your own house that has no internet, and you can run this model and you're totally in control of your data. That is something that is different than how a lot of language model usage is actually done today, which is mostly through APIs, where you send your prompt to GPUs run by certain companies. And these companies will have different distributions and policies on how your data is stored, if it is used to train future models, where it is stored, if it is encrypted, and so on. So the open weights are you have your fate of data in your own hands, and that is something that is deeply connected to the soul of open source.

Lex Fridman 11:37 So it's not the model that steals your data, it's whoever's hosting the model, which could be China, if you're using the DeepSeek app, or it could be Perplexity. Uh, you know, you're trusting them with your data. Or OpenAI, you're trusting them with your data. And some of these are American companies, some of these are Chinese companies. But the model itself is not doing the stealing. It's the host. All right. So, uh, back to the basics. What's the difference between DeepSeek V3 and DeepSeek R1? Can we try to, like-... uh, lay out the confusion potential?

Nathan Lambert 12:11 Yes. So, for one, I have very understanding of many people being confused by these two model names. So, I would say the best way to think about this is that when training a language model, you have what is called pre-training, which is when you're predicting the large amounts of mostly internet text, you're trying to predict the next token. And what to know about these new DeepSeek models is that they do this internet large-scale pre-training once to get what is called DeepSeekV3 base. This is the base model, it's just going to finish your sentences for you, it's going to be harder to work with than ChatGPT. And then, what DeepSeek did is they've done two different post-training regimes to make the models have specific desirable behaviors. So, what is the more normal model in terms of the last few years of AI and instruct model, a chat model, a "aligned" model, a helpful model, there are many ways to describe this, is more standard post-training. So, this is things like instruction tuning, reinforcing Learning from Human Feedback. We'll get into some of these words. And this is what they did to create the DeepSeekV3 model. This was the first model to be released and it is very high re- performant, it's competitive with GPT-4, LLaMa-405B, so on. And then when this r- release was happening, we don't know their exact timeline or soon after, they were finishing the training of a different training process from the same next token prediction based model that I talked about, which is when this new reasoning training that people have heard about comes in, in order to create the model that is called DeepSeekR1. The R through this conversation is good for grounding for reasoning, and the name is also similar to OpenAI's O1, which is the other reasoning model that people have heard about. And while I have to break down the training for R1 in more detail, because for one, we have a paper detailing it, but also it is a far newer set of techniques for the AI community, so it's a much more rapidly evolving area of research.

Lex Fridman 14:11 Maybe we should also say the big two categories of training of pre-training and post-training. These umbrella terms that people use. So, what is pre-training and what is post-training, and what are the different flavors of things underneath post-training umbrella?

Nathan Lambert 14:27 Yeah. So pre-training, I'm using some of the same words that really get the message across is you're doing what is called autoregressive prediction to predict the next token in a series of documents. This is done over standard practices, trillions of tokens. So this is a ton of data that is mostly scraped from the web. In some of DeepSeek's earlier papers, they talk about their training data being distilled for math. I m- I shouldn't use this word yet, but taken from Common Crawl, and that's a public access that anyone listening to this could go download data from the Common Crawl website. This is a crawler that is maintained publicly. Yes, other tech companies eventually shift to their own crawler, and DeepSeek likely has done this as well as most frontier labs do. But this sort of data is something that people can get started with, and you're just predicting text in a series of documents. This is p- can be scaled to be very efficient, and there's a lot of numbers that are thrown around in AI training, like how many floating point operations or FLOPs are used, and then you can also look at how many hours of these GPUs, uh, that are used. And it's largely one loss function taken to a very large amount of (laughs) of compute usage. You just- you set up really efficient systems. And then at the end of that, you have the space model, and pre-training is where there is a lot more of complexity in terms of how the process is emerging or evolving and the different types of training losses that we will use. I think this is a lot of techniques grounded in the natural language processing literature. The oldest technique, which is still used today is something called instruction tuning or also known as supervised fine-tuning. These acronyms will be IFT or SFT, th- that people really go back and forth throughout them, and I will probably do the same, which is where you add this formatting to the model where it knows to take a question that is like, "Explain the history of the Roman Empire to me." And or something you'll- a sort of question you'll see on Reddit or Stack Overflow, and then the model will respond in a information dense but presentable manner. The core of that formatting is in this instruction tuning phase, and then there's two other categories of loss functions that are being used today. One, I will classify as preference fine-tuning. Preference fine-tuning is a generalized term for what came out of Reinforcement Learning from Human Feedback, which is RLHF. This Reinfor- reinforcement Learning from Human Feedback is credited as the technique that helped, uh, ChatGPT break through. It is a technique to make the responses that are nicely formatted, like these Reddit answers, more in tune with what a human would like to read. This is done by collecting pairwise preferences from actual humans out in the world to start, and now AIs are also labeling this data, and we'll get into those tradeoffs. And you have this kind of contrastive loss function between a good answer and a bad answer, and the model learns to pick up these trends. There's different implementation ways. You have things called reward models. You could have direct alignment algorithms. There's a lot of really specific things you can do, but all of this is about fine-tuning to human preferences. And the final stage is much newer and will link to what is done in R1 and these reasoning models is, I think, OpenAI's name for this, they had this new API in the fall which they called the Reinforcement Fine-Tuning API. This is the idea that you use the techniques of reinforcement learning, which is a whole framework of AI, there's a deep literature here. To summarize, it's often known as trial and error learning or the subfield of AI where you're trying to make sequential decisions in a certain potentially un- potentially noisy environment. There's a lot of ways we could go down that, but...... fine-tuning language models where they can generate an answer and then you check to see if the answer matches the true solution. For math or code, you have an ex- exactly correct answer for math, you can have unit tests for code. And what we are doing is we are checking the language models work, and we are giving it multiple opportunities on the same questions to see if it is right. And if you keep doing this, the models can learn to improve in verifiable domains. Uh, to a great extent, it works really well, it's a newer technique in the academic literature. It's been used at frontier labs in the US that don't share every detail, uh, for multiple years. So this is the idea of using reinforcement learning with language models, and it has been taking off, especially in this DeepSeek moment.

Lex Fridman 19:00 And we should say that there's a lot of exciting stuff going on on the, uh, again, across the stack, but the post-training probably this year, there's going to be a lot of interesting developments in the post-training. We'll, we'll talk about it. Uh, I almost forgot to talk about the, the difference between DeepSeek V3 and R1 on the user experience side. So forget the technical stuff, forget all of that. Just people that don't know anything about AI, they show up, like, what's the actual experience? What's the use case for each one when they actually, like, type and talk to it?

Nathan Lambert 19:29 Yeah.

Lex Fridman 19:29 What, what is each good at? That kind of thing.

Nathan Lambert 19:31 So let's start with DeepSeek V3 again. It's what more people would have tried something like it. You ask it a question, it'll start generating tokens very fast, and those tokens will look like a very human legible answer. It'll be some sort of markdown list. It might have formatting to help you draw to the core details in the answer, and it'll generate tens to hundreds of tokens. A token is normally a w- word for common words or a sub-word part in a longer word. And it'll look like a very high-quality Reddit or Stack Overflow answer. These models are really getting good at doing these across a wide variety of domains. Uh, even things that if you're an expert, things that are close to the fringe of knowledge, they will still be fairly good at. I think cutting edge AI topics that I do research on, these models are capable for study aid, and they're regularly updated. Where this changes is with the DeepSeek R1, what is called these reasoning models is when you see tokens coming from these models to start, it will be a large chain of thought process. We'll get back to chain of thought in a second, which looks like a lot of tokens where the model is explaining the problem. The model will often break down the problem and be like, "Okay, they asked me for this. Let's break down the problem. I'm going to need to do this." And you'll see all of this generating from a model. It'll come very fast in most user experiences. These APIs are very fast, so you'll see a lot of tokens, a lot of words show up really fast. It'll keep flowing on the screen, and this is all the reasoning process, and then eventually the model will change its tone in R1, and it'll write the answer where it summarizes its reading pro- reasoning process and writes a similar answer to the first types of model. But in DeepSeek's case, which is part of why this was so popular even outside the AI community, is that you can see how the language model is breaking down problems. And then you get this answer on a technical side, they, they train the model to do this specifically where they have a section which is reasoning and then it generates a special token which is probably hidden from the user most of the time, which says, "Okay, I'm starting the answer." So the model is trained to do this two-stage process on its own. If you use a similar model in, say, OpenAI, OpenAI's user interface is trying to summarize this process for you nicely by kind of showing the sections that the model is doing, and it'll kind of click through. It'll say, "Breaking down the problem, making X calculation, cleaning the result," and then the answer will come for something like OpenAI.

Lex Fridman 22:03 Maybe it's useful here to go through, like, an example of it, DeepSeek R1 reasoning.

Nathan Lambert 22:09 Yeah, so the, if, if you're looking at the screen here, what you'll see is a screenshot of the DeepSeek chat app. And at the top is Thought for 151 s- seven seconds with the drop-down arrow. Underneath that, if we were in an app that we were running, the drop-down arrow would have the reasoning.

Lex Fridman 22:24 So in this case, uh, the ques- the specific question which, you know, I'm philosophically/pothead inclined. So this is asking Deep, DeepSeek R1 for one truly novel insight about humans. And it reveals the reasoning and basically the trul- truly novel aspect is what's pushing the reasoning to constantly sort of the model asking itself, "Is this truly novel?" So it's actually challenging itself to be more novel, more counterintuitive, uh, more, uh, less cringe, I suppose. So some of the reasoning says, uh, this is just snapshots. "Alternatively, humans have a unique meta-emotion where they feel emotions about their own emotions, e.g. feeling guilty about being angry. This recursive emotional layering creates complex motivational drives that don't exist in other animals. The insight is that human emotions are nested." So it's like, it's reasoning through how humans feel emotions. It's reasoning about meta-emotions.

Nathan Lambert 23:21 Gonna have pages and pages of this.

Lex Fridman 23:22 Yeah, exactly.

Nathan Lambert 23:22 It's almost too much to actually read, but it's nice to skim as it's coming.

Lex Fridman 23:26 It's stream of con- it's a James Joyce-like stream of consciousness, and then it goes, "Wait, the user wants something that's not seen anywhere else. Let me dig deeper and consider the human ability to hold contradictory beliefs simultaneously. Cognitive dissonance is known, but perhaps the function is to allow flexible adaptation," so on and so forth. I mean, that really captures the public imagination that, holy shit, this isn't, uh, I mean, intelligent/almost like an inkling of sentience because, like, you're thinking through, you're self-reflecting, you're deliberating. And the final result of that after 157 seconds is, "Humans instinctively convert selfish desires into cooperative systems by collectively pretending abstract rules, money, laws, rights are real. These shared hallucinations act as 'games' where competition is secretly redirected to benefit the group, turning conflict into society's fuel."Pretty profound. I mean, you know comment to-

Nathan Lambert 24:31 This is a potential digression, but a lot of people have found that these reasoning models can sometimes produce much more eloquent text. Like, that is a, at least interesting example. I think depending on how open-minded you are, you find language models interesting or not, and there's a spectrum there.

Lex Fridman 24:48 Well, I mean, it's some of the, we'll talk about different benchmarks and so on, but some is just a vibe. Like that in itself is a f- y- let's say, quote, "fire tweet."

Nathan Lambert 24:57 Yeah.

Lex Fridman 24:58 If I (laughs) -

Nathan Lambert 24:59 (laughs)

Lex Fridman 24:59 ... if I'm trying to produce something, something that where people are like, "Oh, shit." Okay. So, that's chain of thought. We'll probably return to it more.

Lex Fridman 25:08 How were they able to achieve such low cost on the training and the inference? Maybe you could talk the training first.

Dylan Patel 25:15 Yeah. So there's, there's two main techniques that they implemented that are probably the majority of their efficiency, and then there's a lot of implementation details that maybe we'll gloss over or, or get into later that sort of contribute to it. But those two main things are, one, is they went to a mixture of experts model, uh, which, which we'll define in a second, and then the other thing is that they invented this new technique called MLA, latent attention. Both of these are, are big deals. Mixture of experts is something that's been in the literature for a handful of years, and OpenAI with GPT-4 was the first one to productize a mo- mixture of experts model. And what this means is when you look at the common models around, uh, that most people have been able to interact with that are open, right, think LLaMA. LLaMA is a dense model, i.e. every single parameter or neuron is activated as you're going through the model for every single token you generate, right? Now, with a mixture of experts model, you don't do that, right? How d- how does a human actually work, right? Is like, oh, well, my visual cortex is active when I'm thinking about, you know, vision tasks, or, um, like, you know, other things, right? My s- my amygdala is when I'm scared, right? These different aspects of your brain are focused on different things. A mixture of experts model attempts to approximate this to some extent. It's nowhere close to what a brain architecture is, but different portions of the model activate, right? You'll have a set number of experts in the model and a set number that are activated each time, and this dramatically reduces both your training and inference cost. Because now you're, you know, if you think about the parameter count as the sort of total embedding space for all of this knowledge that you're compressing down during training, when you're embedding this data in, instead of having to activate every single parameter every single time you're training or running inference, now you can just activate on a subset and the model will learn which expert to route to for different tasks. And so this is a humongous innovation in terms of, hey, I can continue to grow the total embedding space of parameters. And so DeepSeek's model is, you know, 600-something billion parameters, right? Uh, relative to LLaMA-405B, it's four or five billion parameters, right? Llama to... Relative to LLaMA-70B, it's 70 billion parameters, right? So this model technically has more embedding space for information, right, uh, to compress all of the world's knowledge that's on the internet down, but at the same time, it on- is only activating around 37 billion of the parameters. So only 37 billion of these parameters actually need to be computed every single time you're training data or inferencing data out of it, and so versus, versus, again, a LLaMA model, 70 billion parameters must be activated or 405 billion parameters must be activated. So you've dramatically reduced your compute cost when you're doing training and inference with this mixture of experts architecture.

Nathan Lambert 27:57 Should we break down where it actually applies and go into the transformer? Is that useful?

Lex Fridman 28:02 Let's go. Let's go into the transformer. (laughs)

Nathan Lambert 28:03 Cool. Okay. So the transformer is a thing that is talked about a lot, and we will not cover every detail. Uh, essentially, the transformer is built on repeated blocks of this attention mechanism and then a traditional dense, fully connected, multi-layer perception, whatever word you want to use for your normal neural network, and you alternate these blocks. There's other details. And where mixture of experts is applied is at this dense model. The dense model holds most of the weights if you count them in a transformer model. So, you can get really big gains from those mixture of experts on parameter efficiency, uh, training and inference because you get this efficiency by not activating all of these parameters.

Lex Fridman 28:44 We should also say that a transformer is a giant neural network.

Nathan Lambert 28:48 Yeah.

Lex Fridman 28:49 And then there's, for 15 years now, there's what's called the deep learning revolution. Network's gotten larger and larger. At a certain point, the scaling laws appeared where people realized-

Nathan Lambert 29:00 This is a Scaling Laws shirt, by the way. (laughs)

Lex Fridman 29:02 (laughs) Representing scaling laws, where it became more and more formalized that bigger is better across multiple dimensions of what bigger means. So, uh, and, but these are all sort of neural networks we're talking about, and we're talking about different architectures of how construct- to construct these neural networks such that the training and the inference on them is super efficient.

Nathan Lambert 29:24 Yeah. Every different type of model has a different scaling law for it, which is effectively for how much compute you put in, the architecture will get to different levels of performance at test tasks. And mixture of experts is one of the ones at training time, even if you don't consider the inference benefits, which are also big, at training time, your efficiency with your GPUs is dramatically improved by using this architecture if it is well-implemented. So you can get effectively the same performance model and evaluation scores with numbers like 30% less compute. I think there's gonna be a wide variation depending on your implementation details and stuff, but it is just important to realize that this type of technical innovation is something that gives huge gains, and I expect most companies that are serving their models to move to this mixture of experts' implementation. Historically, the reason why not everyone might do it is because it's a implementation complexity, especially when doing these big models. So this is one of the things that's, uh, DeepSeek gets credit for, is they do this extremely well. They do mixture of experts extremely well. This architecture for what is called DeepSeekMoE, MoE is the shortened version of mixture of experts, is multiple papers old. This-... part of their training infrastructure is not new to these models. Alone, and the same goes for what Dillon mentioned with multi-head latent attention, this is all about reducing memory usage during inference and same things during training by using some fancy low-rank approximation math. If you get into the details with this latent attention, it's one of those things that I look at and it's like, okay, this- they're doing really complex implementations because there's other parts of language models such as, uh, embeddings that are used to extend the context length. The common one that DeepSeek uses is rotary positional embeddings, which is called RoPE. And if you want to use RoPE with a normal MoE, it's kind of a sequential thing. You take these, you take two of the attention matrices and you rotate them by a complex value rotation, which is a matrix multiplication. With DeepSeek's MLA, with this new attention architecture, they need to do some clever things because they're not set up the same and it just makes the implementation complexity much higher. So they're managing all of these things, and these are probably the sort of things that OpenAI, these closed labs are doing. We don't know if they're doing the exact same techniques, but they actually shared them with the world, which is really nice to feel like this is the cutting edge of efficient language model training.

Guest 31:49 And some of this is, requires low level engineering, just i- is a giant mess and trickery. So as I understand, they went below CUDA. So they go super low programming of GPUs.

Dylan Patel 32:00 Effectively, NVIDIA builds this library called NCCL, right? Uh, in which, you know, when you're training a model, you have all these communications between every single layer of the model and you may have over 100 layers, right?

Nathan Lambert 32:12 What does NCCL stand for? It's N-C-C-L?

Dylan Patel 32:14 NVIDIA Communications Collectives Library.

Nathan Lambert 32:16 Yeah.

Guest 32:16 Nice. (laughs)

Nathan Lambert 32:17 (laughs)

Dylan Patel 32:17 Um, and so-

Guest 32:18 Damn. (laughs)

Dylan Patel 32:19 (laughs) When, when you're training a model, right, you're gonna have all these all-reduces and all-gathers, right? Uh, between each layer, between the, uh, multi-layer perception or feed-forward network and the attention mechanism you'll have, you'll have basically the model synchronized, right? Um, or you'll have all the, you'll have all-reducer and all-gather. Um, and, and this is a communication between all the GPUs in the network, whether, whether it's in training or inference. So NVIDIA has a standard library. This is one of the reasons why it's really difficult to use anyone else's hardware, uh, for training is because no one's really built a standard communications library. Um, and, and NVIDIA's done this at a sort of a higher level, right?Uh, DeepSeek, because they have certain limitations around the GPUs that they have access to, the interconnects are limited to some extent, um, by the restrictions of the GPUs that were shipped into China legally, not the ones that are smuggled, but legally shipped in, uh, that they used to train this model. They had to figure out how to get efficiencies, right? And one of those things is that instead of just calling the NVIDIA library NCCL, right, they instead created their, they scheduled their own communications, uh, which, which the la- some of the labs do, right? Um, Meta talked about in Llama 3 how they made their own custom version of NCCL. This is, they didn't, they didn't talk about the implementation details. This is some of what they did. Probably not as well as, maybe not as well as DeepSeek because DeepSeek ne- you know, necessity is the mother of innovation, and they had to do this. Whereas, uh, in the ca- you know, OpenAI has people that do this sort of stuff, Anthropic, et cetera. Uh, but, you know, DeepSeek certainly did it publicly and they may have done it even better because they were gimped on a certain aspect of the chips that they have access to. And so they scheduled communications o- um, you know, by scheduling specific SMs. SMs you could think of as like the core on a GPU, right? So there's hundreds of cores or there's, you know, a bit over 100 cores, SMs on a GPU and they were specifically scheduling, hey, which ones are running the model? Which ones are doing all-reduce? Which one are doing all-gather, right? And they would flip back and forth between them and this requires extremely low level programming.

Nathan Lambert 34:22 This is what NCCL does automatically or other NVIDIA libraries handle this automatically usually.

Dylan Patel 34:26 Yeah, exactly. And so technically they're using, you know, PTX which is like, sort of like you could think of it as like an assembly type language. It's not exactly that or instruction set, right? Like c- coding directly to assembly or instruction set. It's not exactly that, but uh, that's still part of technically CUDA but it's like do I want to write in Python, you know, PyTorch equivalent and call NVIDIA libraries? Do I wanna go down to the C level, right? Or uh, you know, encode even lower level or do I wanna go all the way down to the assembly or ISA level? And, and there are cases where you go all the way down there at the very big labs, but most companies just do not do that, right? Because it's a waste of time and the efficiency gains you get are not worth it. But DeepSeek's implementation is so complex, right? Especially with their mixture of experts, right? Um, people have done mixture of experts but they're generally eight, 16 experts, right? And they activate two. So you know, one of the words we like to use is like sparsity factor, right? Or usage, right? So, so you might have four, you know, one fourth of your model activate, right? And, and, and that's what Mistral's, uh, Mixtral model, right? Uh, their, their model that really catapulted them to like, oh my god, they're really, really good. Um, OpenAI has also had models that are MoE and, and, and so have all the other labs that are major closed. But what DeepSeek did that maybe only the leading labs have only just started recently doing is have such a high sparsity factor, right? It's not one fourth of the model, right? Two out of eight experts activating every time you go through the model. It's eight out of 256. And-

Nathan Lambert 35:51 There's different implementations for mixture of experts where you can have some of these experts that are ev- always activated, which this just looks like a small neural network and then all the tokens go through that and then they also go through some that are selected by this routing mechanism. And one of the innovations in DeepSeek's architecture is that they changed the routing mechanism in mixture of expert models. There's something called an auxiliary loss which effectively means during training you want to make sure that all of these experts are used across the tasks that the model sees. Why there can be failures in mixture of experts is that when you're doing this training, the one objective is token prediction accuracy. And if you just let turning go with a mixture of expert model on your own, it can be that the...... model learns to only use a subset of the experts, and in the MOE literature, there's something called the auxiliary loss, which helps balance them. But if you think about the loss functions of deep learning, this even connects to the bitter lesson, is that you want to have the minimum inductive bias in your model to let the model learn maximally. And this auxiliary loss, this balancing across experts, could be seen as intention with the prediction accuracy of the tokens. So, we don't know the exact extent that the DeepSeek MOE changed, which is instead of doing an auxiliary loss, they have an extra parameter in their routing, which after the batches they update this parameter to make sure that the next batches all have a similar use of experts. And this type of change can be big, it can be small, but they add up over time, and this is the sort of thing that just points to them innovating, and I'm sure all the labs that are training big MOEs are looking at these sort of things, which is getting away from the auxiliary loss. Some of them might already use it, but you just keep, you keep accumulating gains, and we'll talk about the philosophy of training and how you organize these organizations, and a lot of it is just compounding small improvements over time in your data, in your architecture, in your post-training, and how they integrate with each other. DeepSeek does the same thing, and some of them are shared, uh, a lot. We have to take them on face value, that they share their most important details. I mean, architecture and the weights are out there, so we're seeing what they're doing, and it adds up.

Dylan Patel 38:05 Going back to sort of the, like, efficiency and complexity point, right? It's 32, uh, versus a 4, right? For like mixed drawal and other MOE models that have been publicly released. So, this ratio is extremely high, and sort of what Nathan was getting at there was when you have such a different level of sparsity, um, you can't just have every GPU have the entire model, right? The model's too big, there's too much complexity there. So, you have to split up the model, um, with different types of parallelism, right? And so, you might have different experts on different GPU nodes, but now what, what happens when a to- you know, this set of data that you get, hey, all of it looks like this one way and all of it should route to one part of my, you know, model, right? Um, so, so when all of it route- routes to one part of the model, then you can have the, you can have this overloading of a cert- certain set of the GPU resources, or a certain set of the GPUs, and then the rest of the, the training network sits idle because all of the tokens are just routing to that. So, this is the biggest complexity, one of the big complexities with running a very, you know, sparse mixture of experts model, uh, i.e., you know, this 32 ratio versus this, uh, 4 ratio, is that you end up with so many of the experts just sitting there idle. So, how do I load balance between them? How do I schedule the communications between them? This is a lot of the, like, extremely low level detailed work that they figured out in the public first, and potentially, like, second or third in the world, and maybe even first in some cases.

Lex Fridman 39:29 What, uh, lesson do you, uh, in the direction of the bitter lesson, do you take from all of this? Where... Is this going to be the direction where a lot of the gain is going to be, which is this kind of low level optimization? Or is this a short-term thing where the biggest gains will be more on the algorithmic high level side of like post-training? Is, is this like a short-term leap because they've figured out like a hack because constraints, necessity is the mother of invention? Or is th- is there still a lot of gain?

Nathan Lambert 40:01 I think we should summarize what the bitter lesson actually is about, is that-

Lex Fridman 40:05 Okay.

Nathan Lambert 40:05 ... the bitter lesson, essentially, if you paraphrase it, is that the types of training that will win out in deep learning as we go are those methods that are, which are scalable in learning and search, is what it calls out. And this scale word gets a lot of attention in this. The interpretation that I use is effectively to avoid adding in the human priors to your learning process, and if you read the original essay, this is what it talks about, is how researchers will try to come up with what clever solutions to their specific problem that might get them small gains in the short term, while simply enabling these deep learning systems to work efficiently, and for these bigger problems in the long term, might be more likely to scale and continue to drive success. And therefore, we were talking about relatively small implementation changes to the mixture of experts model, and therefore it's like, okay, like, we will need a few more years to know (laughs) if one of these are actually really crucial to the bitter lesson. But the bitter lesson is really this long-term arc of how simplicity can often win, and there's a lot of sayings in the industry, like the models just want to learn. You have to give them the simple loss landscape where you put compute through the model and, and they will learn, and getting barriers out of the way.

Lex Fridman 41:29 Th- that's where the power of something like Nickel comes in, where standardized code that could be used by a lot of people to create s- sort of simple innovations that can scale, which is why the hacks, the, uh, I imagine the, the code base for DeepSeek is probably a giant mess.

Nathan Lambert 41:45 I'm sure they have... DeepSeek definitely has code bases that are extremely messy, where they're testing these new ideas, multi-headway inattention, probably start- could start in something like a Jupyter Notebook, where somebody tries something on a few GPUs, and that is really messy, but the stuff like trains of DeepSeek V3 and DeepSeek R1, those libraries, if you were to present them to us, I would guess are extremely high quality code, very

Lex Fridman 42:10 So, high quality, readable code.

Nathan Lambert 42:12 Yeah.

Dylan Patel 42:12 I, I think there is one aspect to note though, right? Is that there is the generally, general ability for that to transfer across different types of runs, right? You may make really, really high quality code for one specific model architecture at one size.

Nathan Lambert 42:27 Yeah.

Dylan Patel 42:27 And then that is not transferrable to, "Hey, when I, when I ch- make this architecture tweak, everything's broken again," right? Like, that's, that's something that could be, uh, you know, w- with their, with their specific low level coding of like scheduling SMs, is specific to this model architecture and size, right?... and, whereas, like, NVIDIA's collective library is more like, "Hey, it'll work for anything," right? You want to do an all reduce? Great. I don't care what your model architecture is, it'll work. Uh, and you're giving up a lot of performance when you do that, uh, in many cases. But it's, it's worthwhile for them to do the specific, uh, optimization for the specific run, given the constraints that they have regarding compute.

Lex Fridman 43:04 I wonder how stressful it is to, like, you know, these fr- frontier models, like, initiate training, like, to have the code-

Dylan Patel 43:12 Push the button.

Nathan Lambert 43:13 But that's-

Lex Fridman 43:13 To push the button?

Nathan Lambert 43:14 (laughs)

Lex Fridman 43:14 That, like, you're now spending a, a large amount of money and time to train this. Like, there must ... I mean, th- uh, th- there must be a lot of innovation on the debugging stage of, like, making sure there's no issues that you're monitoring and visualizing every aspect of the training, all that kind of stuff.

Dylan Patel 43:32 When, when people are training, they have all these various dashboards-

Nathan Lambert 43:35 Yeah.

Dylan Patel 43:35 ... but, like, the most simple one is your loss, right?

Nathan Lambert 43:38 Right.

Dylan Patel 43:38 Uh, and it continues to go down, but in reality, especially with more complicated stuff like MOE, the biggest problem with it, or FP8 training, which is another innovation, you know, going to a lower precision number format, i.e. less accurate, is that you end up with loss spikes, right? And, and no one knows why the loss spike happened. And for a long-

Nathan Lambert 43:55 Some of them, you do.

Dylan Patel 43:56 Some of them, you do.

Nathan Lambert 43:56 Some of them are bad data. Uh, can I give a AI2's example of what blew up our earlier models is a subreddit called Microwave Gang.

Lex Fridman 44:03 (laughs)

Nathan Lambert 44:03 We love to chalk those out. It's a real thing. You can pull up Microwave Gang. Essentially, it's a subreddit where everybody makes posts that are just the letter M, so it's like, "Mmm."

Dylan Patel 44:12 (laughs)

Nathan Lambert 44:12 So there's extremely long sequences of the letter M-

Lex Fridman 44:15 Yeah, yeah.

Nathan Lambert 44:15 ... and then the comments are like, "Beep, beep," 'cause it's from the microwave ends.

Lex Fridman 44:17 Yeah.

Nathan Lambert 44:18 But if you pass this into a model that's trained to be a normal producing text, it's extremely high loss, 'cause-

Lex Fridman 44:23 Yeah.

Nathan Lambert 44:23 ... normally you see an M-

Dylan Patel 44:24 (laughs)

Nathan Lambert 44:24 ... you don't predict Ms for a long time.

Lex Fridman 44:27 Yeah.

Nathan Lambert 44:27 So, like, this is something that causes the loss spikes for us. But when you have much like ... This is, this is old. This is not recent. And when you have more mature data systems, that's not the thing that causes the loss spike. And what Dillon is saying is true, but it's like ... It's, it's levels to this sort of idea.

Dylan Patel 44:41 Uh, with regards to the stress, right?

Nathan Lambert 44:43 (laughs)

Lex Fridman 44:43 (laughs)

Dylan Patel 44:44 These people are like ... You know, you'll go out to dinner with, like, a friend that works at one of these labs, and they'll just be l-

Nathan Lambert 44:49 (laughs)

Dylan Patel 44:49 They'll just be, like, looking at their phone every, like, 10 minutes, and they're not, like ... You know, it's one thing if they're texting, but they're just like-

Lex Fridman 44:55 Yeah.

Dylan Patel 44:55 ... like, "Is the loss, is the loss per okay?"

Nathan Lambert 44:56 Yeah. It, it's literally like, "Is the loss per okay?"

Dylan Patel 44:57 Is the loss per okay?

Nathan Lambert 44:57 Tokens per, tokens per second. Loss not blown up. They're just walk- watching this. (laughs)

Lex Fridman 45:02 And the heart rate goes up if there's a spike.

Dylan Patel 45:05 And some level of spikes is normal, right? It'll, it'll recover and be back. Sometimes a lot of the old strategy was like, you just stop the run, restart from a old version, and then, like, change the data mix, and then it keeps going.

Nathan Lambert 45:16 There are even different types of spikes. So, Dirk Groeneveld has a theory at AI2 that's, like, fast spikes and slow spikes, where there are ... Sometimes when you're looking at the loss and there are other parameters, you can see it start to creep up, and then blow up, and that's really hard to recover from, so you have to go back much further. So you have this stressful period where it's, like, flat or it might start going up, and you're like, "What do I do?" Whereas there are also loss spikes that are ... It looks good, and then there's one spiky data point. And what you could do is you just skip those. You, you see that there's a spike. You're like, "Okay, I can ignore this data. Don't update the model, and do the next one, and it'll recover quickly." But these, like ... On trickier implementations, so as you get more complex in your architecture and you scale up to more GPUs, you have more potential for your loss blowing up. So it's like there's, there's-

Dylan Patel 45:59 And then, and then-

Nathan Lambert 45:59 ... there's a distribution.

Dylan Patel 46:00 The whole idea of grokking also comes in, right? It's like just because it slowed down from improving in loss doesn't mean it's not learning, because all of a sudden, it could be like this, and then it could just spike down in loss again, because it learned, truly learned something, right? Uh, and it took some time for it to learn that. It's not like a gradual process, right? And that's, that's what humans are like. That's what models are like. So it's, it's really a stressful task, as you mentioned. (laughs)

Lex Fridman 46:21 And the whole time, the, the, the dollar count is going up.

Nathan Lambert 46:24 Every company has failed runs. You need failed runs to push the envelope on your infrastructure. So, a lot of news cycles are made of, "X company had Y failed run." Every company that's trying to push the frontier of AI has these. So, it is ... Yes, it's noteworthy because it's a lot of money and it can be week to month setback, but it is part of the process.

Lex Fridman 46:44 But how do you get ... If you're DeepSeek, how do you get to a place where, holy shit, there's a successful combination of hyper parameters?

Nathan Lambert 46:52 A lot of small failed runs. (laughs)

Lex Fridman 46:54 And so, so rapid, uh, iteration through failed runs-

Nathan Lambert 46:58 Yeah.

Lex Fridman 46:58 ... until-

Nathan Lambert 46:59 And s- and successful ones. You just-

Lex Fridman 47:01 And then you build up some intuition like this, this mixture of expert works, and then, uh, th- this implementation of MLA works.

Nathan Lambert 47:09 Key hyper parameters, like learning rate and regularization, and things like this, and you find the regime that works for your code base. I've ... Talking to people at Frontier Labs, there's a story that you can tell, where training language models is kind of a path that you need to follow. So you need to, like, unlock the ability to train a certain type of model or a certain scale, and then your code base and your internal know-how of which hyper parameters work for it is kind of known. And I ... You look at the DeepSeek papers and models, they've, they've scaled up. They've added complexity. And it's just continuing to build the capabilities that they have.

Dylan Patel 47:42 There, there's the concept of a YOLO run.

Lex Fridman 47:44 (laughs)

Nathan Lambert 47:44 (laughs)

Dylan Patel 47:45 Um, so YOLO, you only live once.

Lex Fridman 47:46 Yep.

Dylan Patel 47:46 Um, and, and what it is is like, you know, there's, there's, there's all this experimentation you do at the small scale, right? Uh, research oblations, right? Like, you have your Jupyter notebook where you're com- experimenting with MLA on, like, three GPUs or whatever, um, and you're doing all these different, uh, things, like, "Hey, do I do four expert, four active experts, 128 experts? Do I arrange the experts this way?" You know, all these different, uh, model architecture things you're testing at a very small scale, right? Couple researchers, few GPUs, tens of GPUs, hundreds of GPUs, whatever it is. And then all of a sudden you're like, "Okay, guys, no more, no more fucking around," right? Uh, "No more screwing around. Everyone take all the resources we have, let's pick what we think will work, and just go for it," right?

Lex Fridman 48:25 Yeah.

Dylan Patel 48:25 YOLO. And this is where that sort of stress comes in is like, "Well, I know it works here, but some things that work here don't work here, and some things that work here don't work down here," right?

Lex Fridman 48:35 Yeah.

Dylan Patel 48:35 In terms of scale, right? So it's, it's, it's really truly a YOLO run. And, and sort of, like ... There's this, like, like discussion of, like, certain researchers just have, like, this methodical nature. Like, they can find the whole search space and, like, figure out all the oblations of different research and really see what is best. And there's certain researchers who just kind of, like-... you know, have that innate gut instinct of, like, "This is the YOLO run." Like, you know-

Nathan Lambert 48:57 Yeah, I look rad for most of the year.

Dylan Patel 48:57 ... "Looking at the data, I think this is it." Uh-

Nathan Lambert 49:00 This is why you wanna work in post-training, because the GPU cost for training is lower, so you can make a higher percentage of your training runs YOLO runs.

Lex Fridman 49:06 Yeah.

Dylan Patel 49:07 For, for now. (laughs)

Lex Fridman 49:07 Yeah, for now.

Nathan Lambert 49:08 For now, for now. (laughs)

Lex Fridman 49:10 So some of this, uh, is fundamentally luck, still.

Dylan Patel 49:14 Luck is skill, right? In many cases.

Lex Fridman 49:16 Yeah. I mean, it looks lucky, right, when you're-

Dylan Patel 49:18 But the hill to climb if you're owned one of these labs and you have an evaluation and you're not crushing, there's a repeated playbook of how you improve things. There are localized improvements, which might be data improvements, and these add up into the whole model just being much better. And when you zoom in really close, it can be really obvious that this model's just really bad at this thing and we could fix it, and you just add these up. So, like, some of it feels like luck, but on the ground, especially with these new reasoning models we're talking to, it's just so many ways that we can poke around. And normally it's that some of them give big improvements.

Nathan Lambert 49:51 The, the search base is near infinite, right? And, and yet the amount of compute and time you have is, is very low and you're sh- you're- you have to hit s- release schedules. You have to not get blown past by everyone. Otherwise, you know, what happened with DeepSeek, you know, crushing meta and Mistral and Cohere and all these guys, they moved too slow, right? They, they maybe were too methodical. I don't know, they didn't hit the YOLO run. Whatever the reason was, maybe they weren't as skilled. Uh, whatever, what- you know, you can call it luck if you want, but at end- at the end of the day, it's skill.

Lex Fridman 50:18 So 2025 is the year of the YOLO run. It seems like all the labs are, like, going in.

Dylan Patel 50:25 I, I, I think it's even more impressive what OpenAI did in 2022, right? At the time, no one believed in mixture of experts models, right, at Google, uh, who had all the researchers. Uh, OpenAI had such little compute and they devoted all of their compute for many months, right, all of it, 100% for many months to GPT-4 with a brand new architecture, with no belief that, hey, let me spend a couple 100 million dollars, which is all of the money I have, on this model, right? That is truly YOLO.

Lex Fridman 50:54 Yeah.

Dylan Patel 50:54 Right?

Lex Fridman 50:55 (laughs) Yeah.

Dylan Patel 50:55 Um, now, now, you know, people are like, all these, like, training run failures that are in the media, right? It's like, okay, great, but, like, actually, m- a lot, a huge chunk of my GPUs are doing inference. I still have a bunch doing research constantly. And yes, my biggest cluster is training, but, like, on, on this YOLO run, but, like, that YOLO run is much less risky than, like, what OpenAI did in 2022. Or maybe what DeepSeek did now, or, you know, like, sort of like, Hey, we're just gonna throw everything at it.

Lex Fridman 51:19 The big winners throughout human history are the ones who are willing to do YOLO at some point.

Lex Fridman 51:25 Okay. Uh, what do we understand about the hardware it's been trained on? DeepSeek.

Dylan Patel 51:30 DeepSeek is very interesting, right? This is where, second to take us to zoom out outta who they are first of all, right? HighFlyer is a hedge fund that has historically done quantitative trading in China as well as elsewhere, and they have always had a significant number of GPUs, right? In the past, a lot of these high frequency trading, algorithmic quant traders used FPGAs, uh, but it's shifted to GPUs definitely, and, and there's both, right? But GPUs especially, and Deep- and, and HighFlyer, which is the hedge fund that owns DeepSeek and everyone who works for DeepSeek is part of HighFlyer to some extent, right? Uh, same, same parent company, same owner, same CEO. They had all these resources and infrastructure for trading, and then they devoted a humongous portion of them to training models, uh, both language models and otherwise, right? Because these, these, these tell- techniques were heavily AI influenced. Um, you know, more recently people have, you know, realized, hey, trading with, um, you know, like even, even when you go back to, like, Renaissance and all these r- uh, all these, like, quantitative firms, natural language processing is the key to, like, trading really fast, right? Understanding a press release, uh, and making the right trade, right? And so DeepSeek has always been really good at this. And even as far back as 2021, they, they have press releases and papers saying like, "Hey, we're the first company in China with an A100 cluster this large."

Lex Fridman 52:47 Okay.

Dylan Patel 52:47 It was 10,000 A100 GPUs, right? This is, this is in 2021. Now, this wasn't all for training, you know, large language models. This was mostly for training models for their quantitative aspects, their, uh, quantitative trading, as well as, you know, a lot of that was natural language processing to be clear, right? Um, and so this is the sort of history, right? So verifiable fact is that in 2021 they built the largest Chinese, uh, cluster, at least they claim it was the largest cluster in China, 10,000 GPUs.

Nathan Lambert 53:12 Before expert controls started.

Dylan Patel 53:14 Yeah. It's like-

Nathan Lambert 53:15 They've had a huge cluster before any conversation of expert controls.

Dylan Patel 53:18 So then you step it forward to like, what have they done over the last four years since then? Right? Um, obviously they've continued to operate the hedge fund, probably make tons of money. And the other thing is that they've leaned more and more and more into AI. The CEO, Lian Chengfeng, uh, Lian-

Nathan Lambert 53:33 You're not putting me on the spot on this. We discussed this before. (laughs)

Dylan Patel 53:35 Yeah, we were. Lian Feng, right, the CEO-

Nathan Lambert 53:38 We're all fans. (laughs)

Dylan Patel 53:39 ... he owns maybe... Lian Feng, he run- r- he owns maybe a little bit more than half the company allegedly, right? Um, is an extremely, like, Elon, Jensen kind of figure where he's just, like, involved in everything, right?

Nathan Lambert 53:50 Mm-hmm.

Dylan Patel 53:50 Um, and so over that time period, he's gotten really in-depth into AI. He actually has a bit of a, like a, if you- if you see some of his statements, a bit of an e/acc vibe almost, right?

Nathan Lambert 53:59 Total AGI vibes and like, "We need to do this. We need to make a new ecosystem of open AI. We need China to lead on this sort of ecosystem, because historically the Western countries have led on th- on software ecosystems." And in or- he straight up acknowledges like, "In order to do this, we need to do something different." DeepSeek is his way of doing this. The- the- some of the translated interviews with him are fantastic.

Lex Fridman 54:23 So he has done interviews?

Nathan Lambert 54:24 Yeah.

Lex Fridman 54:24 Do you think he would do a Western interview or no? Or is there controls on the Chinese-

Nathan Lambert 54:28 There hasn't been one yet, but-

Lex Fridman 54:29 Okay.

Nathan Lambert 54:30 ... I would try it. (laughs)

Lex Fridman 54:31 All right. Well, j- I just got a Chinese translator, so it's great. This is, this is el push. So fascinating figure, engineer pushing full on into AI, leveraging the success from the high frequency trading.

Nathan Lambert 54:44 Very direct quotes like, "We will not switch to closed source," when asked about this stuff.

Dylan Patel 54:49 Oh.

Nathan Lambert 54:49 He, very long term motivated in how the ecosystem of AI should work. And I think from a Chinese perspective, he wants the Chinese company, a, a Chinese company to build this vision.

Dylan Patel 55:03 And so this is sort of like the "visionary" behind the company, right? This hedge fund still exists, right? This, this quantitative firm. And so-Deep Seek is the sort of at- at, you know, slowly he got turned to this full view of, like, AI everything about this, right? But at some point it slowly maneuvered and you made Deep Seek, um, and Deep Seek has done multiple models since then. They've acquired more and more GPUs. They share infrastructure with the fund, right? Um, and so, you know, there is no exact number of public GPU resources that they have, but besides this 10,000 GPUs that they bought in 2021, right? And they were fantastically profitable, right? And then this paper claims they did only 2,000 H800 GPUs, which are a restricted GPU that was previously allowed in China, but no longer allowed, and there's a new version. But it's basically NVIDIA's H100 for China, right? Um, and th- there's some restrictions on it, specifically around the communications, uh, sort of, uh, speed, th- the interconnect speed, right, which is why they had to do this crazy SM, you know, scheduling stuff, right? So, so going back to that, right, it's like this is obviously not true in terms of their total GPU count.

Lex Fridman 56:08 Obvious available GPUs, but for this training run you think 2,000 is the correct number or no?

Dylan Patel 56:14 So this is where it takes, um, you know, significant amount of sort of like zoning in, right? Like what do you call your training run, right? Do you count all of the research and ablations that you ran, right? Picking all this stuff, because yes, you can do a YOLO run, but at some level you have to do the test at the small scale, and then you have to do some tests at medium scale before you go to a large scale.

Nathan Lambert 56:33 A- accepted practice is that for any given model that is a notable advancement, you're gonna do two to 4X compute of the full training run in experiments alone.

Lex Fridman 56:43 So a lot of this compute that's being scaled up is probably used in large part at this time for research.

Dylan Patel 56:49 Yeah, and research will, you know, research begets the new ideas that let you get huge efficiency.

Nathan Lambert 56:53 Right. Research gets you 01.

Dylan Patel 56:54 Right.

Nathan Lambert 56:54 Uh, like research gets you breakthroughs (laughs) , and you need to bet on it.

Lex Fridman 56:57 So some of the pricing strategy they will discuss has the research baked into the price.

Dylan Patel 57:02 So the numbers that Deep Seek specifically said publicly, right, are just the 10,000 GPUs in 2021 and then 2,000 GPUs for only the pre-training for V3.

Lex Fridman 57:12 Mm-hmm.

Dylan Patel 57:12 They did not discuss cost on R1. They did not discuss cost on all the f- other RL, right, for this instruct model that they made, right? They only discussed the pre-training for the base model, and they did not discuss anything on research and ablations. And they do not talk about any of the resources that are shared in terms of, "Hey, the fund is using all these GPUs," right? And, and we know that they're very profitable in that 10,000 GPUs in, in, in, in 2021. So, so the, uh, some, some of the research that we've found is that we actually believe they have closer to 50,000 GPUs.

Lex Fridman 57:43 We as SEMI Analysis.

Dylan Patel 57:44 Yeah.

Lex Fridman 57:44 Oh, we should say that you're, uh, sort of one of the world experts in figuring out what everybody's doing in terms of the semiconductor, in terms of cluster build outs, in terms of, like, who's doing what in terms of training runs. So yeah, so that's the we. Okay, go ahead.

Dylan Patel 57:59 Yeah. Sorry, sorry. Uh, we believe they actually have something closer to 50,000 GPUs, right? Now this is, this is split across many tasks, right? Again, the fund, um, research, and ablations.

Nathan Lambert 58:09 For ballpark, how much would Open AI or Anthropic have? I think the clearest example we have, because Meta is also open, they talk about, like, order of 60K to 100K H100 equivalent GPUs in their training clusters.

Dylan Patel 58:21 Right, so, so, like, LLaMA 3, they said th- they trained on 16,000 H100s, right? But the company of Meta last year publicly disclosed they bought like 400-something thousand GPUs.

Nathan Lambert 58:30 Yeah. (laughs)

Dylan Patel 58:30 Right? So, so of course tiny percentage on the training. Again, like most of it is, like, serving me the best Instagram Reels, right? Um, or whatever, right?

Nathan Lambert 58:37 I mean, we could get into a cost of, like, what is the cost of ownership for a 2,000 GPU cluster, 10,000... Like, the, the... There's just different sizes of companies that can afford these things, and Deep Seek is reasonably big. I... Their compute allocation compared t- is one of the top few in the world. It's not Open AI, Anthropic, et cetera, but

Nathan Lambert 58:57 they have a lot of compute.

Lex Fridman 58:58 Can you in general actually just zoom out and also talk about the, the Hopper architecture, the NVIDIA Hopper GPU architecture and the difference between H100 and H800, like you mentioned the interconnects before?

Dylan Patel 59:09 Yeah, so there's, you know, Ampere was the A100 and then H100 Hopper, right? People use them h- synonymously i- in the US because really there's just H100 and now there's H200, right? But same thing, uh, mostly. In China, they've had tu- there have been different salvos of export restrictions. So initially the US government limited on a two factor scale, right, which is chip interconnect versus, uh, flops, right? So any chip that had interconnects above a certain level and FLOPS abo- above a certain... Floating point operations above a certain level was restricted. Uh, later the government realized that this was a flaw in the restriction, and they cut it down to just floating point operations. And so, um-

Nathan Lambert 59:48 So H- H800 had high FLOPS, low com- communication?

Dylan Patel 59:51 Exactly, so the H800 was the same performance as H100 on FLOPS, right? But it didn't ha- it had, it just had the interconnect bandwidth cut. Deep Seek knew how to utilize this res- you know, "Hey, even though we're cut back on the interconnect, we can do all this fancy stuff to figure out how to use the GPU fully anyways," right?

Lex Fridman 1:00:09 Mm-hmm.

Dylan Patel 1:00:09 And, and so that was back in October 2022, but, uh, later in 2023, end of 2023, implemented in 2024, the US government banned the H800, right? Um, and so by the way, this H800 cluster, these 2,000 GPUs, was not even purchased in 2024, right? It was purchased in late 2023. Um-

Lex Fridman 1:00:27 Mm-hmm.

Dylan Patel 1:00:27 ... and they're just getting the model out now, right, because it takes a lot of research, et cetera. Um, H800 was banned, and now there's a new chip called the H20. Uh, the H20 is, uh, cut back on only FLOPS, but the interconnect bandwidth is the same, and in fact in some ways it's better than the H100 because it has better memory bandwidth and memory capacity. So there are, you know, NVIDIA is working within the constraints of what the government says and then get- builds the best possible GPU for China.

Lex Fridman 1:00:52 Can we take this actual tangent and we'll return back to the hardware, is the, the philosophy, the mo- the motivation, the case for export controls, what is it? Uh, Dariuh Madedj just published a blog post about export controls. The case he makes is that if AI becomes super powerful, and he says by 2026 we'll have AGI or super powerful AI, and that's going to give a significant... Whoever builds that will have a significant military advantage.And so, because the United States is a, is a democracy, and as he l- says, China is, uh, authoritarian, or has authoritarian elements, you want a unipo- polar world with a super powerful military because of the AI is one that's a democracy. It's a much more complicated world geopolitically when you have two superpowers with super powerful AI, and one is authoritarian. So, that's the case he makes, and so we wanna, uh, the United States wants to use export controls to slow down to make sure th- (laughs) th- that, uh, China can't do these gigantic, uh, training runs that would be presumably required to build AGI.

Nathan Lambert 1:02:02 This is very abstract. I think this can be the goal of s- how some people describe export controls is this super powerful AI. There's... And you touched on the training run idea. There's not many worlds where China cannot train AI models. The export controls are kneecapping the amount of compute or the density of compute that China can have, and if you think about the AI ecosystem right now as all of these AI companies' revenue numbers are up and to the right. Their AI usage is just continuing to grow. More GPUs are going to inference. A large part of export controls, if they work, is just that the amount of AI that can be run in China is going to be much lower. So, on the training side, DeepSeek V3 is a great example, which you have a very focused team that can still get to the frontier of AI on... This 2,000 GPUs is not that hard to get, all considering in the world. They're still gonna have those GPUs. They're still gonna be able to train models, but if there's gonna be a huge market for AI, if you have strong export controls and you wanna have 100,000 GPUs just serving the equivalent of ChatGPT clusters, with good export controls it also just makes it so that E- AI can be used much less. And I think that is a much easier goal to achieve than trying to debate on what AGI is, and if you have these extremely intelligent autonomous AIs and data centers, like, those are the things that could be running in these GPU clusters in the United States, but not in China.

Dylan Patel 1:03:30 To some extent, training a model does effectively nothing, right?

Nathan Lambert 1:03:33 Yeah. (laughs)

Dylan Patel 1:03:33 Like, they have a model. The, the thing that Dario is sort of speaking to is the implementation of that model, once trained, to then create huge economic growth, huge increases in military capabilities, huge capability- uh, increases in productivity of people, uh, betterment of lives. Whatever, whatever you want to direct super powerful AI towards, you can, but that requires a significant amount of compute, right? And so, the US government has effectively said, um... And, and, and, and forever, right? Like, train- training will always be a portion of the total compute. Um, you know, we mentioned Meta's 400,000 GPUs. Only 16,000 made LLaMA, right? So, the, the percentage that Meta's dedicating to inference... Now this might be for recommendation systems that are trying to hack our mind into spending more time and watching more ads, or if it's, if it's, uh, or if it's for a super powerful AI that's doing productive things. Doesn't matter about the exact use that our, you know, economic system decides. It's that that can be delivered whatever, in whatever way we want. Whereas with China, right, you know, you're, you know, export restrictions, great. You're never gonna be able to cut everything off, right? Uh, and that's, that's like... I think that's quite well-understood by the US government, uh, is that you can't cut everything off. Um, you know-

Nathan Lambert 1:04:40 And they'll make their own chips. They're, they're

Dylan Patel 1:04:42 And, and they're trying to make their own chips. They'll be worse than ours, but, you know, this is... The whole point is to just keep a gap, right?

Nathan Lambert 1:04:47 Yeah.

Dylan Patel 1:04:47 Um, and therefore at some point as the AI... You know, in a world where 2, 3% economic growth... This is really dumb, by the way, right? To cut off, uh, you know, high tech and not make money off of it. But in a world where super powerful AI comes about and then starts creating significant changes in society, which is what all the AI leaders and big tech companies believe, I think. Super powerful AI is gonna change society massively, and therefore, this compounding effect of the difference in compute is really important. There's some sci-fi out there where, like, AI is ta- is, like, measured in the power of l- in, like, how much power is delivered to compute, right? Or how much, uh, is being... You know, that's sort of a way of thinking about what's the economic output is just how much power are you directing towards that AI.

Nathan Lambert 1:05:26 Should we talk about reasoning models with this as a way that this might be actionable as something that people can actually see? So the reasoning models that are coming out with R1 and O1, they're designed to use more compute. There's a lot of buzzy words in the AI community about this, test-time compute, inference-time compute, whatever, but, um, Dillon has good research on this. You can get to the specific numbers on the ratio of when you train a model you can look at things about the compute, amount of compute used at training and amount of compute used at inference. These reasoning models are making inference way more important to doing complex tasks. In the fall, in December, their OpenAI announced this O3 model. There's another thing in AI when things move fast. We get both announcements and releases. Announcements are essentially blog posts where you pat yourself on the back and you say you did things, and releases are when the model's out there, the paper's out there, et cetera. So OpenAI has announced O3, and we can check if O3 mini is out-

Dylan Patel 1:06:17 Mm-hmm.

Nathan Lambert 1:06:17 ... as of recording potentially, but that doesn't really change the point, which is that the breakthrough result was something called ARC AGI task, which is the abstract reasoning corpus, a task for artificial general intelligence. Um, François Chollet is the guy who's been... It's, it's a multi-year-old paper. It's a brilliant benchmark, and the number for OpenAI O3 to solve this was that it used a thu- some sort of number of samples in the API. The API has, like, thinking effort and number of samples. They used 1,000 samples to solve this task, and it comes out to be, like, five to $20 per question, which you're, you're putting in, effectively, a math puzzle, and then it takes orders of dollars to answer one question. And this is a lot of compute. If this is gonna take off in the US, OpenAI needs a ton of GPUs on inference to capture this. They have this o- um, OpenAI ChatGPT Pro subscription, which is $200 a month, but they-

Dylan Patel 1:07:09 Which Sam said they're losing money on.

Nathan Lambert 1:07:11 Which means that people are burning a lot of GPUs on inference, and I, I've signed up with it. I've played with it. I don't think I'm a power user, but I, I, I use it, and it's like that is the thing that a Chinese company with-... mediumly strong expert controls, there will always be loopholes, might not be about to do it all. And if that, i- i- the main result for O3 is also a spectacular coding performance and if that feeds back into AI companies being able to experiment better.

Lex Fridman 1:07:37 So presumably the idea is for an AGI, a much larger fraction of the compute will be used for this test on compute for the reasoning. For the AGI goes into a room and thinks about how to take over the world and then, you know, come back in 2.7 hours-

Nathan Lambert 1:07:54 This is what it's gonna-

Lex Fridman 1:07:55 ... and, and that it's gonna take a lot of compute.

Nathan Lambert 1:07:56 This is what people at, like, CEO, or leaders of OpenAI and Anthropic talk about, is, like, autonomous AI models, which is you give them a task and they work on it in the background. I think my personal definition of AGI is much simpler. Like, I think language models are a form of AGI and all this super powerful stuff is a next step that's great if we get these tools. But a language model has so much value in so many domains, it is a general intelligence to me. But this next step of agentic things where they're independent and they can do tasks that aren't in the training data is what the m- few year outlook that these AI companies are driving for.

Lex Fridman 1:08:32 I think the terminology here that Dari- Dario uses a super powerful AI, so I agree with you on the AGI, I think we already have something like this exceptionally impressive that Alan Turing would for sure say is AGI. Uh, but he's referring more to something once in possession of, then you would have a significant military and geopolitical advantage over other nations. So it's not just like you can ask it how to cook an omelet.

Nathan Lambert 1:08:58 And he has a much more positive view in his essay, Machines of Love and Grace.

Lex Fridman 1:09:01 Yes. Yeah.

Nathan Lambert 1:09:02 I've read into this, I don't have enough background in physical sciences to gauge exactly how confident I am in if AI can revolutionarize biology, but I'm safe saying that AI is going to accelerate the progress of any computational science.

Lex Fridman 1:09:16 So we're doing a depth first search here on topics, uh, taking tangent of a tangent, so let's continue, uh, on that depth first search. Uh, y- you said that you're both feeling the AGI.

Nathan Lambert 1:09:28 (laughs)

Lex Fridman 1:09:28 So you're, what's, what's your timeline? Dario's 2026 for the super powerful AI that's g- you know, that's basically agentic to a degree where it's a real security threat.

Nathan Lambert 1:09:41 (laughs)

Lex Fridman 1:09:41 That level of AGI. What's your, what's your timeline?

Nathan Lambert 1:09:44 I don't like to attribute specific abilities because predicting specific abilities and when is very hard. I think mostly if you're going to say that I'm feeling the AGI is that I expect continued rapid surprising progress over the next few years. So something like R1 is less surprising to me from DeepSeek because I, I expect there to be new paradigms where substantial progress can be made. I think DeepSeek R1 is so unsettling because we're kind of on this path with, with ChatGPT, it's like it's getting better, it's getting better, it's getting better. And then we have a new direction for, for changing the models and we took one step like this, and we, like, took a step up. So it looks like a really fast sto- slope and then we're gonna just take more steps. So I guess it's really unsettling when you have these big steps and I expect that to keep happening. I see, um, I've tried OpenAI Operator, I've tried Claude computer use, they're not there yet. I understand the idea, but it's just so hard to predict what is the breakthrough that'll make something like that work. And I think it's more likely that we have breakthroughs that work in things that we don't know what they're gonna do. So everyone wants agents, Dario has a very eloquent way of describing this, and I just think that it's, like, there's gonna be more than that, so, like, just expect these things to come.

Lex Fridman 1:10:53 I'm gonna have to try to pin you down to a date on the AGI timeline. Uh, (laughs) like, the nuclear weapon moment, so moment where, on the geopolitical stage, there's a real, like, you know, 'cause we're talking about export controls, when do you think, just even to throw out a date, when do you think that would be? Like, for me, it's probably after 2030. So I'm not as-

Nathan Lambert 1:11:19 That's what I would say.

Dylan Patel 1:11:20 So, so define that, right? Because to me it kind of almost has already happened, right? You look at elections in India and Pakistan, people get AI voice calls and think they're talking to the politician, right? The AI diffusion rules which was enacted in the last couple weeks of the Biden admin and looks like the Trump admin will keep and potentially even strengthen, limit cloud computing and GPU sales to countries that are not even related to China. It's like this is, this-

Nathan Lambert 1:11:44 Portugal and all these, like, normal comp- countries are on the-

Dylan Patel 1:11:47 Yeah. It's like-

Nathan Lambert 1:11:47 ... you need approval from the US list.

Dylan Patel 1:11:49 Like, yeah, Portugal and, like, you know, like, like all these countries that are allies, right? Singapore, right? Like they, they freaking have F-35s and we don't let them buy GPUs.

Nathan Lambert 1:11:57 Mm-hmm.

Dylan Patel 1:11:58 Like this is, this to me is already to the scale of, like, you know...

Lex Fridman 1:12:01 Well, that just means that, uh, the US military is really nervous about this new technology. That doesn't mean the technology is already there. So, like, they might be just very cautious about this thing that they don't quite understand. But that's a really good point, sort of the, the robocalls, swarms of semi-intelligent bots could be a weapon, could be doing a lot of social engineering.

Dylan Patel 1:12:25 I mean there's tons of talk about, you know, from the 2016 elections, like Cambridge Analytica and all this stuff, Russian influence. I, I mean every country in the world is pushing stuff onto the internet and has narratives they want, right? Like that's, th- every, every, like, technically competent whether it's Russia, China, US, Israel, et cetera, right? You know, people are pushing viewpoints onto the internet en masse. And language models crash the cost of, like, very intelligent sounding language.

Nathan Lambert 1:12:48 There's some research that shows that the distribution is actually a limiting factor. So language models haven't yet made misinformation particularly, like, changed the equation there. The internet is still ongoing. I think there's a blog, AI Snake Oil, and some of my friends at Princeton that write on this stuff. So there is research. It's like, it's a default that everyone assumes and I would have thought the same thing is that misinformation is gonna get far worse with language models. I think in terms of internet posts and things that people have been measuring, it hasn't been a exponential increase or something extremely measurable. And things you're talking about with voice calls and stuff like that, it could be in modalities that are harder to measure.... so it's, it's something that it's too soon to tell in terms of, I think that's like political instability via the web is very, it's m- it's monitored by a lot of researchers to see what's happening. I think the, that you're asking about like the AGI thing. I, my, if, if you make me give a year, I would be like, "Okay, I have AI CEO saying this, they've been saying two years for a while."

Dylan Patel 1:13:50 Mm-hmm.

Nathan Lambert 1:13:50 I think that there are people like Dario a- Anthropic, the CEO, had thought about this so deeply. I need to take their work seriously, but also understand that they have difference of, different (laughs) incentives, so I feel like add a few years to that, which is how you get something similar to 2030 or a little after 2030.

Dylan Patel 1:14:08 I think to some extent we have capabilities that hit a certain point where any one person could say, "Oh, okay, if I can le- leverage those capabilities for X amount of time, this is AGI," right? Call it 27, 28. But then the cost of actually operating that capability-

Nathan Lambert 1:14:23 Yeah, this is gonna be my point.

Dylan Patel 1:14:24 ... is so, so extreme that no one can actually deploy it at scale en masse to actually completely revolutionize the economy on a click and a snap of a finger. So I don't think it will be like a snap of the finger moment-

Nathan Lambert 1:14:35 Yeah. It's the physical constraint timeline.

Dylan Patel 1:14:36 Rather it'll be a, you know, "Oh, the capabilities are here, but I can't deploy it everywhere," right? And so one, one simple example going back sort of to 2023 was when, uh, you know, Bing with GPT-4 came out and everyone was freaking out about search, right?

Nathan Lambert 1:14:51 (laughs) Oh, my gosh.

Dylan Patel 1:14:51 Perplexity came out.

Nathan Lambert 1:14:52 (laughs)

Dylan Patel 1:14:52 If you did the cost on like, hey, implementing GPT-3 into every Google search, was like, "Oh, okay, this is just, like, physically impossible to implement," right? And, and, and as we step forward to, like, going back to the test time compute thing, right? A query for, you know, you ask ChatGPT a question, it costs cents, right? For their most capable model of chat, right? To get a query back. To solve an Arc AGI problem though cost five to 20 bucks, right? And this is, this is in a, you know-

Nathan Lambert 1:15:18 It's only going up from there.

Dylan Patel 1:15:19 This is a thousand, 10,000 X factor difference in cost to respond to a query versus cr- do a task. And the task of Arc AGI is not like it's like, it's- it's simple to some extent, um, you know, but it's also like what are the tasks that we want AG... Okay, AGI, quote unquote, "what we have today can do Arc AGI." Three years from now it can do much more complicated problems, but the cost is gonna be measured in thousands and thousands and hundreds of thousands of dollars of GPU time, and there just won't be enough power-

Nathan Lambert 1:15:47 Yeah.

Dylan Patel 1:15:47 ... to use infrastructure to operate this and therefore shift everything in the world on the snap of the finger. But at that moment who gets to constraint, control and point the AGI at a task? And so this was in Dario's post that he's like, "Hey, China can effectively and more quickly than us point their AGI at military tasks," right? And they have been in many ways faster adopting certain new technologies into, into their military, right? Especially with regards to drones, right? Uh, the US maybe has a longstanding, you know, large air sort of, you know, fighter jet type of thing, bombers, but when it comes to asymmetric arms such as drones they've, they've completely leapfrogged the US and the West. And the, the fear that Dario sort of, uh, pointing out there, I think, is that, yeah,