Skip to content

By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

Invest Like the BestInvest Like the Best

Ex-NVIDIA Engineer: Why AI Is About to Get 1000x Cheaper

Neil Movva, co-founder of Sail Research, joins Patrick to explain why the next era of AI may be defined not by faster chatbots, but by background agents that work autonomously for hours, days, or even weeks—and what it will take to make…

Neil MovvaguestPatrick O'Shaughnessyhost
Aug 25, 20261h 23mWatch on YouTube ↗

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

Transcript

  1. 0:00 – 0:38

    Intro

    1. NM

      My job is to make the tokens as cheap as humanly possible. I will achieve that, and I will do it through every layer in the stack available to me. I love the supply-side levers. I will use every chip, I will use every source of power, and I will use every piece of land in the United States that's, you know, suitable for this. We still treat the agent as a person that is expensive to consult, and you should ask them when you have a hard question. That's not the way to think about intelligence. It's incredible that the machine can think, and we should try to get that into as many hands of as many people as possible.

    2. PO

      [upbeat music]

  2. 0:38 – 4:21

    Building a “Token Factory”

    1. PO

      I think it's important early in these conversations to just say the thing, like literally what you're building and what it does today. So maybe just orient us there with a, with a brief description, like literally what the system is that you're building and why it should exist.

    2. NM

      Sail Research is a token factory. We have an API where anyone can send us requests where they can use large language models, open source large language models for any task they want. Um, we will serve those tokens to them at a price that is unbeatable in the market. We also support their ability to build agents on top of this. We host what we call Sailboxes, which are long-running agent virtual machines hosted in the cloud that are designed for agents that run for hours, days, or weeks.

    3. PO

      Mm-hmm. And so I should think about you as, uh, a peer company to others that serve different kinds of inference. You're serving one specific kind of inference-

    4. NM

      That's right

    5. PO

      ... and your goal is to be the absolute cheapest provider and enabler of a certain kind of use of intelligence.

    6. NM

      Exactly. The theme of our company is abundance. We want to deliver this new commodity of intelligence to as many people as possible at a, at a cost that is sustainable for almost every industry. We think that whenever you make something ten times cheaper, it's a new product category.

    7. PO

      Yeah.

    8. NM

      And, uh, we aspire to do that for tokens. We think it's so profound that the machine can think, and now our job is to make as many machines as possible in the world work towards thinking.

    9. PO

      So if you think about, uh, the theme of the day being token costs, is token cost the right way to think about this? Like, is there some other way you'd put it?

    10. NM

      To start with, absolutely token cost. Today, my North Star is I want to have the lowest cost per token in the industry-

    11. PO

      Mm-hmm

    12. NM

      ... and do that by a mile. I don't think tokens are the final unit of, uh, of work or intelligence, but they are what we use today, and so it's very straightforward. I think after tokens, you start to move more towards, um, more outcomes, which is like a vague direction. Uh, you can imagine, for example, today, when you consume tokens through an agent, you don't actually control how many tokens the agent reasons for. It can reason for a certain amount of time, or it could call a certain number of tools. And increasingly, I think we will have agents do some unit of work, take as many shots on goal as they can, and however many tokens they use to get there is going to be kind of a dependent variable depending on the task.

    13. PO

      Mm-hmm.

    14. NM

      So you think about like agents that self-administer a token budget as opposed to a company setting a budget for how many tokens engineers can spend per month.

    15. PO

      Why is there an opportunity that you can tackle? It, it seems like the entire world is oriented around more, better, faster, cheaper-

    16. NM

      Yeah

    17. PO

      ... tokens right now. It seems like the world is trying to solve this problem very aggressively.

    18. NM

      Yeah.

    19. PO

      What was the unique opening that, that you saw that's maybe the market's not being efficient in its attempt to tackle this?

    20. NM

      I think there's two things that are tailwinds for our company. One has got to be the rise of open source. I have to talk about that first. I, I think we are starting to see an increasing number of our customers and the broader market care about owning intelligence. They, they want to have control, sovereignty over the thing that they depend on. Uh, and so that created a much more robust market for customized models or even just like these vanilla open source models that no one can ever take away from you. You always have the weights, you always have the right to deploy them however you like. Uh, in that world, there's been a reasonably robust market for the past couple of years, uh, serving these models at large scale. The challenge is all those companies, you could take your pick, Base10, Fireworks, Together, they all focus on low latency inference, and they were pulled in that direction by one very important customer, uh, Cursor. And I think that that was the right choice about a year ago. And as of six months ago, it started to look like maybe low latency wasn't the only thing you wanted from an agent. You wanted more persistence, more long-horizon tasks. And now it's, to me, very obvious that the future of agentic inference is long-horizon tasks. You're gonna run the machine for hours or days at a time. It doesn't matter if it spits out tokens at 100 tokens per second. Maybe ten is just fine if that comes with corresponding advantages

  3. 4:21 – 13:09

    The Future of Background Agents

    1. NM

      in efficiency.

    2. PO

      Why are you so s- confident in that? It, it, like-

    3. NM

      Yeah

    4. PO

      ... to me, it seems like I want everything f- as fast as possible.

    5. NM

      When you're waiting on it, you absolutely deserve the fastest answer possible.

    6. PO

      Yeah.

    7. NM

      My trick is I don't want you to be waiting on it. I want it to be proactive. I want it to be in the background. One way to say it is like the best latency is no latency at all. When you wake up in the morning, the work's already been done overnight. You didn't even have to ask for it. Uh, that's the dream. We're not quite there yet. But more importantly, I think the more you're in the loop as you prompt agents and wait for a response, in fact, you're the bottleneck, uh, in helping the a- in having the agent do more or less work. What we'd like is the agent to operate on more human time scales. You don't manage your colleagues every five minutes. You ask them to do a high-level task, and you come back and check in maybe every day, but more likely once a week. Uh, and that to me is the future of human-agent collaboration, more like human time scales.

    8. PO

      Say more about the early indications that this is happening and therefore you should be building this company.

    9. NM

      Well, so the first and most important thing is the idea of test-time compute scaling, uh, the idea that you can give an agent more time, and it will give you a better answer. So that was theorized about two years ago now, and, uh, but it wasn't really something that we could actually bet on until, I would say, late last year with Opus 4-5. Opus 4-5 was the first agent that was at all suitable for longer horizon tasks. Uh, and you know, it was pretty mediocre and when it first came out, but you look at the more recent models and what we've done on open source as well, and you see that agents are capable of running for an hour at a time. I wouldn't say it's days, but definitely an hour is quite suitable today. And so just seeing that like average turn or task length get longer and longer, uh, it doesn't take many points to have you kind of draw out the exponential and see that agents are worth running for longer periods.

    10. PO

      What, what do you think will be the market share of long-running agents in three years or something like this?

    11. NM

      You know, I love this market because it's unbounded. There's no human in the loop, so you can consume as many tokens as you like in the background. Uh, versus human attention span, if you tell me to consume ten X as many tokens at Codex or at Claude Code, I'm actually not sure if I can anymore. I'm already in a loop and locked in coding for most of the, the day that I'm at the laptop. What is undoubted is how many tokens can be consumed in the background or proactively. So long term, I think, you know, we're gonna end this year at maybe fifty-fifty background and, uh, and real time workloads, but I see this going to ninety-ten in favor of background.

    12. PO

      What are the sorts of things-- Like, what are your favorite examples of something that gets accomplished much better as a background task than as a human-in-the-loop task?

    13. NM

      Most deep research. Most que-questions where you want to have a definitive answer over not a hundred sources, not a thousand sources, but ten thousand sources or more. If you want to build an authoritative index of information, like for example, one of our customers, Parallel Web Systems, seeks to do. They want to build an index over the whole internet, and they want to monitor the internet in real time for changes.

    14. PO

      Mm-hmm.

    15. NM

      That is the kind of crazy exabyte scale task that you need a very different kind of intelligence or scale of intelligence to achieve. Deep research is a top category for us.

    16. PO

      Mm-hmm.

    17. NM

      And then increasingly, we see cybersecurity following this direction. If you think about, yes, there's so much code you can generate, but there's, uh, exponentially more ways to break that same code than it is to generate that code. And there are some great customers out there who are working very hard to, uh, find agents that can break any piece of software and proactively patch them. So when Fable first came out, for example, or Mythos first came out, basically, there was this push in the cybersecurity community to run Fable against every line of code we've ever written and look for bugs in twenty different ways. Uh, meaning you're looking for both memory errors, you're looking for business logic errors, and looking for, like, network vulnerabilities, all these things. And these are all actually things that you would write specialized agents for. You wouldn't just have Fable look at the source code once. You'd have it actually set up environments where you can pen- pen test these applications. And at some point, people started to make this joke that security has become proof of work. When you want secure software, it's really a question of how many dollars did you spend on Anthropic's APIs trying to break into your software? Uh, that is the best indication for how secure it is because that's the best tool in the world. And increasingly, we found that the frontier of intelligence here is quite jagged. It's not the case that Fable finds a superset of all bugs in software. You would find some bugs with a very small model that you don't find with a large model. Uh, you'd find some bugs with Haiku that you wouldn't find with Fable, and vice versa. So it encouraged this very diverse approach to sampling and, and trying to build cybersecurity agents that break software autonomously such that you can patch them.

    18. PO

      If you were to get sort of, like, speculative and imaginative about the sorts of things that long-running, very cheap, very long-running agents can enable, we talked about some very practical examples, deep research, um, cybersecurity, et cetera. But if you, if you get a little bit dreamier about the use cases, new product category that this sort of inference will unlock.

    19. NM

      Yeah.

    20. PO

      And I guess the question is just like, so what? Like, what, what-- If you're maximally successful-

    21. NM

      Yeah

    22. PO

      ... dream a little bit about what that might enable.

    23. NM

      Yeah, absolutely. So I think for individual users, what I'm excited about most is this idea of proactive intelligent agents. Um, you can imagine a Siri that is running in the background all the time to understand what's all the emails you received in a day, all the text messages you receive in a day, and it has a much more encyclopedic view of your life and how to be helpful in that life. Right now, there's still point solutions, and so you have to-- you end up doing a lot of prompting. Siri is not very proactive.

    24. PO

      Mm-hmm.

    25. NM

      That's something that we can fix with abundant, abundant inference. If you trust, uh, the machine enough that it's reliable and also trustworthy, as in private, um, you might even imagine the machine can understand how you interact with it and proactively surface your next action. Whenever you open your phone, can we build a good model of what you're gonna do next?

    26. PO

      Mm.

    27. NM

      My estimation is yes, we totally can.

    28. PO

      And the key to that is incredibly cheap intelligence.

    29. NM

      You have to be willing to spend tokens without any promise of return. That's the unlock. The long lens view to take on this is that we have, we have a form of intelligence that can tackle any verifiable problem. Any verifiable problem means most software, it means a lot of formal, like, math proofs and similar, and it could also mean scientific discovery. These are all relatively verifiable problems. And all those things currently have a dollar cost attached to them, essentially, that's a hidden one. It's like how many tokens could you possibly harness to make this work? And we have actually started to bring it within view, a dollar cost for these long horizon tasks that is reasonable. It's not millions, it's thousands, and maybe it could be hundreds or even tens of dollars in the near future to have a definitive answer to any scientific question, to any research problem.

    30. PO

      [upbeat music] Ramp is the only platform built to make your finance team leaner, faster, and better, saving businesses five percent annually on average, so you can stay focused on growth. Ramp customers grow revenue three point two times faster than the average American business. Visa, Vercel, Cursor, Stripe, Notion, Eleven Lab, Shopify, and seventy thousand other businesses all now run on Ramp. Mine does too, and so should yours. Learn more at ramp.com/invest. OpenAI, Cursor, Anthropic, Perplexity, and Vercel all have something in common. They all use WorkOS. To achieve enterprise adoption at scale, you have to deliver on core capabilities like SSO, SCIM, RBAC, and audit logs. Instead of spending months building these mission-critical capabilities yourself, you can just use WorkOS APIs to gain all of them on day zero. That's why so many of the top AI teams you hear about already run on WorkOS. WorkOS is the fastest way to become enterprise-ready and stay focused on what matters most, your product. Visit workos.com to get started. Felix by Rogo is a personal finance agent that turns a single prompt into finished, client-ready work using your firm's own templates, context, and standards. Send Felix an email like, "Take these comments and turn them for me," or, "Update my tracker with the context of these emails," and Felix sends back finished PowerPoint decks, Excel models, and sourced research. Felix works the way your team already does, delivering work quickly and accurately around the clock. Learn more at rogo.ai/felix. And so if we dream about that future, we're-- we then become limited just by the questions that people can ask, basically?

  4. 13:09 – 23:27

    Nvidia and the GPU Stack

    1. PO

      about doing.

    2. NM

      You know, we always had to start with software. You know, where is the opportunity on today's chips with today's data centers to improve efficiency? And the first thing we did was we tried to build the entire LLM software stack around peak GPU efficiency, meaning we're using NVIDIA GPUs. We wanted to squeeze out more tokens from the same chip than anyone else in the world, and that starts with the lowest level of programming, kernels. It's actually my background. I spent my whole life actually, uh, my whole professional life working on GPUs and kernels. NVIDIA was my first job while I was in college, and, uh, I got to see how the tensor cores got to earn their right to be on the chip. Um, this is back in 2016 for-

    3. PO

      Just describe what that means for the-

    4. NM

      Yeah

    5. PO

      ... for the layperson.

    6. NM

      So, okay, the tensor core is a specialized unit on the GPU that accelerates matrix multiplication.

    7. PO

      Yeah.

    8. NM

      Simple as that. There's been a long history of how we evolved that tensor core over time that we'll get into.

    9. PO

      And why is matrix multiplication so important?

    10. NM

      That's a great question. I actually, I cannot say that there is a divine truth of the universe that explains why matrix multipliers seem to be the atomic unit of computation. But, uh, one way I've heard it described to me is, well, it's a really succinct way to mix two blocks of numbers together and have them interact in some interesting way. That's as much as I can say about it.

    11. PO

      Yeah.

    12. NM

      It is really convenient that linear algebra turns out to be a very compact representation of arbitrary relationships in data. So NVIDIA, great graphics company, obviously has, had market share dominance in GPUs and, and gaming graphics for quite some time. And then starting in like the mid twenty tens, they started to actually start these like skunkworks projects to make the graphics processor more suitable for machine learning tasks that they were tracking. Uh, I remember actually reading some of the, like, lab notebooks of some of my managers when I was at NVIDIA. They would visit these small ML conferences like ICML or NeurIPS at the time, and they would just take note of these papers like, "Oh, this deep learning thing seems to be catching on. And what's really interesting is that these grad students are using gaming NVIDIA GPUs in order to train their large models. We should double-click on this and figure out what's going on here." And by 2015, 2016, at least Jensen had the conviction to, to kinda double down on, hey, this usage of our models is only, or of our chips is only gonna grow. Let's start allocating more and more precious silicon die area to this capability that seems to be emerging. Let's put the first version of tensor cores on the chip. So we're talking about, you know, taking this gaming chip, which is designed for painting pixels on a screen, and, uh, adapting it to do matrix multiplies. And it was early, and you'd be competing against the graphics teams essentially. When you ask for more silicon area at any chip company, there's always competition for that. It is something that the, the, the designers guard so carefully.

    13. PO

      Mm.

    14. NM

      You don't ever want to invest in the wrong technology because that's opportunity cost that you could've allocated to some other functionality. Uh, and so we, we kind of like fought and tooth and nail and got just a tiny bit of die area, maybe like five, 10%, something like that, for the first generation of these chips, uh, to get some, some amount of acceleration for basic convolutions, which were the fundamental operation for computer vision models in the day. Uh, and then we had a software team that was trying to squeeze all the performance we could out of the chip. And I think on that software team, which is where I worked, that's what actually taught me the most about, um, just the ethos that NVIDIA has around, they have this term called speed of light. Uh, they always chase the speed of light for any piece of hardware that they make. It is so ingrained in every engineer's mind that if the machine can do it, we're going to push the machine to the frontier until it does what we think is-

    15. PO

      And the speed of light is the edge of what's possible.

    16. NM

      The speed of light is the edge of what's possible. Exactly. Uh, if we think the chip can run at this frequency and produce this many multiplies per cycle, we're gonna get there. We're gonna break every bottleneck and get to that peak level of performance. And so to this day, I tell all my engineers, like, "We're chasing 100% speed of light." I don't care about relative numbers versus the competition. I only care about absolute numbers. Uh, what are we able to do on the chip, and how do we achieve that?

    17. PO

      Before we leave that chapter of your time at NVIDIA, anything else beyond that cultural touch point that really, like, changed the way you think about things, or that stood out the most about how the business ran back then or its culture?

    18. NM

      I have a ton of stories about NVIDIA. We can, I could tell you [chuckles] a few of them. Um, one of my favorites is that on the tenure side, a lot of people I worked with in NVIDIA in 2015, 2016 are still there today.

    19. PO

      Mm.

    20. NM

      That company has incredible retention, and these are the best engineers, uh, frankly, uh, on silicon side at least, I've worked with in my whole career. They're extremely, extremely motivated and passionate. They believed in parallel computing as a concept through its various incarnations and have loved seeing the chip evolve. This is their life's work, and they're extremely, extremely competent in that direction.

    21. PO

      Mm-hmm.

    22. NM

      They're also a very frugal company. NVIDIA and all of, I guess, all the Silicon Valley companies after 2008, they had some cutbacks in like perks. So no free lunch, uh, for example. NVIDIA took it one step further. There was no free milk in the fridge. So if you wanted to drink coffee at NVIDIA and you wanted some milk, you'd actually have to chip in a dollar every month to the milk club, and the milk club would stock Costco milk in the fridge.

    23. PO

      Mm.

    24. NM

      And I remember that distinctly [chuckles] . We don't do that at Sail.

    25. PO

      Mm.

    26. NM

      But, uh-

    27. PO

      Mm

    28. NM

      ... it's a, it's a frugality that permeates the company.

    29. PO

      And, and so coming out of this time there, y- you get this experience of what it's like to develop more efficient usage of the underlying hardware through software.

    30. NM

      Yes.

  5. 23:27 – 36:14

    Chips, Memory, and Transformers

    1. PO

      So before we leave the latency part of the story-

    2. NM

      Yeah

    3. PO

      ... can you comment on companies like Cerebras or others that can perform incredibly fast-

    4. NM

      Yes

    5. PO

      ... operations? I'm curious like what you think about those approach of those companies, what might happen in the future. What is your prediction for the future of very low latency focused hardware?

    6. NM

      Cerebras, Groq, uh, and a couple others that are coming out of stealth now, uh, I think have made a very interesting bet on not just building another GPU, but actually building a different kind of accelerator that focuses on a different memory hierarchy. Uh, they want to maximize the amount of SRAM on the chip and use that as very, very fast memory for, for weights and KB cache. So SRAM versus DRAM, there's two ways to make memory for a chip. One is to integrate the memory on the logic die itself, like meaning you tell TSMC, "I want this many megabytes of, uh, storage on my chip," uh, and there's a way to build that. TSMC has a standard cell library you can use, and you can just print out Uh, a bunch of cells of SRAM. The problem with SRAM is it takes a lot of area on the silicon die. Um, so if you wanna build a large die, like let's say the Nvidia Blackwell at eight hundred millimeter square, if you made that whole die SRAM, it would be in the maybe like single-digit gigabytes. It's not a crazy amount of, of data storage. Compare that to if you're willing to take a different process technology entirely, so not TSMC anymore, but now Micron, SK Hynix, Samsung. They build, uh, DRAM, which is a whole different way to build memory that's more focused on capacitors than, uh, transistor cells. So SRAM, the standard way to build SRAM is what's called the 6T transistor cell. It's a stable transistor arrangement that allows you to write a bit to it, and then it holds that state in that bit, regardless of whether you keep applying... Well, you had to apply some power, but, uh, it, it's holding that bit without any sort of like active management. It's static. Now, dynamic RAM, DRAM, it's dynamic because what you do to write some data is you write a charge onto a capacitor. And as soon as you write that charge into that capacitor, the charge is dissipating. It's leaking. And so the dynamic part of DRAM is that you must every fifty milliseconds or so refresh every bit you've written. So you're constantly juggling billions of balls in the air, essentially. Billions of bits have to be managed by a memory controller, which is reading and refreshing every bit on the DRAM. Now, the benefit of that is you can get much, much higher density, and it's a whole different process technology. There's a ton of different trade-offs, hence why we split the DRAM manufacturing into an entirely different company like Micron, SK Hynix and Samsung. These are the best companies in the world to do this. They build DRAM, and if you take DRAM from those companies and you stack it, uh, into many layers, and you kind of print them or, or, uh, solder them around the main logic die that you get from Nvidia, you can now get hundreds of gigabytes. Uh, like Blackwell has two hundred and eighty-eight gigabytes of HBM capacity around the logic die, and the logic die itself maybe only has like five hundred megabytes of, of SRAM. So it's possibly multiple orders of magnitude, three orders of magnitude difference in density for DRAM versus SRAM. Okay, so let's go back to Cerebras. What are they doing? Well, they see this problem. There's not really an obvious way to increase SRAM density on the chip, but the thing with SRAM is because it's so physically close to the logic gates that actually do the computation, the arithmetic logic units are right next to the SRAM that they're gonna pull from. The compute units that are doing the matrix multiplies can pull data from SRAM at just mind-boggling speeds. You know, Cerebras quotes petabytes per second, twenty-one petabytes per second for their Wafer Scale Engine 3. And so, uh, compare that to HBM on an Nvidia Blackwell is, uh, you know, ten terabytes per second or so, in that range. So once again, many orders of magnitude difference. More capacity, but proportionally less bandwidth, essentially.

    7. PO

      Yeah.

    8. NM

      And so what Cerebras does is they say that we're gonna take as many of these dies as we can. We're not gonna limit ourselves to the eight hundred millimeter, uh, radical limit that TSMC, eight hundred square millimeter limit that TSMC imposes on us. We're gonna take the entire wafer and have actually every die connect to every other die over scribe lines, and we're just going to try to get as much SRAM as we can on the whole wafer. And we can get to like, let's say, fifty gigabytes of SRAM per wafer. And then we're gonna stack many wafers together in a pipeline or similar, and now we can have, you know, up to a terabyte of memory. Very, very fast memory. And you do all that work just to get to the ability to read data from SRAM at, uh, yeah, twenty-one petabytes per second per wafer. Therefore, you can now serve these language models at extremely high tokens per second because you can move the entire parameter count of a large model like Kimi. Uh, you can move all that data in and off the chip, or sorry, in and off the logic cores in about a millisecond or something like that.

    9. PO

      Mm.

    10. NM

      So there you go. You have a path to a thousand tokens per second.

    11. PO

      And so what is your prediction for, like, that segment of the market?

    12. NM

      Okay, so I think what happens to them is some hybrid sort of ou- outcome. Like we, we had to pair the Cerebras chip, where it's very strong, it's very, very good at fast access to memory, with something that has more capacity for memory. Because it's true that you can take a one trillion parameter model like Kimi and fit it on a large number of Cerebras die, uh, wafers, but you can't do something about the KB cache very easily. The KB cache is something that, uh, grows as people use the model more, and that is always dynamic. You don't even know how much KB cache you're gonna need. It depends on what your users, how many users you have and how many users you want to serve.

    13. PO

      Can you explain KB cache just like in basic terms?

    14. NM

      Uh, yeah, yeah. So KB cache, when you ever use a language model, every token you send through the language model actually, uh, stays in the context window of the language model for as long as you're having a conversation. So if I-- we talk for a hundred thousand tokens, the hundred and thousand and oneth token is still in the conversation, uh, behind us, and the model is referencing all the past conversation history in order to-

    15. PO

      Okay

    16. NM

      ... uh, make better predictions about what the next thing we're gonna say is. A- and so that KB cache is a bunch of memory. Um, you have to store a representation for every token that we send through the language model, and it frequently gets to be larger than the weights of the model themselves. You have this, like, crystallized knowledge in the model weights, and you have the dynamic knowledge of the exact conversation we're having in the KB cache, is the way I like to think about that.

    17. PO

      Yep. And, and this is why sometimes people would observe, like deep in a conversation, things start to degrade because there's some sort of like technical problem?

    18. NM

      Yeah. So the KB cache is quite interesting in that regard. The KB cache is an exact representation of everything that came before. We, we store all the information that we've seen in the conversation. However, during training, the model did not get trained primarily on very long context conversations. It got trained primarily on, let's say, eight thousand token conversations or sixteen thousand token conversations. So if you take the model to two hundred thousand tokens, there was some training that happened at that context length, but it's not the model's like core strength. And so there's, there's always been a challenge for the frontier labs to figure out, how do we make the model exactly as intelligent at ten thousand tokens as we expect them to be at two hundred thousand tokens?

    19. PO

      Mm.

    20. NM

      And it's gonna be a perennial battle for us. We've had one million context windows as a concept for, for years now. Anthropic was, I think, the first to hit that one million context window li- length. I still, you know, use slash compact in my cloud code, uh- Well before a hund- one million context length. I don't think it's actually great to hit the full length.

    21. PO

      And so these extremely fast, extremely low latency-

    22. NM

      Yes

    23. PO

      ... approaches ultimately are limited by, by this factor.

    24. NM

      Yes. Y- you can do whatever you want for the weights. It's very possible to have just unbeatable performance on weight storage. However, the KB cache is gonna be a big thorn in your side.

    25. PO

      And so three years from now, five years from now, what role do you think these kinds of chips play? Like, what sort of market share do they have in the heterogeneous chip market?

    26. NM

      Cerebras and Groq and maybe a couple others, you should think of them as accelerators. What they are really good at is being used in conjunction with an more traditional GPU-like device that critically has this off-chip memory built in. You want off-chip memory for capacity and on-chip memory for speed. We want to hybridize these two things. So if you take, uh, transformers in the limit, you take a transformer to a million context length, what ends up happening is you have this, you know, compute bound stage, which is the actual matrix multiplies for the, uh, what we call the MLP, which is where most of the model's knowledge, world knowledge, is encoded. And then you have the attention layer, which is where we're kind of dynamically adapting to the current conversation. Attention in the limit is usually memory bound, and the MLP in the limit is compute bound at large and at batch size. And I would say the original sin of transformers is that you've taken this extremely fundamentally memory bound layer and juxtaposed it right next to a compute bound layer. It is very difficult to have a single chip that is good at both compute operations and memory operations. The GPU is quite balanced in this regard, but you had to choose one or the other. Cerebras has a very fast memory access for something like a matrix multiply, and it's really good to host the, the MLP, the, the weights essentially, on the Cerebras chip. But the GPU has the capacity to scale to really long context lengths, and so you would like to put the, uh, attention possibly on the GPU and the MLP on the Cerebras chip.

    27. PO

      Mm.

    28. NM

      And I believe this is what's happening with Nvidia and Groq.

    29. PO

      The-- Can you riff for a minute just on transformers and, uh... Yeah, you've been so good at explaining some of the core concepts. Just for people that, again, aren't, aren't deeply familiar with what this innovation was in-

    30. NM

      Yes

  6. 36:14 – 44:32

    The Future of AI Training Data

    1. PO

      What's your prediction for the future of data and riff on the importance of data in this whole story?

    2. NM

      I like the phrase that internet was a one-time subsidy on data. We got it for free. Uh, it's extremely high quality. About thirty trillion tokens of high quality text. Uh, three hundred trillion tokens if you take a wider view on what qualifies as good text. And we've basically looked at it all already. Models have seen the entire internet many times over at this point, and there is not a whole lot more to be done on human data from the internet The next phase of data, in my mind, is model self-improvement through RL environment gyms, basically. In fact, w-we don't even benefit from getting more, like, random user interactions with AI. It used to be that y- you know, the, the new type of data that we cared about a lot was the interaction data from people using ChatGPT and giving ChatGPT signals on what they liked and didn't like. I like the argument now that the median model that we serve is so much more advanced than the, kind of un- uh, than, than like a random human, uh, giving feedback, that the signal you get from random human preference or, I guess, unconditioned human preference is not actually worth anything anymore. You want expert human preference at this point. The model has outgrown-

    3. PO

      Everyday Joe

    4. NM

      ... generic- Yeah, everyday Joe. [laughs] Exactly. So the future of data to me is giving the model a hard verifiable task and letting it run in this gym where it's kind of isolated and it just has a, a problem that it can make progress on and get measurement of whether it made progress on that problem or not. You can imagine coding problems are in this category. Math problems are also in this category. And, um, increasingly, more and more we have-- we can just give the agent a computer essentially and have it act like it's a human worker and just give it feedback on whether it's making progress towards the target outcome. That environment becomes the data.

    5. PO

      Mm.

    6. NM

      I think this is not a s- super differentiated take, but, uh, it, it's been really, really productive from what I've seen so far.

    7. PO

      And you think that just goes on for a really long period of time, or is that another like, if I think about the internet as this one big block, like this is another big block that will have its, you know, day in the sun and will kinda get it all and-

    8. NM

      Yeah

    9. PO

      ... and then we'll have to move on to something else?

    10. NM

      I think it's actually more profound than that. Basically, the idea is that if you want artificial general intelligence, the best way to get there is to just keep stacking specialized intelligences until you have no more gaps to fill.

    11. PO

      Mm.

    12. NM

      And the test here, the only thing you need to make sure you do to make this work is you must make sure that your task is verifiable. You need to give the model a self-grading system. If you have that, you have the recipe for self-improvement on any task you like.

    13. PO

      Mm.

    14. NM

      And I think you've seen this held up by the way frontier labs spend. They used to spend much, that much on data, now they spend a lot more on RL environments. And, uh, these environments absolutely capture that relationship of recursive self-improvement on a verifiable task.

    15. PO

      Okay. Okay, now, so [laughs] I like that we've veered off in different little sidecars here, but coming back to your initial task-

    16. NM

      Yeah

    17. PO

      ... of making existing hardware more efficient-

    18. NM

      Yes

    19. PO

      ... by being more in control of what's going on at the hardware level through software.

    20. NM

      Yeah.

    21. PO

      Um, so, so yeah, just keep going on what you've done so far and what you wanna do, and then we're gonna jump to hardware and then jump to energy finally.

    22. NM

      Sounds good. So yeah, I mentioned kernels. It's surprising, people think kernels are done. There are great people like Tridao who write excellent kernels, and they're, they form the bedrock of all of our, um, modern deep learning is built on flash attention. Modern transformers are built on flash attention. But if you deviate from the happy path at all, if there's a new model that comes out that has a slightly different way to embed positional information, like the change of the rope system, suddenly the kernel that we had is not suitable for this new model and we may have to make a ch- a patch to this kernel. I wouldn't say we're in the phase where we have to invent new kernels from scratch, but having the ability to quickly modify existing GPU kernel-- Sorry, a kernel, by the way, is a, it's a general term for any program you run on the GPU. And so historically, kernels tend to be put into a library where every kernel has a very, very scoped purpose. You specifically you have a kernel for a matrix multiply. You have another kernel for even something as simple as addition. You wanna add two tensors together, that's another kernel. [clears throat] And then increasingly, we've started to fuse those kernels together. So if I do a matrix multiply and then I wanna add it to another matrix that I've also multiplied, maybe those two become one kernel and I just fuse the operations where instead of writing the data out to DRAM and then reading it back in just to do the addition, maybe I can just do this, uh, easily.

    23. PO

      Why are humans still doing this? Like, it seems like the sort of thing that AIs would be exceptionally good at engineering-

    24. NM

      Yeah

    25. PO

      ... more efficient kernels. Maybe that's where we're going, and we're just not quite there yet. But if, if we aren't there yet, is that where we're going? If we're not there yet, why, why humans still doing this? Why, why is Tridao so well known? You know, it's a name I know.

    26. NM

      I don't wanna speak for Tri, but what he taught me was, uh, you shouldn't write kernels by hand anymore necessarily.

    27. PO

      Mm-hmm.

    28. NM

      I like to say we write kernels on the whiteboard. We go to the whiteboard, we describe what we think the machine should be doing.

    29. PO

      Mm-hmm.

    30. NM

      Then we succinctly describe that in, in natural language to the, a model, and then the model is able to do the execution of, okay, here is my input and output. Here is the strategy of how we want to dispatch this work onto the GPU. I'm gonna go implement this work.

  7. 44:32 – 52:44

    Chip Scarcity and Compute Arbitrage

    1. PO

      One of the things you hear is that the market for the best chips, Blackwells let's say-

    2. NM

      Yes

    3. PO

      ... is like a drug market or something right now.

    4. NM

      Yeah. Oh, yeah.

    5. PO

      Like, there's all sorts of fascinating things happening to get as many of them as possible 'cause everyone's so short.

    6. NM

      Yeah.

    7. PO

      I'd love you to react to that analogy. Like, is that what it feels like?

    8. NM

      Hundred percent.

    9. PO

      But then also to talk about what the market is like for, like, not the bleeding edge chips.

    10. NM

      Yeah.

    11. PO

      Like, if I, if I am willing to accept a slightly or, or moderately inferior chip-

    12. NM

      Yeah

    13. PO

      ... what's that market like? Like-

    14. NM

      Yeah

    15. PO

      ... g- g- let us into that world.

    16. NM

      Yeah. Okay, a couple things. Number one, uh, yes. Basically, NVIDIA has a long-term view on, uh, on all their chips. They, they see this immense demand for the Blackwell chips, and they, they can do what other suppliers have done in the past, which is, like, just crank prices and m- make the market, you know, supply and demand curves will correct. They'll intersect at some point, and everyone will be technically happier. But NVIDIA sees the... If they just let the most deep pockets buy all the chips, that maybe hurts them in the long term if the customer ends up accruing a lot of more power. They understand the compute is power today, and so, uh, they're quite strategic about how they allocate compute. That's the first thought. The second thought is that relationships matter a lot. Nobody wants to have a huge order of, of a chip rental come in from this new startup that says, "Oh, yeah, I'm gonna rent ten thousand Blackwells for, for three years or five years." The startup has only been operating for months typically. Wh- who knows that they're good for the money? Uh, the, the way you convince someone to give you access to compute is, is quite challenging, uh, these days, and requires some pretty either great relationships or, uh, just incredible financial backing to make this happen on the NVIDIA side, and it's all because the scarcity, uh, is so high, and demand is just off the charts. Now, for other chips, I wouldn't even call them inferior. I, I like to say there's no bad chips. There's only bad pricing, and, uh, I will make any chip work at the right price. That's, like, kind of one of the ethoses of the company. Uh, let's talk about AMD.

    17. PO

      Yeah.

    18. NM

      AMD, I think great chips overall. The challenge is that people don't, um, understand how to program them very well. So, you know, I've been talking to you about how we have such a great kernel team. We're so serious about squeezing the performance out of the hardware. NVIDIA is pretty good at doing that for their own chips, frankly. There's some alpha that we can squeeze out, but actually there's a lot more to be done on other chips because the vendor does a little bit less work than NVIDIA does to make the best kernels out of the box. Or, or even better for me, there is alpha, and just like other people have this perception that AMD is not as, as good as NVIDIA. That's music to my ears. I'm very happy for them to sleep on this chip and for me to buy as much as I can. Now, I think that that's not actually super true anymore. I think AMD is actually, uh, somewhat popular amongst the, some large buyers. Um, you know, I think publicly Meta and OpenAI have bought a ton of AMD chips. And so we're increasingly seeing that, uh, all the AMD supply is also being allocated. But there's a long tail of other companies that are popping up. Yet net new companies are great, uh, such as Hatched or S- uh, or SambaNova or, uh, DMatrix. All these companies are popping up, and I think the main challenge for them is scale. Can they actually get enough wafer allocation from TSMC to pump out chips to make an it to the market? But certainly, if there's a new chip on the market, I'd like to know about it as quickly as possible and evaluate whether we can buy a good fraction of that supply.

    19. PO

      And so it's fundamentally an arbitrage for you. Like, if you can be much better at eking out performance from chips that have received less attention-

    20. NM

      Right

    21. PO

      ... you can then resell that at a margin, and it can be a great business.

    22. NM

      Exactly. Exactly. And I think that it's not the case that everyone else is just, you know, has a skill issue that they can't, uh, you know, make these chips work as well. I think we're quite competent at this. I think we're probably one of the best teams in the world to use multiple silicon architectures and, and be quite aggressive in chasing down performance in unlikely places. But, um, yeah, I think it's the speed at which we- we're willing to kind of build our stack around a new chip. We don't have a huge amount of incumbency around, well, our data center providers are only stuck with, with this class of chip, and it's gonna be a huge pain for us to, to deploy, uh, these, uh, new chips.

    23. PO

      Yeah.

    24. NM

      We have some very creative data center partners who are willing to move very quickly, and there's a new class of those that we can talk about. And most importantly, we don't shy away from the challenge.

    25. PO

      Mm.

    26. NM

      Uh, that's frankly a big part of this, is just saying, "Yes, we love TPUs. We're gonna make TPUs work. Yes, we love Trainium. We're gonna make Trainium work." And if it doesn't work, um, easily, we're gonna find a way to fit it in with a heterogeneous serving system. It will have a place. Every chip has a comparative advantage. We had to find that advantage and then squeeze it-

    27. PO

      Yeah

    28. NM

      ... in that direction.

    29. PO

      Ju- just as an interlude, before we get to hardware data centers, energy, et cetera, which will be a really fun part of the conversation, I'd love you to talk about your perception of the investor class's worry.

    30. NM

      Yeah.

  8. 52:44 – 59:01

    Reinventing the AI Data Center

    1. PO

      Coming back now to your take on hardware.

    2. NM

      Yeah.

    3. PO

      And so the unit level is interesting to me, like talk about chips, talk about racks, talk about, you know, clusters. I'd love to talk about data centers.

    4. NM

      Yeah.

    5. PO

      And you said you've had some interesting partners doing some cool things.

    6. NM

      Yeah.

    7. PO

      Talk us about the, the present and future of data centers as you see it.

    8. NM

      Yep.

    9. PO

      Because this seems like, you know, obviously a critical thing for being able to serve all this inference. There's, like, lots of innovation in, in this part of the world-

    10. NM

      Yeah

    11. PO

      ... and obviously you're focused on it.

    12. NM

      I think one of the themes in our conversation has come back to what is training versus inference, and what is the difference between these two categories, and, you know, what was different about two years ago being training-oriented and today being inference-oriented. And I think the most conservative players in the entire AS stack are, have got to be the infra players, whether that's data centers or even more conservative is TSMC, the chip infra people. Uh, and so data centers historically were built, like AI data centers, they were built for training. Uh, and training is a superset workload over inference. You can make any training cluster work for inference, but maybe not vice versa. And what the difference there is networking. Uh, how much do you invest in bandwidth between chips, uh, and how large of a cluster do you need? There's actually a diseconomy of scale to, to data centers in some way. Like, it's way more expensive and difficult to build a, um, you know, 100,000 GPUs in one data center than it is to build 10,000 than it is to build 1,000. And, and we now we just talk about, you know, how many megawatts or gigawatts do you have? And basically, there's no way to build a gigawatt data center in the United States easily, uh, anymore. Even 100 megawatts is, is increasingly hard. Uh, it's basically impossible unless you're a very special set of customers. Uh, 10 megawatts is probably on the edge of what's possible today, and one megawatt I would argue is plentiful. So there's this in- incredible lore on the market where you can find lots of aggregate power, but it will not be concentrated, and that was not interesting to anyone who's building training, uh, data centers because you just assume-

    13. PO

      It needs to all be in one spot

    14. NM

      ... it needs to all be in one spot for, uh, no one wants to deal with cross data center training. So the market has some lag in it. I think that the market still assumes that we had to go shake down those 100 megawatt and 10 megawatt data centers wherever we can find them. It's still the attitude I hear from a lot of data center developers.

    15. PO

      Mm-hmm.

    16. NM

      But increasingly, we're seeing a few new thinkers realize that inference is gonna be suitable for these distributed one megawatt data centers. And, uh, we're, we're quite in agreement with that, and we are very happy to buy small pools of compute across the United States and use that as our inference fleet.

    17. PO

      Give us a sense of literal physical size of-

    18. NM

      Yeah

    19. PO

      ... uh, one megawatt versus 10.

    20. NM

      Yeah. Well, so this got really wonky with the advent of liquid cooling. Now you can pack insane levels of power density into a single physical rack. Like, a megawatt of compute, you know, you'd imagine this, like, massive data hall, like a huge warehouse basically, and now you can actually pack that into, yeah, around, like, around, like, eight racks worth of compute. Each rack is about the size of a refrigerator. You can just imagine eight of them lined up. Um, yeah.

    21. PO

      That's a megawatt. And so your view would be that the future that you wanna help build-

    22. NM

      Yep

    23. PO

      ... is a whole bunch of different chips that can be used together-

    24. NM

      Yes

    25. PO

      ... that you can buy, you know, y- y- you're a buyer to eke out the most per chip.

    26. NM

      Yep.

    27. PO

      And that those chips can then be coupled in very small-

    28. NM

      Yes

    29. PO

      ... data centers to just do inference.

    30. NM

      Yep.

  9. 59:01 – 1:10:10

    Power and the “Scavenger Strategy”

    1. NM

      Could I even take 80% uptime at the right price? Probably.

    2. PO

      Mm-hmm.

    3. NM

      Um, and what does that mean? Well, I'm a son of California. I love solar and wind. I think solar and wind power is way under-tapped in the United States, and the challenge has always been this intermittency. You would even consider solar and wind unsuitable for data centers because you have a persistent base load and an intermittent power source. What are you gonna do? Well, I think we're actually not that far from solving that problem. I am totally capable of tolerating a outage from a data center that's measured in, in even days or weeks, which is, like, the worst-case nightmare scenario for a data center, is that we're gonna have a long-term outage because the wind doesn't blow and the clouds are in the sky. Fog is hanging over the valley for some time. That's the worst-case scenario. It's, in fact, highly predictable, and I can just call in capacity in some other place of the world whenever that happens. Uh, I'll just model the weather and figure out when my data center's gonna be offline, move my data s- my workload somewhere else, and it's fine. The trick is that it's gonna give me better access to power that no one else is gonna touch because it is so annoying to deal with that kind of outage.

    4. PO

      Hmm.

    5. NM

      And if my chips are cheap enough, they're probably not gonna be in video racks. And if my chips are cheap enough, I don't mind the capital cost of having idle chips.

    6. PO

      Yeah. And so I've heard you describe this entire system as, like, a scavenger strategy.

    7. NM

      Yeah. Yeah, that's right.

    8. PO

      Is, is that... Yeah, unpack that analogy a little bit.

    9. NM

      Well, first we scavenge chips, and then we scavenge power for those chips. The idea is, in both cases, I do not wanna be in, bidding against Anthropic or OpenAI for compute capacity. I'm not gonna win against them, and I don't want to. I want to be more creative and use the supply that they don't find legible today. And over time, I amass enough aggregate supply. I'm never gonna get concentrated supply. I will only get aggre- aggregate supply. And over time, I build my aggregate factory that is unbeatable at economics.

    10. PO

      Hmm.

    11. NM

      We are building a factory. We're trying to build the best steel factory in the world, uh, but it will come through mini mills-

    12. PO

      Hmm

    13. NM

      ... not through large-

    14. PO

      Hmm

    15. NM

      ... monolithic steel plants.

    16. PO

      And, and if I imagine the different versions of this, like, how vertically integrated you can be-

    17. NM

      Yeah

    18. PO

      One version would be, the extreme would be you own everything, so that it's a very capital intensive business. You own the power, you know, source, you build the data centers, you design your own chips, you [laughs] control the software that ekes the most out of those chips, and you sell the end finished token-

    19. NM

      Yeah

    20. PO

      ... to your user. Like, your user is me.

    21. NM

      Mm-hmm.

    22. PO

      And you just own the whole stack.

    23. NM

      Yeah.

    24. PO

      But you can imagine many other permutations of the business-

    25. NM

      Mm-hmm

    26. PO

      ... where, you know, you could, whatever, you could draw the line anywhere. You could be incredibly capital light, own nothing, and just be, like, the coordination plane-

    27. NM

      Mm-hmm

    28. PO

      ... across all this stuff, the virtual scavenger.

    29. NM

      Right.

    30. PO

      How do you think about that question of, like, which ty- which type of these businesses to be?

  10. 1:10:10 – 1:23:34

    Open vs. Closed AI

    1. PO

      Can you give your assessment of the major labs-

    2. NM

      Mm

    3. PO

      ... one by one, but also then the relationship of, like, closed source as a category to open source-

    4. NM

      Mm-hmm

    5. PO

      ... and, like, what you think is happening and will happen?

    6. NM

      In a line, I would say the labs pay an immense premium to be three to six months ahead, um, of everything else. Uh, and I think that's probably still worth it. I think it makes perfect sense for OpenAI and, and Anthropic to do what they do. You know, there's a sensitive topic around distillation, which I think is part, a very core piece of the relationship between closed and open frontier. And, you know, I'd like to offer an alternative view on that, which is there is the sense that distillation is theft, that you are taking something from the frontier models when you distill on their outputs. And in fact, even if that's not your intent, even if you don't ever try to, you know, scrape data from Anthropic, one thing I'll offer is that an increasingly large percentage of the artifacts we put out on the internet are AI-generated. Even if you just look at GitHub alone, you know, what percentage of repos created in the last year do we think were created by Claude Code? Um, do we consider that to be distillation? Because that's probably all we need. I would not be surprised if you could train a Fable-class model only [chuckles] on the outputs of code you consider good on GitHub that's open source. And certainly if we take the position that users own the outputs of their interaction with AI, and they choose to put that up on GitHub, which a lot of them do, we're gonna have latent distillation for a long time. It seems fundamentally impossible for me. Uh, like I, I don't think f- it's fundamentally possible to prevent the diffusion of, of information or model capabilities. It will happen. The question is just how fast.

    7. PO

      And so then the question becomes do scaling and improvement laws hold forever or for a really long period of time? And if they do, then there's value to being three and six months ahead, and that will just last as long as it lasts, and they can charge a huge premium for those tokens relative to a very cheap open source token. Is that the right way to think about it?

    8. NM

      I think it's possible. I don't know that the premium for being three to six months ahead is gonna last that long. I mean, if you look at, like, enterprise deployments, uh, they don't move at three to six-month speed. A lot of enterprises are probably still on like 4.6, Opus 4.6 or Opus 4.7. They don't, they don't adopt the bleeding edge rapidly. There's a lot of questions that people have around rolling out any change at all. And I think we're just so early in scratching the surface that, um, I don't think there's any way to call a winner in this race, and certainly I don't even think this is a race that can be decided ever. There's always-- It's a continual process. And fundamentally, I don't think open source ever goes away. If there's a vacuum because one leader steps out, a new leader will step in. There's too much incentive and too much, th-there's a lot of tailwinds too. It's just it gets easier every day to tre- to train a frontier class model.

    9. PO

      And so your hope of what the future looks like is what? Like, what balance between closed and open? You know, what balance between model companies doing everything 'cause they have the advantage of owning the stack or whatever, you know, Anthropic can do that, you know, as like the new Google could, Google just do that or something. What do you hope the future looks like?

    10. NM

      I want abundant tokens and diverse harnesses. I want everyone to build their own harness and, and-

    11. PO

      Every company

    12. NM

      Every company, every user even. Make the agent yours- your own. Uh, I think we're, we're very-- not that far away from that level of customization and capability. I want people to own their intelligence, and I want that intelligence to be customized, probably not through weight fine-tuning, but probably through more in-context learning. Uh, that's a more technical detail, but the underlying input to this abundance future is about-- is basically cheap tokens. My job is to make the tokens as cheap as humanly possible. I will achieve that, and I will do it through every layer in the stack available to me. I love the supply-side levers. I will use every chip, I'll use every source of power, and I will use every piece of land in the United States that's, you know, suitable for this. And in return, people will have the incentive to explore what it's like to have abundant intelligence. We still treat the agent as a person that is expensive to consult, and you should ask them when you have a hard question. That's not the way to think about intelligence. It's incredible that the machine can think, and we should try to get that into as many hands, as as many people as possible.

    13. PO

      You sit in such a unique seat, and you have such a unique perspective on, like, what you're trying to do, uh, to make this future a reality. What do you think are your most, like, divergent views of the world versus your friends who are really well-informed and interested in this stuff? Like, what, what make your-- what ideas of yours make your friends look at you like you have three heads?

    14. NM

      Most of the ideas on chips, I would say. You know, when I talk about building custom chips, and they ask me, "Oh, so what's different?" Basically, it's, it's about sidestepping the HBM shortage and focusing on more extreme offload to other forms of memory, such as flash. Um, I'm quite passionate about that idea. Everyone on my team knows that I keep banging the drum around, like, what would we have to change about the model architecture to make offloading KB cache to flash work at a much greater level? And, um, you know, I'm my part in that all the time. Uh, that's, like, in the community of, like, inference people. You know, we have some divergent views on what you can do if you design a system around serving at, you know, one to ten tokens per second, which is our whole North Star. More broadly, I think there is this, like, larger sense around, you know, what do you do? How do people consume a trillion tokens per day? Like, that's the world we want to create, the capability for them to do that.

    15. PO

      What's a trillion tokens? Like, ground us in how much that is.

    16. NM

      A trillion tokens. Well, okay, at OpenAI pricing, that's at least five million dollars at the very least.

    17. PO

      Wow.

    18. NM

      For five point five or five point six. Yeah, I think the dollars is probably the most real way to look at it.

    19. PO

      Yeah, that's a good metric, yeah.

    20. NM

      Yeah. It's millions of dollars.

    21. PO

      Yeah, so what's the world in which we consume what currently costs five million dollars per person per day?

    22. NM

      Yeah. I mean, we were asking for at least, at least, um, you know, three to six orders of magnitude improvement in cost per token. Uh, get that into five thousand, you probably have some customers. And in fact, I would argue that we're, for some size of model, we are approaching a trillion tokens being, uh, measured in, you know, tens of thousands of dollars.

    23. PO

      Mm.

    24. NM

      And that's something that you could imagine running for a single job.

    25. PO

      Are you at all worried that just, like, the average person just can't and won't do that? Like, doesn't do that now with their own brain. Like, there actually isn't that much demand for intelligence in the world.

    26. NM

      I never will believe in that. There is always demand for intelligence in the world. I think that the way and the on-ramps to that intelligence are our challenge as a product, um, you know, community. Uh, I'm not a product person, so I cannot say I have the best vision on this.

    27. PO

      Yeah, you wanna enable those people, yeah.

    28. NM

      I wanna enable those people. I want them to never be held back by the sense that, oh, I-- my free tier users cannot use, or I can't afford to give them this many tokens. And I hear that from my customers all the time. Um, we wanna fix that.

    29. PO

      What about the inverse question? Not what you think is craziest, but, like, what consensus thing you think is wrong?

    30. NM

      One of the things I keep coming back to is in this question of Nvidia. I am bullish on Nvidia in the short term, and, you know, Nvidia, you should never bet against them. They're always gonna reinvent themselves. But, like, fundamentally, I think one thing that surprises people is when I tell them that, "Hey, if you look at, you know, Hopper to Blackwell to Rubin, and you compare like for like, like, what is the performance per watt of, of bfloat16 multiply, it hasn't improved all that much." Or, or even you take that one step further and go to TSMC. If you look at TSMC five nanometer versus four versus three versus two, the performance per watt on these chips doesn't change, like, a dramatic amount. So the consequence of this is people lose their minds over geopolitics, like what, what would happen if we lost access to TSMC for any reason? And, um, my contrarian take is that it, it wouldn't be that bad. Supply would take a shock for sure, but the best processes that we have in the West, uh, like Intel, not that far behind, at worst, like maybe two X, uh, worse performance per watt. Uh, and the gap is just far smaller than, than you would make it out to be if you talk-- if you follow, like, the chip war dialogue.

Episode duration: 1:23:36

In the uListen app, the transcript lights up word by word as you listen. Get uListen

Transcript of episode uyzqxIoiobU

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.

iOSAndroidClaudeChatGPTChrome