Dylan Patel on the AI Chip Race - NVIDIA, Intel & the US Government vs. China

1. 0:00 – 0:29

   Introduction

   1. DPDylan Patel0:00

      How you buy GPUs is like buying cocaine. You call up a couple people, you text a couple people, you ask, "Yo, how much you got? What's the price?"

   2. GAGuido Appenzeller0:07

      If your two arch nemesis suddenly team up, right? [laughs] It's the worst possible news you can have. I did not see this coming. I think it's, it's amazing development.

   3. SWSarah Wang0:14

      Like a Warren Buffett coming into a, a stock. Jensen is like the Buffett effect-

   4. GAGuido Appenzeller0:19

      Yeah

   5. SWSarah Wang0:19

      ... to the semiconductor world.

   6. DPDylan Patel0:20

      It's kind of poetic that everything's gone full circle and Intel's sort of crawling to Nvidia.

2. 0:29 – 2:11

   Nvidia and Intel: Unlikely Allies

   1. ETErik Torenberg0:29

      Dylan, welcome back to the podcast.

   2. DPDylan Patel0:31

      Thanks for having me. Yeah.

   3. ETErik Torenberg0:33

      Uh, it just so happens that there's some big news. Just as we're, as we're having you, Nvidia, uh, uh, announcing a five billion dollar investment in Intel and them teaming up to jointly develop custom data centers and PC products. What do you think about the collaboration?

   4. DPDylan Patel0:45

      I think, I think it's hilarious that, like, Nvidia could invest, it gets announced, and their investment's already up 30%. [laughing] A $5 billion investment, $2 billion profit already, right? Like, I think it's fun 'cause, uh, they need their customers to really, uh, have big buy-in. So when their customer- potential customers, uh, buy in and commit to certain types of products, it makes a lot of sense, right? And it's kind of, uh, funny in a way because in the past, um, there was this whole, like, thing around how Intel was sued for being, uh, anti-competitive with their chipsets, and Nvidia actually got, like, a settlement from Nvidia-- uh, Intel, right? Way back when, when, like, the graphics were separate from the GPU, and the, the gra- the graphics were really put on the chipset, which had, like, all this other IO, um, like USB and all this stuff. Um, so, so it's kind of a, a, a funny, like, turn of events that now Intel is going to make, like, a chiplet and package it alongside a chiplet from, from Nvidia, and then that's, like, a PC product, right? So, you know, it's kind of poetic that everything's gone full circle and Intel's sort of crawling to Nvidia. But actually it might just be the best, like, device, right? Um-

   5. SWSarah Wang2:00

      Absolutely

   6. DPDylan Patel2:00

      ... I don't want an ARM laptop because it can't do a lot of things. And so an X86 laptop with Nvidia graphics fully integrated would be probably the best product in the market. Um-

   7. ETErik Torenberg2:10

      So,

3. 2:11 – 4:27

   Investment and Capital in Semiconductors

   1. ETErik Torenberg2:11

      so are you optimistic? How, how do you think this will go?

   2. DPDylan Patel2:13

      I mean, sure. I mean, I hope. I hope, right? I'm, I'm, I'm, I'm a perpetual optimist on Intel because, uh, have to be. I was thinking that, uh, the structure of the deal that at least, like, a lot of the government folks and, uh, Intel were sort of trying to go for was people get-- you know, cu- big customers and the biggest suppliers directly give capital to Intel. Um, but this is sort of the other way around, where they're buying some of the stock, uh, having some ownership, but they're not really, like, diluting the other shareholders. And then the other shareholders will get diluted/everyone will get diluted when Intel finally does raise the capital from the capital markets. But because they've announced these deals, and they're pretty small, right? Five billion Nvidia, two billion SoftBank, um, US Government was 10. Um, you know, these, these are still relatively small.

   3. GAGuido Appenzeller3:00

      Pretty small, yeah.

   4. DPDylan Patel3:01

      Yeah. On, on the nature of things, right? I mean, like, uh, you know, last time I think I said Intel needs, like, $50 billion, right? Now, now when they go to the capital markets, it's, it's better. And may-- and, and, and hopefully they get another, you know, couple of these announcements, maybe, uh, you know, there, there's, there's all sorts of speculation that Trump is involved in, you know, sort of to, um, getting these companies to invest. W- uh, Nvidia, um, and now, now, you know, the government as well, of course, and now, you know, is Apple gonna come invest, right, and also do something with Intel, or who else will come in? And that'll really boost investor confidence, then they can dilute/go get debt.

   5. SWSarah Wang3:37

      Like a Warren Buffett coming into a, a stock. The Jensen is like the Buffett effect-

   6. DPDylan Patel3:42

      Yeah

   7. SWSarah Wang3:42

      ... to the semiconductor world. Um, Guido, you were the CTO of the Intel Data Center and AI BU.

   8. GAGuido Appenzeller3:49

      Yep.

   9. SWSarah Wang3:49

      What are your thoughts?

   10. GAGuido Appenzeller3:50

       I think it's really good for customers and consumers in the short term, right? Having, having both Intel and, like, specifically the laptop market, right? Having the two collaborate is, is, is amazing. Um, it-- I wonder what's gonna happen with any of the internal graphics or AI products at Intel, right? They might just push a reset and give up on that for now, right? They currently don't have anything competitive, right? There was the Gaudi effort that's more or less done, right? There was the internal graphics chips which never competed really at the high end, right? So from that perspective, it makes a lot of sense, right? It's, um, uh, for, for, for both sides. Look, I think the [coughs] for Intel, they, they needed a breath of fresh air,

4. 4:27 – 5:21

   The Impact on AMD and ARM

   1. GAGuido Appenzeller4:27

      right? They were sort of desperate, so I think it's, it's, it's a very good thing. I think AMD is fucked. [laughing] You know what I mean? They're-- You're just-- If, if your two arch nemesis suddenly team up, right [laughs] , it's the worst possible news you can have, right? They were already struggling, right? Their, their cards are good, their software stack is not, right? They, they were getting very limited traction, right? They, they now, they now have a bigger problem that side. I think ARM is a little bit screwed as well, right? Because their, their, their biggest selling point was sort of like, "Look, we can partner with everybody that doesn't want to partner with Intel." And that's what they-- In a, in a sense, they're number one. You know, like, uh, Nvidia is probably the most dangerous of the future CPU competitors, right? And so they now suddenly have access to Intel technologies and might get in that direction. It, it, it remixes the card, right? It's, it's-- Uh, I did not see this coming. I think it's, uh, it's amazing development.

   2. SWSarah Wang5:11

      Yeah. Will be very interesting to see this play out. Um, to Eric's point, packed news, uh, week. Um, [clears throat] the other thing that we wanted to pick your brain on, since we have you here,

5. 5:21 – 14:01

   China’s AI Chip Race: Huawei’s Rise

   1. SWSarah Wang5:21

      Dylan, is the other news dropping on Huawei unveiling their kind of AI roadmap. And, you know, obviously they're hyping up the capabilities. Um, I think you guys have been sort of ahead of the curve of trying to gauge, hey, what, what can the 950 supercluster actually do? Um, but would love your thoughts on everything that's going on from the China front, right? And this is kind of coupled with DeepSeek saying their next models are gonna be on domestically produced Chinese chips. The Chinese government, uh, kind of banning companies from buying the, uh, produced specifically for China Nvidia chips. So there's just sort of a lot of dominoes falling right now in the semi market in China, but would love your take overall and, I mean, drill into some, uh, detail.

   2. DPDylan Patel6:01

      Yeah. I think-When you sort of zoom out to even like, you know, let's, let's, let's walk from 2020, because I think it's really important to recognize how cracked Huawei is, or even just historically, like they've always been really good. Sure, initially they stole like Cisco, uh, source code and firmware and all this stuff, but then they rapidly passed them up as well as every other telecom company. In 2020, they released, uh, an Ascend chip and submitted to impartial public benchmarks, and they were the first to bring seven-nanometer AI chips to market. They were the first to, um, have that, right? Now, you could still say Nvidia was ahead, uh, but the gap was like, like nothing, right? And this is when they could access the full foreign supply chain. This was when they had just passed Apple to be TSMC's largest customer. They were, you know, clearly ahead of everyone on a manufacturing supply chain sort of design standpoint on, in a total basis, right? Now, of course, Nvidia still had higher market share, but it was so nascent then, like it could've, they could've really taken over the market. Huawei got banned by the Trump one administration from accessing, and then it went into effect in 2020, right? The, the full ban. And so they were only able to make a small volume of these chips, but they had trained significant models on these chips that they made then. And then over the next couple years, right, N-Nvidia continued to accelerate. Huawei, because they were banned from TSMC, had to go and try and figure out how to manufacture at SMIC, the domestic TSMC. Um, and then they were also in parallel trying to go through shell companies to manufacture, uh, at TSMC and acquire memory from Korea and so on and so forth. So by the end of '24, they had-- This had gotten in full swing and it was, uh, caught, right? It was caught and they finally shut it down. But they were able to acquire three million, uh, chips, two point nine million chips from TSMC through these other entities, right?

   3. SPSpeaker8:03

      Yeah.

   4. DPDylan Patel8:04

      Uh, roughly five hundred million dollars worth of orders, um, which, which ends up being a billion dollar fine that the US government gave TSMC, if I recall correctly. Or at least there was a Reuters article about it. I don't know if they actually, they actually issued it. Which is, which is important and interesting to gauge because the number of Ascends floating out there is not, has not consumed this entire capacity yet.

   5. SPSpeaker8:28

      Mm.

   6. DPDylan Patel8:28

      Right? So now we get to 2025, right? The H20 got banned in the beginning of the year. Um, Nvidia had to write off, you know, huge amounts of money. Uh, our, our revenue estimate for Nvidia in China for just H20 was north of twenty billion-

   7. SPSpeaker8:41

      Oh

   8. DPDylan Patel8:41

      ... because that's what they were booking in capacity slash had to write off.

   9. SPSpeaker8:44

      Yeah.

   10. DPDylan Patel8:44

       And then it got banned. They cut the supply chain. Like, they just said, "No, we're not doing this anymore." They had their inventory. It gets reapproved. They resell the inventory, but now they're like, "Do we even restart production?" Um, is, is Nvidia's question. And, and now you have China saying, "Hey, like, we don't need Nvidia. We have domestic alternatives," right? Whether it be Huawei or Cambricon.

   11. SPSpeaker9:04

       Yeah.

   12. DPDylan Patel9:04

       Um, these companies have, you know, capacity, but most of this capacity is dem-- is still foreign produced, right? Whether it be wafers from TSMC, uh, memory from S- Korea, right? Samsung and SK Hynix. So the question is sort of like how much can they do domestically? And there's sort of two fronts there, right? There's the logic, i.e. replacing TSMC, and there's the memory, i.e. replacing Hynix, Samsung, Micron. And on the logic side, they are, they're behind, but they're really ramping there, and I think they can, they can sort of get to the production capacity, uh, estimates needed, and the US is still allowing them to import all the equipment necessary pretty much. Uh, the bans are really for beyond the current generation of technol- beyond seven nanometer. The, the, the bans are really for five nanometer and below. Um, even though the government says they're for fourteen nanometer, the, the actual equipment that's banned is only for below seven nanometer. And so they'll be able to make a lot of seven nanometer A-AI chips and maybe even get to five with, you know, using existing, uh, equipment, four, five nanometer rather than using, uh, rather than like taking the new techniques. And so, like there's the logic side and then there's the memory side. And the, the aspect of Huawei's announcement that was surprising was that they're doing custom memory-

   13. SPSpeaker10:23

       Mm.

   14. DPDylan Patel10:23

       Right?

   15. SPSpeaker10:24

       Yep.

   16. DPDylan Patel10:25

       That's, that's the part that is sort of like, "Hey, this is really exciting." They announced, you know, two different types of chips for next year. Um, one that's focused on recommendation systems and prefill, and then one that's focused on, uh, decode.

   17. SPSpeaker10:38

       Th-there's the trend these days. [chuckles]

   18. DPDylan Patel10:39

       Yeah. So in, in Nvidia, the same thing. They just announced a prefill specific chip recently. Um, there's numerous AI hardware startups that are really focusing on prefill versus decode. And so the sort of split of inference into two workloads, you know, Huawei's doing the same thing for their next year chip. And what's interesting is the decode one has, you know, custom HBM. What does that mean? What is the manufacturing supply chain? 'Cause that, that's the, that's the one that's tricky, right? How much can they manufacture of that custom HBM? And Nvidia and others are also adopting custom HBM only starting next year, right? So it's not like, you know, yes, the manufacturing capacity is not there. The... Maybe, maybe it consume-- It, it is gonna consume a bit more power. It's gonna be slightly lower bandwidth. But the fact that they're able to do, you know, some of the same things that Nvidia's plans to do, AMD plans to do in their memory, uh, is, is, you know, evidence that they're catching up. But then, you know, the, the main question that remains is production capacity. So as far as like, hey, Nvidia's banned in China, right? Like they're saying, "Don't buy Nvidia chips." I think for a period of time that's fine because, uh, fine for China, right, from a perspective of, "Hey, I'm China." That's fine because you have all this capacity that you, you know, shipped in, uh, in 2024 that you haven't turned into AI chips. Now you're turning them into AI chips, you're running all that stockpile down. Um, what about the transition from running that stockpile down to ramping your new stuff, right?

   19. SPSpeaker12:00

       Right.

   20. DPDylan Patel12:00

       And that, that d- that transition is the one that's really tricky. China's either shooting itself in the foot by not purchasing Nvidia chips during that time period-

   21. SPSpeaker12:08

       Yeah

   22. DPDylan Patel12:08

       ... or China's able to ramp. Um, I think they'll be able to ramp. I think it'll take a little bit longer, um, and there will be like a sort of a gap.Uh, in between where China probably backtracks and says it's fine. Like, like ByteDance and, uh, is like begging for Nvidia chips, right? Like they, they don't wanna use, um... They use some CameraCon, they use some, uh, Huawei, but they really wanna use Nvidia because it's way better. They don't care about like the domestic supply chain. They wanna make the best models. They wanna deploy their AI as efficiently as possible. And so this is like, you know, the, the government can mandate them to like not do it, right? So, so it's not that Nvidia's not competitive, it's that the government's sort of trying to, um, instigate it. And, and then like I guess the, the, the last sort of thing is like, you know, there's always the argument of like, hey, if, if, if banning Nvidia chips to China is so good for China, why didn't China do it for itself? And they're finally doing it for themselves, so it'll... A- again, like it'll be interesting to see. Smuggling is still happening, right? Re-exportation of chips from, you know, other countries to China. That is still happening, uh, at some volume, low volume. Um, lower, lower medium volume, right? But then, you know, the direct shipments of Nvidia chips that are legally allowed to China are not necessarily happening today, but may, may have to restart at some point 'cause China won't have the production capacity to... You know, they, they would just have so many fewer AI chips being deployed domestically versus the US, and at some point you kinda have to pick like, am I, am I all about the internal supply chain or am I all about chasing, you know, super powerful AI?

   23. SPSpeaker13:46

       Yeah.

   24. GAGuido Appenzeller13:47

       So is, is, is there, is there an angle here about a negotiation angle as well? Because currently there's still discussions ongoing, what exactly are the boundaries, what can be exported to China. So these are sort of well-timed announcements, uh, if you want to make a point, um, that,

6. 14:01 – 19:00

   The HBM Bottleneck and Manufacturing

   1. GAGuido Appenzeller14:01

      you know, US should allow more exports. Is-- Do you think that's a factor or not?

   2. DPDylan Patel14:05

      Yeah. So, so I, uh, you know, in the report we did a few weeks ago about, uh, the production capacity of Huawei, um, and the supply chain, there was a bit in there that we wrote about how, you know, honestly like if you were China and you want Nvidia, you do want Nvidia chips actually. How do you play this? Right?

   3. GAGuido Appenzeller14:22

      Yeah.

   4. DPDylan Patel14:22

      And, and, and it's by-

   5. GAGuido Appenzeller14:24

      Like this. [laughs]

   6. DPDylan Patel14:24

      It's by hyping up your domestic supply chain.

   7. GAGuido Appenzeller14:26

      Exactly. Yeah. Yeah.

   8. DPDylan Patel14:27

      And it's by... It's like, it's like, yes, we can do everything. It's Huawei announced the most crazy shit possible.

   9. SPSpeaker14:32

      [laughs]

   10. DPDylan Patel14:32

       Announced seven years of fucking r- or f- three years of roadmaps that are like-

   11. GAGuido Appenzeller14:36

       So you think they read your report basically. [laughs]

   12. DPDylan Patel14:39

       No, no, no, no, no. I think, I think they do. I mean, they were already bid, and then like, say, "We're banning Nvidia," right? Like, and then it's like, then the government official's gonna think, alongside sort of like lobbying from domestic players, like, "Of course, we wanna ship them better AI chips," like, "We're losing this market. We can't lose this market."

   13. SPSpeaker14:54

       [laughs]

   14. DPDylan Patel14:55

       Um, and it's sort of like it is 10,000 IQ, right?

   15. SPSpeaker14:58

       [laughs]

   16. DPDylan Patel14:58

       And, and, and we're here playing checkers while they're playing chess.

   17. SPSpeaker15:00

       [laughs]

   18. SWSarah Wang15:01

       Well, so I guess negotiating chip aside, um, in that report you talked about HBM or high bandwidth memory being a bottleneck to Huawei. Um, to your point on one of the surprising aspects of the announcement, do you, do you think it's credible that it's no longer a bottleneck based on what they're saying or are they... is it just hype?

   19. DPDylan Patel15:20

       I think, I think production capacity-wise it is still absolutely a bottleneck. They, um... Certain types of equipment required for making HBM need to be imported. They're working on domestic solutions but as far as we know they have not imported e- enough equipment for this. Although, um, if you look at Chinese import data, uh, for different types of equipment, right? There's, there's sort of like fabs spend, you know, roughly... It depends on the process technology, but fabs spend roughly different amounts of money on lithography, etch, deposition, metrology, right? Like these different steps. Um, and historically lithography has hovered around, um, you know, 17, 18%. With EUV it's, it grew to 25%, right? Um, but China because they, they wanted, they, they sort of like wanted to stockpile lithography and they were worried about the coming ban, they were importing lithography at a much higher rate than that, right? Like 30, 40% of their equipment imports were lithography, and they were just stockpiling lithography equipment. This is sort of like reversed now in that like, hey if I want to... And, and so if you look at the monthly import export data both into provinces in China but also out of countries, uh, you can see that etch, etch, uh, specifically is skyrocketing. And, and the main thing about, um, you know, stacking HBM is that you have to, you know, when you have each wafer you have to etch, create like this thing called a through-silicon via, so it can connect from the top to bottom and then you stack them on top of each other, right? 12 high, 16 high for HBM.

   20. SWSarah Wang16:43

       Mm.

   21. DPDylan Patel16:44

       That's how you make super high bandwidth memory. And, and their imports for etch is like skyrocketing now.

   22. SWSarah Wang16:48

       Mm.

   23. DPDylan Patel16:49

       So it's like-

   24. SWSarah Wang16:49

       Continuing, yeah

   25. DPDylan Patel16:50

       ... it, it, it's, it's they don't have the production capacity yet. How fast can they ramp it as a function of how much equipment can they get, A, and B, like the yields, right? Improving yields is really hard on manufacturing. Intel and Samsung are really good and TSMC is just amazing. Not, not that those companies suck, like I think is a better way to put it. And, and so, you know, it's those two things I think. Yield, they haven't even started production of high speed, of, of HBM3, right? They, they've only done some sampling of HBM2. HBM3 came out like a few years ago. So there, there's still quite a bit of ways on like going up the learning curve. I, I, obviously I expect them to catch up faster than it took, you know, the technology to be developed 'cause it exists, right? Um, in the world. We know how to do it, it's just a mat- matter of actually doing it versus, uh, inventing it. Um, and then the other one is sort of the production capacity. Uh, you know, a couple months of import export data is not enough to, you know, set up for, you know, years' worth of supply chain build-up.

   26. SWSarah Wang17:46

       Right.

   27. DPDylan Patel17:46

       Right? Which is what we have today in, in Korea, um, for the Korean companies. Now Hynix is also investing in the US, in Illinois, and then Microns primarily in Japan, the American memory companies primarily in Japan and Taiwan, but they're also expanding in Singapore and the US now. Like, there's so much capital that's been invested, it would take some time for China to build up that production capacity to actually match the West. And when I say the West, I mean East Asia, uh, in production. It-- Non-China East Asia in production capacity. So it'll take some time to get there and I don't think... I think it's like, "Hey, we can design this." It's, it's always a question of can we manufacture? And then, and then the thing like that Jensen would say is like, "You're betting on China not being able to manufacture?" Like-

   28. SWSarah Wang18:29

       Right. [laughs]

   29. DPDylan Patel18:30

       ... you know, it's a, it's, it's a matter of when, not, uh, if.And that's the whole calculus that, like, I think the US Government has to be aware of when they're like, "Hey, what level of AI chips do we sell? Do we sell everything?" Um, probably not because AI is far more powerful and a lar- the end market of AI is gonna be way larger than the end market of semiconductors and equipment. Do we sell-- You know, what level do we sell at? Well, how much can China make at each specific, you know, sort of performance tier and then, you know, analyze that and at what's the volume, and then figure out, like, what is okay, which

7. 19:00 – 22:32

   Nvidia’s Global Competition: The Huawei Threat

   1. DPDylan Patel19:00

      is, like, maybe a little bit above or around the same level.

   2. SWSarah Wang19:03

      Yeah. So, so, um, if you-- to your point on, like, playing chess versus checkers, if you're Jensen, what would your next move be given the situation at hand?

   3. DPDylan Patel19:13

      It's both, like, tru- partially true that he's afraid of Huawei more than he is, like, an AMD.

   4. SWSarah Wang19:20

      Right. He called them formidable.

   5. DPDylan Patel19:22

      Yeah.

   6. SWSarah Wang19:22

      Yeah.

   7. DPDylan Patel19:22

      Well, well, like... I mean, like, every other, like, Huawei's beat Apple, right? They, they've passed Apple up in TSMC orders. They passed Apple up in phone market share. Um, not in the US, but, like, in many parts of the world, um, before the bans came down, and then even now they're growing back again in market share without, like, Western supply chains. Um, you know, they, they've done this to numerous other industries. I would say Apple's like a formidable competitor, right? Like, they've, they've beaten a lot of industries and so it's, it's reasonable that he's afraid of them. It's, it's sort of, you know, and he's not afraid of AMD. So, like, I think, like, the best thing is, like, try and sow as much, like, Huawei, what Huawei announced is reality-

   8. SWSarah Wang20:06

      Mm.

   9. DPDylan Patel20:06

      -rather than, like, their hoped target.

   10. SWSarah Wang20:08

       Yeah.

   11. DPDylan Patel20:08

       Um, and sow away all doubt on manufacturing capacity, which I think is not fair, right? Like, I think manufacturing capacity is a real, uh, bottleneck for them. Um, and then the yield learning's a real bottleneck, like, temporary maybe. Um, we'll see how long, and we'll see how fast the rest of the, you know, the, the Nvidia technology advances past what Huawei's capable of, right? Um, and, and how fast Huawei's able to close the gap. But I think, I think his main sort of pitch would be Huawei is, is real. They're a formidable competitor. They're going to take over not just the Chinese market, but also-

   12. SWSarah Wang20:45

       Right

   13. DPDylan Patel20:45

       ... uh, foreign markets, right? Whether it be, uh, the Middle East or Southeast Asia or South Asia or-

   14. SWSarah Wang20:51

       Totally

   15. DPDylan Patel20:52

       ... Europe or LATAM, right? Everywhere besides America. And the, the sort of-- there's a... I think, I think, uh, Noah Smith has this analogy, right? His whole idea is that you should Galapagos China, right? Make them have their own domestic industry that is so different from the rest of the world, right? Kind of what happened with Japan in the '70s and '80s.

   16. SWSarah Wang21:14

       Wow.

   17. DPDylan Patel21:14

       Their... And, and '90s. Their PCs were so specific and hyper-optimized to the Japanese market with, like, you know, the weird, like, I don't know if you've seen the weird scroll wheel on the-

   18. SWSarah Wang21:23

       Yeah

   19. DPDylan Patel21:23

       ... on these Japanese PCs. Like, you literally-

   20. SWSarah Wang21:25

       [laughing]

   21. DPDylan Patel21:25

       Like, it's like you go like this and it scrolls, right? And it's like... And then the touchpad is a circle, and then that's around it. It's like things like that are so weird.

   22. SWSarah Wang21:33

       Yeah. Totally.

   23. DPDylan Patel21:33

       And the rest of the world doesn't care, but Japan market likes it, right? And his whole idea is, like, let's Galapagos them, i.e.-

   24. SWSarah Wang21:39

       Mm

   25. DPDylan Patel21:39

       ... keep their technology within China, and then that's like deadweight loss and they never expand outside versus, uh, that we serve the whole world. Uh, but the whole risk is that the opposite can also happen, right? Our, our technology's hyper-optimized to running, you know, uh, language models at this scale and RL, and you keep, you, you keep, like, hardware, software co-design can take you down a tra- path of the tree that, like, is a dead end. And then China, like, because they're not allowed to access this tree, they're like, "Oh, okay," and then they end up in the, like, optimal spot, right? We-

   26. SWSarah Wang22:10

       Right

   27. DPDylan Patel22:10

       ... we hit a local minima, they hit a loc- uh, local maxima, they hit a local, uh, a global maxima, right? Like, that, that sort of, like, technological Galapagos-ing is sort of what Noah Smith's analogy is. I like it a lot. Um, I don't know if it's accurate, but, uh, it's an interesting one.

   28. SWSarah Wang22:24

       Yeah. I love that. Um, well, actually, maybe just taking a step back from current events, uh, even though there's so much to talk about right now.

8. 22:32 – 29:44

   Jensen’s Next Move: Nvidia’s Strategy

   1. SWSarah Wang22:32

      Um, last time you appeared with us, Nvidia came up obviously. Uh, and you talked about a couple of the potential paths forward for Nvidia.

   2. ETErik Torenberg22:41

      Give us maybe the bull case, the bear case. [laughing]

   3. SWSarah Wang22:44

      [laughing] Yeah. Fair enough.

   4. DPDylan Patel22:46

      There's a lot embedded in their numbers now. Uh, but what's interesting is, um, consensus for, uh, the banks is, is, like, for across, like, the s- the hyperscalers, so, uh, Microsoft, CoreWeave, Amazon, Google, and Oracle, right? Uh, Meta, right? So it's the six cor- hyperscalers, right, who I would consider hyperscalers. The consensus for the banks is $360 billion of spend next year across all of them. Um, and my number is closer to f- like, it's, like, 450, 500. Um, and that's, that's based on, like, you know, all the research we do on, like, data centers and, like, tracking each individual data center and the supply chains, right? So, so-

   5. ETErik Torenberg23:27

      So th- this is just Nvidia spend on-

   6. DPDylan Patel23:28

      This is, this is CapEx for the hyperscalers, right? And then that CapEx-

   7. ETErik Torenberg23:31

      Got it

   8. DPDylan Patel23:32

      ... gets split up across different companies, but the vast, vast majority still goes to Nvidia, right?

   9. SWSarah Wang23:37

      Right.

   10. DPDylan Patel23:37

       Um, and Nvidia's in a position now where they take-- they can't take share, right? It's they grow with the market/defend share.

   11. ETErik Torenberg23:44

       Yeah. [chuckles]

   12. DPDylan Patel23:44

       Um, and so the question is, like, how fast is the growth rate of, of CapEx for hyperscalers and other users, right? And the reason I included Oracle and CoreWeave as hyperscalers even though they're traditionally not called hyperscalers-

   13. SWSarah Wang23:57

       Mm

   14. DPDylan Patel23:57

       ... is because they are OpenAI's hyperscaler, right?

   15. SWSarah Wang24:01

       Right.

   16. DPDylan Patel24:01

       So, you know, when you look... And you, you look at the Oracle announcement, right? Like, uh, first of all, the Oracle announcement, I don't understand why people don't think this is crazier. They did the most unprecedented thing in the history of, like, stocks and, and public, and companies ever. They gave a four-year guidance. Um, and it made Larry the richest man in the world. You know, like, all these things-

   17. SWSarah Wang24:21

       Yeah

   18. DPDylan Patel24:21

       ... they, like, uh... Anyways, you know, the, the question is, like, how fast does revenue grow, right? Do you think Oracle and Open... Do you think OpenAI, which signed a $300 billion plus deal with, with Oracle, will actually be able to pay $300 billion, right, across raising capital and revenue?And I think most... And, and, and, and it gets to a rate of like over 80 billion, eight- over $90 billion a year, uh, in just a handful of years, right? So it's like how... Do you believe the market will grow that fast? Um, it's, it's very possible, yes, and it's very possible for like, you know, OpenAI, what is their revenue gonna be exiting next, next year? Some people think $35 billion, some people think $40 billion, some think- people think $45 billion, you know, ARR by the end of the year next year. This year they hit 20, right? Um, ARR. You know, so, so if that growth rate is maintained, then all of that cost goes to compute, plus all the capital they continue to raise, right? And again, their financials that they sort of like gave to investors for their last round was like, "Hey, we're gonna burn, we're gonna burn like $15 billion next year." It's probably more likely gonna be like 20, but like, you know, and you, you stack this on and they're not turning a cash flow, they're not gonna be profitable until 2029. So you sort of have like they're gonna continue to ba- burn $15 billion, $20 billion, $25 billion of cash each year, plus revenue growth. That's their compute spend. And you do this for Anthropic, you do this for OpenAI, you do this for all the labs. It's very possible that the pie does get to, you know, you know, more than 500 f- you know, not $360 billion next year, $500 billion next year and, uh, for ca- total CapEx and the pie continues to grow for hyperscalers. Nvidia says actually it's gonna be multiple trillions a year on AI infrastructure, and he's gonna capture a huge portion of it. That's his bull case, right? That's the bull case is, is AI is actually, um, so transformative and the world just gets covered in data centers and, and the majority of, uh, your interactions are with AI, whether it's like, you know, business productivity and t-telling an agent to do some code, or you're just talking to your AI girlfriend, Annie, right? Like, it doesn't matter. Um, you know, all of this is running on Nvidia for the most part. The bear case is, you know, even if it does grow a lot-

   19. GAGuido Appenzeller26:30

       So-

   20. DPDylan Patel26:31

       Yeah, go ahead.

   21. GAGuido Appenzeller26:31

       Save the bull case for a second. I think fundamentally the value creation, I think personally is there, right? I mean t- okay, trillions of dollars of value with AI, I, I can totally see this happen. So assume it's true, where will Nvidia top out?

   22. DPDylan Patel26:44

       I guess how, how much do you believe in takeoffs, right?

   23. GAGuido Appenzeller26:47

       Yes. [laughs]

   24. DPDylan Patel26:48

       Uh, yeah. So, so like if there, if there is like a takeoff scenario, right, where like powerful AI builds more powerful AI, builds more powerful AI or, you know, that creates more and more... You know, each level of intelligence like, uh, enables more for the economy, right? Like how many m- how many monkeys can you employ in your business versus how many like humans, right? You know, sort of the same or how many dogs, right? Like, you know, the, the, they're sort of like what is the value creation of a human versus a dog? Uh, sort of like the same with AI. So, so like a-

   25. GAGuido Appenzeller27:17

       It-

   26. DPDylan Patel27:17

       I mean, in, in this case the tr- the value creation could be hundreds of trillions, if not, you know, the next-

   27. GAGuido Appenzeller27:22

       Yeah, yeah

   28. DPDylan Patel27:22

       ... after that.

   29. GAGuido Appenzeller27:23

       But I mean the... Do you, do you need this? I mean, if you take every white collar worker and make them twice as productive with AI, that's in the hundreds of trillions, isn't it?

   30. DPDylan Patel27:31

       Yeah, but like what is-
9. 29:44 – 36:15

   Nvidia’s Moat: How They Built It

   1. SWSarah Wang29:44

      currently have. Um, and I love this sort of historic journey you took us through with Huawei just earlier. Um, can you kind of walk through what Nvidia did throughout history to build their moat?

   2. DPDylan Patel29:55

      It's super awesome because, you know, they failed multiple times in the beginning, and they bet the whole company multiple times, right? Like Jensen's just crazy enough to bet the whole company, right? Like, um, whether it was like certain chips ordering volume before he knew it w- even worked and it was like th- all the money he had left, or like ordering volumes for projects he had not won yet. Like I heard a rumor that, or not a rumor, but like a story from someone who's like a gray beard in the industry and I think would know was like, "Yeah, no, no, no, like Nvidia ordered the volume for the Xbox before Microsoft gave them the order." They're just like, they were literally just like, "Fuck it, YOLO."

   3. GAGuido Appenzeller30:37

      Yeah.

   4. SWSarah Wang30:38

      [laughs]

   5. DPDylan Patel30:38

      Um, I don't, I don't know. Like, I don't know how real true this... I'm sure there's more nuance there, like, you know, verbal indication or whatever.

   6. SWSarah Wang30:44

      [laughs]

   7. DPDylan Patel30:44

      But, like, the order was placed before he got the order, right? Like, is, is what he said. Um, you know, there's, there's cases like with the crypto bubbles, right?

   8. SWSarah Wang30:52

      Yeah.

   9. DPDylan Patel30:52

      Like, there was a couple of them, but, like, Nvidia did their damn best to convince everyone in the supply chain that it wasn't crypto and that it was gaming and that it was durable real demand, and it was da- gaming and data center and, and, uh, professional visualization, and therefore you guys should ramp your production. And they all ramped production and spent all this CapEx on increasing production and, and building out new lines for them, and they pay, they pay per item, and then they bought them and sold them at... and made shitloads of money. And then, and then when it all fell apart, they just had to write down a quarter's worth of inventory. Whatever.

   10. SWSarah Wang31:25

       Yeah.

   11. DPDylan Patel31:25

       Everyone else was like, "Well, crap, I have all these empty production lines," right? And so it's like-

   12. SWSarah Wang31:30

       Totally

   13. DPDylan Patel31:30

       ... you know. But, but, like, what did AMD do then, right? Their chips, they were actually better for crypto mining, right, on a, on a, you know, amount of silicon, uh, cost versus how much you hash. But, like, they just didn't... They, they j- AMD was like, "Ah, we're gonna not really raise production," right? Like, as a reasonable, you know, thing, right? It wasn't a... It's that sort of, like, strike while the iron's hot. And so, like, you know, the same has happened with Nvidia, right? They've, uh, in recent, in recent times, like, sort of... They've ordered capacity that no one believes, right? Multiple times. Um, they, they see the end demand obviously, but in many cases they're just like... Their number for, like, Microsoft was higher than Microsoft's internal planning, right? And, and then Microsoft's internal planning went up, but, like, their number for Microsoft was way higher, and it's like, "Ah, we just don't think Microsoft's gonna need this much, even though they tell us this."

   14. SWSarah Wang32:21

       [laughs]

   15. DPDylan Patel32:21

       It's like, who the heck is like, "No, no, no, customer, you're gonna buy more"?

   16. SWSarah Wang32:26

       [laughs]

   17. DPDylan Patel32:27

       Like, and, and orders, right? And then, and then when the orders come through the supply chain, it's like I have to put pay NCNR, right? Non-cancelable, non-returnable. Like, you know, this is-

   18. SWSarah Wang32:35

       Right

   19. DPDylan Patel32:35

       ... you know, this is, uh... I asked a question in Taiwan once. Uh, there was like a... It was, it was Colette, which is the CFO, and Jensen, CEO. They were, they were both there, um, and it was, it was a room full of, like, mostly finance bros, and they were asking stupid finance questions, like, three days before earnings. So obviously they just could not answer anything, 'cause it's like, you know, SEC regulations.

   20. SWSarah Wang32:56

       [laughs]

   21. DPDylan Patel32:56

       But then my question to them was like, "Look, Jensen, you're, like, so vibes, uh, like, driven, and, like, very gut feel and, like, very visionary, and then Colette's, you know, CFO. Like, she's, she's amazing in her own right. But, like, you know, th- that, those, those personalities clash. How do you work together?" And he's like, "I hate spreadsheets. I don't look at them. I just know," right?

   22. SWSarah Wang33:16

       [laughs]

   23. DPDylan Patel33:16

       Like, is his response. And it's like, of course, you know, the, the best innovators in the world have really good gut instinct.

   24. SWSarah Wang33:21

       Right.

   25. DPDylan Patel33:21

       Right?

   26. SWSarah Wang33:21

       Right. Totally.

   27. DPDylan Patel33:21

       And so, like, the gut instinct to, like, order with, you know, with non-cancelable when you don't know, and they've had to write down over their history multiple times, right? Many, many billions of dollars in accumulative orders, right? So accumulate in total orders, whether it be, you know, the H20, which is more regulatory, but, like, other cases they've ordered and had to cancel. Um-

   28. GAGuido Appenzeller33:44

       Y- you said many billions?

   29. DPDylan Patel33:45

       It's many billions.

   30. GAGuido Appenzeller33:45

       Peanuts. [laughs]
10. 36:15 – 39:40

    How Jensen Has Changed Over the Years

    1. SWSarah Wang36:15

       is." Um, but in this case, he, he remembers all the times they al- they almost went belly up, and he's like, "I've gotta bet. Keep making bets like that." Um, how do you think he's changed over... I mean, he's been one of the longest running CEOs, over 30... He's kind of right up there with Larry Ellison now. Um, how do you think he's changed over the last 30 years or so?

    2. DPDylan Patel36:33

       Um, I, I, I, I mean, obviously, like, I, I'm, I'm 29. I don't freaking know what he was like.Uh, I've, I've watched a lot of old interviews.

    3. SWSarah Wang36:42

       Yeah, yeah.

    4. DPDylan Patel36:43

       I won't say he wasn't-

    5. SWSarah Wang36:44

       He's been CEO longer than you've been alive. [laughs]

    6. DPDylan Patel36:46

       Yeah, exactly. Exactly. Like, uh, N- Nvidia was founded before I was born. I'm '96, right? Like, you know? Uh-

    7. SWSarah Wang36:53

       Yeah. Maybe just anything over the last couple of years-

    8. DPDylan Patel36:56

       I think, no-

    9. SWSarah Wang36:56

       ... is probably better

    10. DPDylan Patel36:56

        ... I think even, like watching old interviews, right?

    11. SWSarah Wang36:58

        Yeah.

    12. DPDylan Patel36:58

        Like, I watched a lot of old interviews, a lot of old, like, uh, presentations he's given. Uh, one thing is that he's just, like, sauced up and dripped up, like way... Like, the charisma he's gotten has only gotten stronger.

    13. SWSarah Wang37:09

        Mm.

    14. DPDylan Patel37:09

        Right? Um-

    15. SWSarah Wang37:10

        Yeah

    16. DPDylan Patel37:11

        ... which is, which is an interesting point. I don't know if it's quite relevant.

    17. SWSarah Wang37:14

        Totally agree with that, yeah.

    18. DPDylan Patel37:15

        Uh, but, like, the man, like, has learned to be a rock star more. Even though he was always charismatic-

    19. SWSarah Wang37:22

        Yeah

    20. DPDylan Patel37:22

        ... it was like he's a complete rock star now. Uh, and he was a rock star, you know, a decade ago, too. It's just people maybe didn't recognize it. I think, I think the first-

    21. SWSarah Wang37:32

        Yeah. Great point

    22. DPDylan Patel37:32

        ... live presentation that I watched, it was extreme, was like, um, it was- what's the, what's the con- it was CES, like 2014 or 2015 or whatever. Um, he's, he's, he's- it's, it's consumer electronics show. I'm, I'm like moderating, like, gaming sub- gaming hardware subReddits, right? Like-

    23. SWSarah Wang37:52

        Yeah, yeah

    24. DPDylan Patel37:52

        ... at the time, I'm a teenager. And like, the dude is like talking only about AI. He's telling, he's telling, like, all these gamers about AlexNet and self-driving cars, right? It's like, know your audience, first of all-

    25. SWSarah Wang38:05

        [laughs]

    26. DPDylan Patel38:05

        ... but also, like, like, um-

    27. SWSarah Wang38:08

        Amazing

    28. DPDylan Patel38:09

        ... it's not, it has nothing to do with consumer electronics and gaming. You know, at the time, I was also like, I was half like, "Holy crap, this is amazing," but also h- half like, "I want you to announce new gaming GPU," right?

    29. SWSarah Wang38:21

        [laughs]

    30. DPDylan Patel38:21

        Like, you know? But I know, like, on the forums, on the forums, quickly everyone was like, you know, "Screw this," you know?
11. 39:40 – 46:37

    Jensen Huang’s Leadership and Company Culture

    1. SWSarah Wang39:40

       you know, there's, there's sort of a famously loyal crew at Nvidia, even though all of the OGs could retire at this point. Um, is there anyone akin to a Gwynne Shotwell at SpaceX, or previously a Tim Cook to Steve Jobs at Apple that is at Nvidia today?

    2. DPDylan Patel39:56

       I mean, he had two co-founders, right? Like, that's, you know, let's not overlook that. Um, one of them, one of them is, like, you know, not involved and hasn't been for a long time. But the other one was involved up until just a, you know, s- few years ago, right? Uh, so it's not just Jensen running the show, right?

    3. SWSarah Wang40:11

       Totally.

    4. DPDylan Patel40:12

       Uh, although he was running the show. Um, there's quite a few people on the hardware side. Um, I've always, uh, there- there's someone at J- at Nvidia that's, like, mythical to me. Like, when you talk to the engineering teams, he leads a lot of the engineering teams. Um, uh, he is a private person, so I don't want to say his name actually. [laughs]

    5. SWSarah Wang40:30

       Fair enough. [laughs]

    6. DPDylan Patel40:32

       Um, but, you know, he, he's, he's, he's like, uh, he's like effectively, like, chief engineering officer is, like, his role. Um, and people within his org will know who he is and, um, I think, I think there are people like that. But, you know, they're, they're, they're- he's intensely loyal and there's, there's a number of these types of people. There's another fella who's, like, you know, like, there's all these, like, innovative ideas at Nvidia, and he's the guy who literally is like, "We need to get this silicon out now. We're cutting features." And that's, like, that's, like, what he's famously known for, and all the technologists at Nvidia-

    7. SWSarah Wang41:06

       Wow

    8. DPDylan Patel41:06

       ... hate him.

    9. SWSarah Wang41:07

       [laughs]

    10. DPDylan Patel41:07

        There's a, there's like a second guy. There's a second guy.

    11. SWSarah Wang41:09

        Yeah, yeah.

    12. DPDylan Patel41:09

        Also intensely loyal to Nvidia, has been around for a long time. But it's like, you know, it's sort of like when you have such a visionary company and forward, you know, one, one problem is that you get lost in the sauce, right? You know, "Oh, I wanna make this. It's gotta be perfect, amazing." And it's like, you know, you gotta have that sort of, like... And, and these people are, like, you know, obviously they're close to Jensen for a reason 'cause Jensen also believes, like, these things, right? Have the visionary future looking, but also, like, "Screw it. Cut it. We'll put it in the next one. Ship," right? Like, uh, you know, ship now, ship faster. Like, um, in, in a space like silicon, which is, like, really hard to do so. Um, and, and, and sort of like the thing about Nvidia that's always been, you know, super impressive, and it's from the beginning days, right, he's talked about this before, is their first chip, their, their first successful chip, they, they were gonna run out of money, and he had to go get money from other people, um, to even finish the development. And even then, he just had enough money, 'cause he'd already had a failed chip before this. Um, was the chip came back and it had to work, otherwise it would not, you know. And so they, they were like, 'cause they could only pay for, it's called a mask set, right? Basically, you put these, like, I'll call them stencils into the lithography tool, and then it, like, says where the patterns are, and you, you know, you put the stencil in, you deposit stuff, you etch stuff, you deposit materials on the wafer, etch it away. Um, and you put the stencil in and, like, you, you, like, tell it where to put stuff, right? And then the, the dep- deposition and etch keeps happening in those spots, and you stack dozens of layers on top of each other, and then you make up a chip. These stencils are custom to each chip.Right? And they cost today in the orders of tens and tens of billions of dollars. Uh, but even back then, it was still a lot of money.

    13. SWSarah Wang42:49

        Yeah.

    14. DPDylan Patel42:49

        Um, it w- it wasn't that much then, of course. Um, you know, it, it, it, it sort of-- he c- they could only pay for one set. Um, but the typical thing with semiconductor manufacturing is, you know, as good as you can simulate, as good as you can do all the verification, you'll send a design in and you have to change it. You-- There's gonna be something. It's, it's so hard to simulate everything perfectly, and the thing about Nvidia is they tend to just get it, right, the first time.

    15. SWSarah Wang43:16

        Yeah.

    16. GAGuido Appenzeller43:16

        Yeah.

    17. DPDylan Patel43:16

        Um, e- even like, even great executing companies like AMD or Broadcom or whoever, they often have to ship, you know... They're, they're denoted in, like, A and then a number or B and then a number, so it's like two different parts of the masks. So, like, Nvidia always ships A0. Almost always. They sometimes ship A1. And a lot of times, even if-- They, they'll start production of the, you know, the A is basically the transistor layer, then the number's like the wiring that connects all the transistors together. So Nvidia will start production of the A and ramp it really high, and then just hold it right before you transition to the metal, just in case they do need to change the metal layers.

    18. SWSarah Wang43:53

        Hmm.

    19. DPDylan Patel43:53

        And so, like, the moment they're ready and they've confirmed that it works, they can just, you know, blast through a lot of production, whereas everyone else is like, "Oh, let's get the chip back."

    20. SWSarah Wang44:01

        Hmm.

    21. DPDylan Patel44:01

        "Oh, okay, A0 doesn't work. We gotta make this tweak, make this tweak, and get the chip back."

    22. GAGuido Appenzeller44:04

        A step, yeah.

    23. DPDylan Patel44:05

        Um, it's called a stepping, right? Um-

    24. GAGuido Appenzeller44:07

        We, we, at Intel, we were very jealous of, of Nvidia at that time, right? They consistently delivered-

    25. SWSarah Wang44:11

        [laughs]

    26. GAGuido Appenzeller44:11

        ...and the first one we did not.

    27. SWSarah Wang44:12

        [laughs]

    28. GAGuido Appenzeller44:12

        So I think it was, uh-

    29. DPDylan Patel44:13

        The, the, the, the data center CPU group, uh, there was one product where, you know, I said A1, A, you know, A0, A1, or you go to B if it's you have to change the transistor layer as well. So it's like B. Nvidia-- Uh, sorry. Uh, Intel got to, like, E2 once.

    30. SWSarah Wang44:29

        Hmm.
12. 46:37 – 56:11

    The Future of Nvidia: Cash, Data Centers, and AI Infrastructure

    1. SWSarah Wang46:37

       more than 10 years ago talking about self-driving cars. Um, but you know, if you think about nailing the video game tailwind, VR, Bitcoin mining, obviously AI now, um, you know, one thing that... or one of the things that Jensen talks about today is robotics, AI factories. Um, maybe my last question on Nvidia, what do you gu- what do you think about the next 10 to 15 years? Um, I know calling beyond five is hard. Um, but, like, what does Nvidia, Nvidia's business look like?

    2. DPDylan Patel47:05

       Um, it's, it's, it's really a question of... And this is, like, I think every time I've talked to, uh, you know, some executives at S- uh, Nvidia have asked this question 'cause I really wanna know and, you know, they won't answer it obviously.

    3. SWSarah Wang47:21

       [laughs]

    4. DPDylan Patel47:21

       But it's like, what are you gonna do with your balance sheet? Like, you are the most high cash flow company and, like, like, you have so much cash flow.

    5. SWSarah Wang47:30

       Right.

    6. DPDylan Patel47:30

       Um, now the hyperscalers are all taking their cash flow, like, way down, right?

    7. SWSarah Wang47:35

       Right.

    8. DPDylan Patel47:35

       Uh, 'cause they're spending on GPUs.

    9. SWSarah Wang47:37

       [laughs]

    10. DPDylan Patel47:37

        Um, what, what is, what, what are you gonna do with all this cash flow, right? Like, you know. Even, even before this whole takeoff, he wasn't allowed to buy ARM, right? Um, so, so what can he do with all this capital and all this cash, right? Even this $5 billion investment in Intel is, uh, there's regulatory scrutiny there, right? Like-

    11. SWSarah Wang47:58

        Yeah.

    12. GAGuido Appenzeller47:58

        Yeah.

    13. DPDylan Patel47:58

        Um, it's, it's in the announcement, like, yeah, this is subject to review, right? Like-

    14. SWSarah Wang48:02

        Yeah

    15. DPDylan Patel48:02

        ...you know, I, I imagine that'll get passed, but, like, he can't buy anything big. He's gonna have hundreds of billions of dollars of cash on his balance sheet. What do you do? Is it, is it start to build AI infrastructure in data centers? Maybe. Um, but, like, why would you do that if you can just get, uh, other people to do it, um, and just take the cash? Uh-

    16. GAGuido Appenzeller48:22

        Well, he's investing in those, right?

    17. DPDylan Patel48:24

        Investing peanuts.

    18. SWSarah Wang48:25

        Hmm.

    19. DPDylan Patel48:26

        Right? You, you know, like, he, he gave recently, like, uh, CoreWeave a backstop, uh, 'cause, 'cause today it's really hard to find a large number of GPUs for burst capacity.

    20. GAGuido Appenzeller48:36

        Yeah.

    21. DPDylan Patel48:36

        Right? Like, hey, I wanna train a model for three months.

    22. SWSarah Wang48:39

        Hmm.

    23. DPDylan Patel48:39

        Right? I have my base capacity where I don't know my experiments, but I wanna train a big model, three months, done.

    24. GAGuido Appenzeller48:43

        We know from our portfolio. Yeah. [laughs]

    25. DPDylan Patel48:44

        Yeah, yeah. So, so, so, like, Nvidia sees this issue. They think it's a real problem with startups. It's why the labs have such an advantage. Uh, but what if I could-You know, right now, like, you know, most companies in the, uh, in the Valley spend, what, 75% of their round on GPUs, right? Or at least-

    26. SPSpeaker49:00

        At least, yeah. [laughs]

    27. DPDylan Patel49:00

        Yeah. We're seeing...

    28. SPSpeaker49:02

        [laughs]

    29. DPDylan Patel49:02

        Um, what if you could do 75% in three months on one model run, right-

    30. SPSpeaker49:05

        Mm.
13. 56:11 – 1:03:01

    The Hyperscalers: Amazon, Oracle, and the Cloud Wars

    1. SWSarah Wang56:11

       comment is actually a really good, uh, segue into something else we wanted to talk to you about, which, um, is the hyperscalers. Uh, and one of the reasons that I love reading SemiAnalysis is you guys make these out of consensus calls that you're often right about. Um, and one of them recently-

    2. DPDylan Patel56:28

       Wow

    3. SWSarah Wang56:28

       ... was calling-

    4. DPDylan Patel56:29

       Only often? [laughing]

    5. SWSarah Wang56:32

       But you have a Jensen hit rate. It's very high. Um, but, uh-

    6. DPDylan Patel56:35

       Where's my billion-dollar, you know, uh, PV positive bet? [laughing]

    7. SWSarah Wang56:40

       [laughing] Um, but, uh, the one that caught my eye was, uh, Amazon's AI resurgence. Um, so I wanted to talk to you a little bit about that just because, you know, I think we found it pretty interesting being on the ground helping our portfolio companies pick who their partners are. Um, and so we have some micro data on this, but you sort of walk through why they're behind.

    8. DPDylan Patel57:02

       Yeah. So in, um, Q1 2023, I wrote an article called, uh, Amazon's Cloud Crisis. Um, and it was about all these neo clouds are gonna commoditize Amazon. Um, it was about how Amazon's entire infrastructure was really good for the last era of computing, right?

    9. SWSarah Wang57:21

       Mm-hmm.

    10. DPDylan Patel57:21

        What they do with their, um, Elastic Fabric, um, ENA and EFA, right, their NICs, uh, what they... And the, the whole protocol and everything behind them, what they do for, uh, custom CPUs, um, et cetera, right? Like it was really good for the last era of scale-out computing and not this era of sort of scale-up AI infra, um, and how neo clouds were gonna commoditize them and how their silicon teams were focused on, you know, cost optimization, whereas the name of the game today is, uh, max performance per cost, right? But like that often means you just drive up performance like crazy. Uh, even if cost doubles, you drive up performance more triples, uh, because then the cost per performance falls still. Uh, that's sort of the name of the game today with Nvidia's hardware. Um, and it, it ended up being like really good call. Everyone like, like was calling us out like, "No, you're wrong," because... And this was like when Amazon was like th- like the best stock, and Microsoft really hadn't like started taking off yet and, and nor had like all these other, um, you know, Oracle and so on and so forth. And, and since then, Amazon has been the worst performing, uh, hyperscaler. Um-

    11. SWSarah Wang58:25

        Totally

    12. DPDylan Patel58:26

        ... and the call here is that, you know, they, they still have structural issues, right? They still use, um, Elastic Fabric, although that's getting better. Uh, still behind Nvidia's networking, still behind Broadcom's, uh, slash Arista like type networking NICs. Um, they still use... You know, their, their, their internal AI chip is okay, but the main thing is that they're now waking up and being able to actually capture business, right? So the main call here is that, uh, since, since that p- report, um, AWS has been decelerating revenue. Year-on-year revenue has been falling consistently. And, and our big call is that it's actually going to start re-accelerating, right? And that's because of, um, Anthropic, it's because of all the work we do on data centers, right? Tracking every single data center, one that goes online and what's in there. Uh, the flow through on cost, right? If you know how much the chips cost, the networking cost, the power cost, uh, you, you know how much, you know, generally margins are for these things, then you can sort of, uh, start estimating revenue. So when we build all that up, it's very clear to us that they trough on, um, AWS revenue growth this quarter, right?

    13. SWSarah Wang59:32

        Mm.

    14. DPDylan Patel59:32

        This is the lowest AWS revenue growth-

    15. SWSarah Wang59:34

        That's what I'm saying, yeah

    16. DPDylan Patel59:34

        ... will be year, uh, on a year-on-year basis for at least the next year, right? Um, and it's re-accelerating to north of 20% again, um, because of all these massive data centers they have online with, uh, Trainium and GPUs, right? Depends on which one. Depends on which customer. Um, the experience is not as good as, you know, say a CoreWeave or whatever, but the name of the game is capacity today. Um, CoreWeave can only deploy so much. They have to get... They only ha- can get so much data center capacity, and they're really fast at building. But the company with the most data center capacity in the world, that, and still today, although, um, it, they may get passed up in the next two years, um, is Amazon. Actually, they will get passed up based on what we see is Amazon, but incrementally, Amazon still has the most spare data center capacity that's going to ramp into AI revenue over the next year.

    17. GAGuido Appenzeller1:00:25

        The... Is... Let, let me ask one question. Is that the right type of data center capacity? Like for the high density AI build outs today, you need, you know, massively more cooling. You need to have enough water close by, you need to have enough power close by. Is, is that, uh, is it in the right place or is it, is it-

    18. DPDylan Patel1:00:38

        So, so-

    19. GAGuido Appenzeller1:00:38

        ... the wrong type of data?

    20. DPDylan Patel1:00:39

        So data center capacity, um, in this sense, I mean all the way from power secured, to substations built, to transformers, to, uh, you can provide the power whips to the racks. Now, obviously the data center capacity will differ, right? Um, you know, historically, actually, Amazon's had the highest density data centers in the world.

    21. GAGuido Appenzeller1:00:57

        Yeah.

    22. DPDylan Patel1:00:58

        Right? Uh, they went to like 40 kilowatt racks when everyone was still at 12.

    23. SWSarah Wang1:01:03

        Mm.

    24. DPDylan Patel1:01:03

        And if you've ever stepped inside of, foot inside of most data centers, they're like pretty cool.Um, and dry-ish. If you step inside of a Amazon data center, they feel like a swamp. It feels like where I grew up, right? It's like, uh, it's like, it's like humid and hot.

    25. SWSarah Wang1:01:17

        Mm.

    26. DPDylan Patel1:01:17

        Uh, because they're like optimizing every percentage. And so sort of like your point in here is that like Amazon's data centers aren't equipped for the new type of infrastructure, but when you compare them to the cost of the GPU, like getting, getting... You know, having a complex cooling arrangement is fine, right? Um, you know, we made a call on Astera Labs a few months ago, a couple of months ago, when they were like at 90, and it's, it's gone to 250 the month after, uh, because of what they're-- what am- orders Amazon is placing with them. But there's certain things with Amazon's infrastructure, I won't get too much into it, but their rack infrastructure requires them using a lot more of like Astera Labs connectivity, uh, products. Um, and the same applies to cooling, right?

    27. SWSarah Wang1:01:58

        Mm-hmm.

    28. DPDylan Patel1:01:58

        So it's on the networking and cooling side. They just have to use a lot more of this stuff. But again, this stuff is inconsequential on cost compared to the GPU. Um-

    29. SWSarah Wang1:02:07

        Yeah.

    30. DPDylan Patel1:02:08

        You can build, right? My question was more like, look, uh, I may need a major river close by for cooling at this point, right? Uh, it's in many areas I just can't get enough water, and you know-
14. 1:03:01 – 1:07:40

    The Era of Mega Data Centers

    1. SWSarah Wang1:03:01

       How, how important has Anthropic been to the co-design for Trainium? 'Cause I, I remember we had a portfolio company, this was summer 2023, they invited them to AWS. They spent, man, I think eight hours with them over the course of, uh, a week trying to figure out Trainium. Back then it was just impossible to work through. Um, is that, you know, that-- Obviously that portfolio company hasn't gone back and, and tried it now, but like how, how different is it now based on what you're hearing? And-

    2. DPDylan Patel1:03:29

       Oh, it's still bad.

    3. SWSarah Wang1:03:30

       Okay.

    4. DPDylan Patel1:03:31

       It's still bad.

    5. SWSarah Wang1:03:31

       Okay. [chuckles]

    6. DPDylan Patel1:03:32

       Um-

    7. SWSarah Wang1:03:33

       Good

    8. DPDylan Patel1:03:33

       ... you know, it's, it's tough to use. Um, so there's sort of like... This is sort of the argument that every inference company offers, right?

    9. SWSarah Wang1:03:41

       Yeah.

    10. DPDylan Patel1:03:41

        I-including the AI hardware startups, is because I'm only running like three different models at most, I can just hand optimize everything and write kernels for everything and even like go down to like an assembly level, right?

    11. SWSarah Wang1:03:54

        How hard can it be, yeah.

    12. DPDylan Patel1:03:55

        It is, it is pretty hard. It is pretty hard.

    13. SWSarah Wang1:03:57

        [laughs]

    14. DPDylan Patel1:03:57

        Uh, but like you tend to do this for production inference anyways.

    15. SWSarah Wang1:04:01

        Mm.

    16. DPDylan Patel1:04:02

        Like you aren't using cuDNN, which is NVIDIA's like library that's like super easy to generate your, you know, to generate kernels and stuff, right? Like you're not... Or not generate kernels, but anyways, um, you're, you're, you're still us- you're not using these like ease of use libraries. You know, when you're running inference, you're either, um, you know, using Cutlass or stamping out your own PTX or, you know, in some cases people are even going down to the SaaS level, right? Um, and like when you look at like, say, an OpenAI or like, you know, an Anthropic, when they run inference on GPUs, they're doing this, right?

    17. SWSarah Wang1:04:36

        Mm.

    18. DPDylan Patel1:04:36

        Um, and the ecosystem is not that amazing, uh, when you-- Once you get all the way down to that level. It's not like, it's not like using NVIDIA GPUs is, is easy now. I mean, you have an intuitive understanding of the hardware architecture because you've worked on it so much and everyone's worked on it, and you can talk to other people, but at the end of the day, it's not like easy, right? Whereas, you know, Anthropic Trainium or TPUs, actually the hardware architecture is a little bit more simple than a GPU. Um, larger, more simple cores, rather than having all this functionality. Um, you know, less general. Uh, so it's a little bit easier to code on. Um, there's, there's tweets from Anthropic people saying they, uh, when they are doing that low level, actually they prefer working on Trainium and TPU because of the simplicity.

    19. SWSarah Wang1:05:20

        Really? Huh.

    20. DPDylan Patel1:05:21

        Um, now-

    21. SWSarah Wang1:05:21

        Interesting

    22. DPDylan Patel1:05:22

        ... to be clear, Trainium and TPU at... I mean T- Trainium especially is very hard to use.

    23. SWSarah Wang1:05:27

        Yeah. Okay.

    24. DPDylan Patel1:05:28

        Like not for the faint of heart. Um, it's, it's very difficult, but you can do it if you're just running like... If I'm Anthropic and I must only run Claude 4.1 Opus for S-uh, Sonnet, and, and screw it, I won't even run Haiku. I'll just run Haiku on like on, on GPUs or whatever, right? I'm just gonna run two models. And actually screw it, I'm just gonna run Opus on GPUs too, and, and True TPUs. Sonnet is the majority of my traffic anyways. I could, I could spend the time.

    25. SWSarah Wang1:05:55

        Mm.

    26. DPDylan Patel1:05:55

        And how often am I changing that architecture? Every-

    27. SWSarah Wang1:05:57

        Right

    28. DPDylan Patel1:05:57

        ... four or six months, right? Like how much-

    29. SWSarah Wang1:06:00

        It's, it's not even changing that much, honestly, right? Uh-

    30. DPDylan Patel1:06:03

        I think, I think from three to four definitely did change, right?
15. 1:07:40 – 1:16:03

    Hardware Cycles: GB200, Blackwell, and the Next Generation

    1. SWSarah Wang1:07:40

       Uh, can you walk us through why you made that call then, um, and just sort of why Oracle is poised to do so well in a such a competitive space?

    2. DPDylan Patel1:07:50

       Yeah. So Oracle, they're the largest balance sheet in the industry that is not dogmatic to any type of hardware, right?

    3. SWSarah Wang1:07:58

       Mm.

    4. DPDylan Patel1:07:58

       Um, they're not dogmatic to any type of networking. They will deploy, uh, Ethernet with Arista. They'll deploy Ethernet-

    5. SWSarah Wang1:08:09

       Mm

    6. DPDylan Patel1:08:09

       ... through their own white boxes. They'll deploy Nvidia networking, um, Infiniband or, or Spectrum X. Um, and they have really good network engineers. They have really great software across the board, right? Again, like ClusterMax. Um, they were, they were ClusterMax Gold because their software is great. There's a couple things that they needed to add, uh, that would take them higher, and they're, they're adding those, right? Um, to Platinum, right? Which was where CoreWeave was. Um, and so, like, when you, you couple, you couple two things, right? Like, OpenAI's got insane compute demand. Uh, Microsoft is quite pansy. Um, they're not willing to, uh, invest in... They don't believe OpenAI can actually pay the amount of money, right? I mentioned earlier, right?

    7. SWSarah Wang1:08:49

       Right. [laughs]

    8. DPDylan Patel1:08:49

       The three hundred billion dollar deal-

    9. SWSarah Wang1:08:51

       Yeah

    10. DPDylan Patel1:08:51

        ... OpenAI, you don't have three hundred billion dollars, and Oracle's willing to take the bet. Now, of course, um, the bet is a bit like there's a bit more security in the bet in that, um, Oracle really only needs to secure the data center capacity, right?

    11. SWSarah Wang1:09:04

        Mm.

    12. DPDylan Patel1:09:04

        So, so this is sort of like how we, how we came across the bet, right, is and, and we've been telling our institutional clients, especially in like a super detailed way, whether it be the hyperscalers or AI labs or semiconductor companies or, you know, investors, uh, in our data center model because we're tracking every single data center in the world. Uh, Oracle doesn't build their own data centers either, right, by the way. They, they get them from other companies. They co-engineer, uh, but they don't physically build them themselves. And so they're quite nimble in terms of like being able to assess new data centers, engineer them. So we, we saw all these different data centers Oracle was snatching up in deep discussions, snatching up, signing, et cetera. And so we have, you know, hey, gigawatt here, gigawatt there, gigawatt there, right? Um, Abilene, you know, two gigawatts, right? You know, you-

    13. SWSarah Wang1:09:44

        Yeah

    14. DPDylan Patel1:09:44

        ... all these different sites-

    15. SWSarah Wang1:09:45

        Yeah

    16. DPDylan Patel1:09:46

        ... that they're, they're signing up in discussions with, and we're, we're noting them, and then we have the timeline because we're tracking the entire supply chain. We're tracking all the permits, uh, regulatory filings, you know, um, through, you know, language models, u- using satellite photos constantly, um, and then supply chain of like chillers, transformer equipment, um, generators, et cetera. Um, we're able to make a pretty strong estimate of quarter by quarter in our data center model, quarter by quarter, how much power there is for each of these sites, right? So some of these sites that we know of aren't even ramping until twenty twenty-seven, uh, but we know that Oracle signed it, right? Um, and we, we have this sort of ramp path. So then it's this question of like, okay, let's say you have a... you, you have a megawatt, right? Like for simple sake, simplicity's sake, which is a ton of power, but now it doesn't feel like much. It's-

    17. SWSarah Wang1:10:32

        [laughs]

    18. DPDylan Patel1:10:32

        ... you know, we're in the gigawatt era.

    19. SWSarah Wang1:10:34

        The gigawatt. Yeah.

    20. DPDylan Patel1:10:34

        But, you know, if you're talking about a megawatt, right, um, you fill it up with GPUs. How much do the GPUs for a megawatt cost, right? Uh, or actually it's even simpler to do the math, right? If I'm talking about, um, a GB200, right? Each individual GPU is twelve hundred watts. Uh, but when you talk about the CPU, the whole system, it's roughly two thousand watts. Um, at the same time, you know, all in, everything, simplicity sake, fifty thousand dollars per GPU, right? The GPU doesn't cost them. There's all the peripheries, right? Um, so fifty thousand dollars CapEx for two thousand watts. So twenty-five thousand dollars for one, one thousand watts. Um, and, and then what's the rental price for GPU? Um, if you're on a really long-term deal volume two seventy, right? Two sixty-

    21. SWSarah Wang1:11:21

        Mm

    22. DPDylan Patel1:11:21

        ... in that range. Um, then you end up with, oh, it costs like twelve million dollars per megawatt, uh, to rent a megawatt.

    23. SWSarah Wang1:11:28

        Yeah.

    24. DPDylan Patel1:11:29

        Uh, and then you f- and then each chip is different. So we track each chip, what the CapEx is, what the networking is. So you know what each chip is. You can predict what each, you know, what chips they're putting in which data centers, what- when those data centers go online, how many megawatts by quarter, and then you end up with, oh, well, Stargate goes online in this time period. They're gonna start renting at this time. It's this many chips. Each Stargate site, right? Um, and so therefore, this is how much OpenAI would have to spend to rent it, and then you, you, you prick that out, and we were able to predict, um, Oracle's revenue with pretty high certainty, and we matched pretty dead on what they announced for '25, '26, '27.

    25. SWSarah Wang1:12:04

        Mm.

    26. DPDylan Patel1:12:04

        And we were pretty close on '28. The, the surprise for us was that, you know, they announced some stuff that twenty-eight, twenty-nine data centers that they, uh, we don't, we haven't found yet, but we'll find them, right, of course. Uh, um, and sort of like this methodology lets you see-

    27. SWSarah Wang1:12:17

        Yeah

    28. DPDylan Patel1:12:18

        ... sort of, hey, what data centers are you getting? How much power? Um, what are they signing? Uh, how much incremental revenue that is when that comes online. And so that's sort of the basis of our Oracle, uh, bet. Um, obviously in the newsletter we included a lot less detail, um, but, you know, you know, sort of i- it was, it was that thesis, right? That like, hey, they have all this capacity. They're gonna sign these deals. Um, in, in our, in our newsletter, we talked about two main things. We talked about the OpenAI business, and then we talked about the ByteDance business. Um, and presumably tomorrow or, you know, uh, on Friday there's gonna be an announcement about TikTok and all this, but like the ByteDance business, um, you know, huge amounts of data center capacity that Oracle is also gonna lease out, uh, to ByteDance, right? And so we did the same methodology there. Um, you know, with ByteDance it's pretty certain they'll pay 'cause they're a profitable company. With OpenAI, it's not, and so there's gotta be some like error bars as you go further out in terms of like-Will OpenAI exist in '28, '29, uh, '30? And will they be able to pay the 80 plus billion dollars a year that they've signed up to Oracle with, right? That's the only, like, risk here.

    29. SWSarah Wang1:13:24

        Yeah.

    30. DPDylan Patel1:13:24

        Uh, and if that happens, then Oracle's downside is also somewhat protected because they only signed the data center, which is a minority of the cost, right? The GPUs are everything. And the GPUs they purchase one to two quarters before they start renting them.
16. 1:16:03 – 1:22:06

    xAI’s Colossus 2

    1. SWSarah Wang1:16:03

       you know, on the topic of these giant data center build outs, um, you guys just released a piece on xAI and, um, Colossus 2. Do you... Are you getting less impressed by these feats of building something this massive in six months? Or is it still very impressive to you guys?

    2. DPDylan Patel1:16:21

       Um, you know, th- this is the, like, uh, thing I've said about AI researchers is that they're, like, the first class of humans to think about things on an order of magnitude scale.

    3. SWSarah Wang1:16:29

       Yeah.

    4. DPDylan Patel1:16:30

       Whereas, like, people have always thought about things in terms of, like, percentage growth, like, ever since industrialization, and before that it was just, like, absolute numbers, right? Uh-

    5. SWSarah Wang1:16:39

       Yeah. [laughs]

    6. DPDylan Patel1:16:40

       ... you know, sort of like humanity is evolving in terms of how we think because things are changing faster.

    7. SPSpeaker1:16:45

       Everything is in log scale, right? [laughs]

    8. SWSarah Wang1:16:48

       [laughs]

    9. DPDylan Patel1:16:48

       Yeah. And so, like, you know, it was, like, really impressive when GPT, uh, you know, 2 was trained on so many chips, and then GPT-3 was trained on that, you know, like, on, on 20K A100s and, you know... Or sorry, GPT-4, 20K A100s, GPT... You know, sort of like... It's like, holy crap. And then it was like, oh, the era of 100K GPUs clusters, right? And we did some reports around 100K GPU clusters. But now there's like, there's like 10 G- 800K GPU clusters-

    10. SWSarah Wang1:17:15

        Exactly

    11. DPDylan Patel1:17:15

        ... in the world. I was like, okay, this is kind of boring.

    12. SWSarah Wang1:17:18

        [laughs]

    13. DPDylan Patel1:17:19

        But it's like 100K GPUs-

    14. SWSarah Wang1:17:20

        [laughs]

    15. DPDylan Patel1:17:20

        ... is like, you know, over 100 megawatts. Now it's like, you know, you know, like, literally-

    16. SWSarah Wang1:17:26

        Yeah

    17. DPDylan Patel1:17:26

        ... you know, i- we, in our, in our Slack, in, in some of these channels, like, oh, we found another 200 megawatt data center. There, there, there's a, there's someone who, like, puts the yawning emoji-

    18. SWSarah Wang1:17:35

        [laughs]

    19. DPDylan Patel1:17:36

        ... every time. And I'm like, dude, what... [laughs] Like, now it's only, it's only exciting if you do gigawatt scale data centers.

    20. SWSarah Wang1:17:42

        Like, we're in gigawatt era. Yeah.

    21. DPDylan Patel1:17:44

        Yeah, yeah.

    22. SWSarah Wang1:17:44

        [laughs]

    23. DPDylan Patel1:17:44

        And, and I'm sure, like, you know, in... You know, I'm not sure. Maybe, maybe we'll start yawning to that too. But, like-

    24. SWSarah Wang1:17:50

        [laughs]

    25. DPDylan Patel1:17:50

        ... you know, the log scale of this is like-

    26. SWSarah Wang1:17:52

        Yeah

    27. DPDylan Patel1:17:53

        ... the capital numbers are crazy, right? Like, you know, it's like, it's crazy enough that OpenAI did, like, $100 billion training run. Um, you know, or, or, you know, like, and then they did a billion dollar training run. Now we're talking about $10 billion training runs, right? Like, you know, it's, it's, it's crazy that we think in log scale. But yes, uh, things are only impressive-

    28. SWSarah Wang1:18:11

        Yeah

    29. DPDylan Patel1:18:12

        ... when they do it... Like, what Elon's doing, so what, what Elon's doing in, in, in, in, uh, Tennessee, in Memphis, first time was crazy.

    30. SWSarah Wang1:18:19

        Yeah.
17. 1:22:06 – 1:34:49

    Recommendations to Start-Ups

    1. SWSarah Wang1:22:06

       Um, well, uh, I guess on the topic of just maybe new hardware, um, you had this piece analyzing TCO for GB200s. Um, and I'm kinda gonna ask this question on behalf of our portfolio companies, which it sounds like you're helping them already. Um, but one of the findings that I thought was really interesting was TCO was sort of one point six- six X, uh, H100s for GB200s. Um, and so obviously, you know, th- there's this point on, okay, that's sort of the benchmark for the performance boost that you're gonna need to at least make the sort of performance cost, uh, ratio benefit, um, from switching over. Maybe just talk about what you've seen, um, from a performance standpoint, and what do you recommend to portfolio companies, maybe on a smaller scale than xAI, who are, you know, thinking about new hardware, try to get it. There's capacity constraints, obviously.

    2. DPDylan Patel1:22:57

       Yeah. I mean, um, that's the challenge, right? Is, uh, with each generation of GPU, it gets so much faster, um, that you end up like you want the new one. And it, and, and, you know, in some metrics, you could say GB200 is three times faster than, or two times faster than the prior generation. Other metrics, you can say it's way more than that, right? Um, so if you're doing pre-training versus inference, right?

    3. SWSarah Wang1:23:22

       They can run everything four bits, right?

    4. DPDylan Patel1:23:24

       Yeah. If, if you can run it four bit or just inference and take advantage of the huge NVLink, uh, s- NVL 72-

    5. SWSarah Wang1:23:31

       Yep

    6. DPDylan Patel1:23:31

       ... you know. Um, there, there's, there's ways you can, you could squint and say GB200 is only two X faster than H100, uh, in which case one point six X TCO.

    7. SWSarah Wang1:23:40

       Right.

    8. DPDylan Patel1:23:40

       It's, you know, it's worthwhile, right?

    9. SWSarah Wang1:23:42

       Yeah.

    10. DPDylan Patel1:23:43

        It's worth going to the next gen.

    11. SWSarah Wang1:23:44

        But more marginal.

    12. DPDylan Patel1:23:45

        It's more marginal.

    13. SWSarah Wang1:23:46

        Yeah.

    14. DPDylan Patel1:23:46

        Um, it's not a big deal. Then there's other cases where it's like well on, um, if you're running DeepSeek inference, the performance difference per GPU is like north of like six, seven X, and it continues to optimize, um, you know, for, for DeepSeek inference. Um, and so the que- you know, then, then it's like, well, sh- I'm only paying sixty percent more for six X. You know, it's like-

    15. SWSarah Wang1:24:07

        Right

    16. DPDylan Patel1:24:07

        ... it's a four X or three X performance per dollar gain. Like absolutely, right? And if you're like in running inference of DeepSeek-

    17. SWSarah Wang1:24:13

        Yeah

    18. DPDylan Patel1:24:13

        ... that could also include RL, right?

    19. SWSarah Wang1:24:14

        Right.

    20. DPDylan Patel1:24:15

        Um, and so the question is sort of... And then, and then the, the other question is like, well, the GPU is new. You know, there's also B200. There's GB200, there's B200. B200 is much more simple from a hardware perspective. It's just eight GPUs in a box, so then it's not as much of a performance gain, especially in inference.

    21. SWSarah Wang1:24:30

        Mm.

    22. DPDylan Patel1:24:30

        But you have, um, you have all the stability, right? It's an eight GPU box. It's not gonna be unreliable. The GB200s are still having some reliability challenges. Those are being worked through. It's getting better and better by the day, uh, but it's still a challenge. Um, but you know, when, when you have a GB2... When you have a, uh, H100, right, box or H200, eight GPUs, one of them fails, you take the entire server offline. You have to fix it, right? So usually your... If your cloud's good, they'll swap it in.

    23. SWSarah Wang1:25:00

        Mm-hmm.

    24. DPDylan Patel1:25:00

        Right? Um, but if, if it's GB200, what do you, what do you now do with 72 GPUs? If one fails, do you brick the whole thing and get a new semi truck? The blast radius of a failure, right? Nope. GPU failure rates at best are the same and likely worse, right? Gen on gen because thing, everything's getting hotter faster, et cetera. So at best, the failure rates are the same. Even if you model the failure rates as the exact same because you go from one out of eight to one out of 72, it's a huge problem. So now what a lot of people are doing is they run a high priority workload on 64 of them-And then the other, um, eight, you run low-priority workloads, which is then like, okay, this is this whole like infrastructure challenge. Like-

    25. SPSpeaker1:25:37

        Mm.

    26. DPDylan Patel1:25:37

        I have to have high-priority workloads.

    27. SPSpeaker1:25:38

        Yeah.

    28. DPDylan Patel1:25:38

        I have to have low-priority workloads. When a high-priority workload has a failure, instead of taking the whole rack offline, you just take some of the GPUs from the low-priority one, put it in the high-priority one, and then like you just let the dead GPU sit there until you service the rack at a later date. And it's like there's all these like complicated infrastructure things that make it so... Oh, wait, actually, that, that two-- that three X or two X performance increase in tr- pre-training-

    29. SPSpeaker1:26:03

        Yeah

    30. DPDylan Patel1:26:04

        ... is lower because the downtime is higher/
18. 1:34:49 – 1:38:48

    The State of the GPU Market Today

    1. DPDylan Patel1:34:49

       right.

    2. SWSarah Wang1:34:49

       Yeah.

    3. ETErik Torenberg1:34:49

       That's it.

    4. SWSarah Wang1:34:49

       Well, so I, I love that we're actually going into all this detail because I had a more ten thousand foot view, uh, question for you, which is, um, I haven't been following the semi market as closely as you have. I probably started with the A100, um, and I, I remember helping Gnome at Character, this is summer of June 2023, um, chase down GPUs. And the only thing that mattered at that time was delivery date because there was a huge capacity crunch. Um, and then to see that over the last two years evolve where, you know, let's say six to twelve months ago, people were doing these RFPs to twenty Neoclouds, right? And the only thing that mattered to some degree was price.

    5. DPDylan Patel1:35:31

       Wait, people actually do RFPs for GPUs?

    6. SWSarah Wang1:35:33

       Yes.

    7. DPDylan Patel1:35:34

       So, so, so just to be clear, my opinion on, uh, how you buy GPUs is that it's like buying cocaine, um, or any other drug.

    8. SWSarah Wang1:35:41

       [laughs]

    9. DPDylan Patel1:35:41

       This is described to me, not me. I don't buy cocaine. Just to be clear.

    10. SWSarah Wang1:35:45

        Okay. Yeah. Great.

    11. DPDylan Patel1:35:45

        Someone tells me this. Someone tells me this. I'm like, "Holy shit, it's right." You call up a couple of people, you text a couple of people, you ask, "Yo, how much you got? What's the price?"

    12. SWSarah Wang1:35:52

        [laughs]

    13. DPDylan Patel1:35:53

        It's like-

    14. SWSarah Wang1:35:54

        Exactly

    15. DPDylan Patel1:35:55

        ... this is fucking like buying drugs. Like-

    16. SWSarah Wang1:35:57

        [laughs]

    17. DPDylan Patel1:35:59

        Oh, sorry, sorry.

    18. SWSarah Wang1:36:00

        No, I mean, very accurate.

    19. DPDylan Patel1:36:01

        To this day, like all... I just... It's the same way. You just send like... We, we have Slack connects with like thirty Neoclouds. I'm like-

    20. SWSarah Wang1:36:06

        There you go. Thirty

    21. DPDylan Patel1:36:07

        ... as well as like some of the major ones, and we just send them a message like, "Hey, customer wants this much. You know, this is what they're looking for." And then they send quotes and then-

    22. ETErik Torenberg1:36:15

        I know this guy. [laughs]

    23. SWSarah Wang1:36:16

        I know a guy. Well, so I think that's actually a very accurate description, and I've sent countless port codes, your Cluster Max original post because I thought it did a really good job breaking them down. Um, but maybe one question to end on for me is just what era are we in now with Blackwell's coming online? Are we sort of back to the summer 2023 era, and it's-- that's kind of the, the cycle, uh, that we've just entered? Or what, what's sort of your view on where we are?

    24. DPDylan Patel1:36:42

        So, so, so for-

    25. ETErik Torenberg1:36:42

        That's a very good question

    26. DPDylan Patel1:36:43

        ... for one of your port codes, um, we, we were like, you know, after their difficulties with Amazon, we tried to... We were like, "Okay, let's, let's actually like get two GPUs. The original deals we got you were gone, but like here's some other deals," right? It turned out that, um, multiple major Neoclouds had sold out of Hopper capacity. Um, and, and their Blackwell capacity comes online in a few months. Um, so it's, it's a bit of a challenge, right? In that-

    27. SWSarah Wang1:37:10

        Due to inference?

    28. DPDylan Patel1:37:11

        Um, inference demand has been skyrocketing this year, right?

    29. SWSarah Wang1:37:14

        Right.

    30. ETErik Torenberg1:37:14

        Reasoning models, yeah.

Episode duration: 1:38:57