No Priors Ep. 27 | With Sarah Guo & Elad Gil

Hey everyone. Welcome to No Priors. I'm Sarah Gua.

I'm Elad Gal.

This week on No Priors we're back with another episode where we answer your questions about tech, AI and everything in between.

I think we have a lot of different questions that people have brought up this week that they were hoping we could cover and some topics that we thought were- would be kind of interesting.

I wanna go to one of our listener questions and I think a topic that's really popular with many of the companies that you and I work with i- in terms of access to computing for much smaller scale experiments.

What, uh, what's going on with the GPU crunch?

Yeah. The companies that you and I work with, many of them are, uh, companies that, you know, they need to use very specific infrastructure to train and serve large models, right? Um, these work on GPUs and the structure of the industry is like, um, it's just not very robust, right? So you have a very small number of producers, NVIDIA and AMB- AMD generally, and then NVIDIA is very far ahead on the high-end processors that are most efficient for large-scale training and inference. Uh, then you have the pandemic supply disruption which we haven't fully recovered for. If you actually look at the supply chain, um, you go from the actual designers to, you know, the reliance on a few major foundries like TSMC. Um, you know, expansion of this capacity is not easy, right? New fabs are billions of dollars. Yield is a very complicated thing. You can think of it as a massive precision manufacturing problem where temperature, pressure, chemical concentration, tool imperfections, new processes, materials issues, like anything can make production have lower yield or lower quality, right? And- and so like if you think about the speed with which, uh, the industry, um, driven by both large and small players has decided that they want to do AI, uh, like the physical processes cannot keep up with that demand. It's as if, you know, half the companies in the world over a year-long period decided like, "Yeah, we need super computers." Not superconductors but, uh, gigantic networked GPUs.

So like what is the actual gap? So you're- to your point it sounds like, uh, much of the AI world is dependent on GPUs in order to train and then, uh, do inference on these big AI models and the big suppliers are basically NVIDIA, AMD and then there's like a long tail of smaller folks. What is the delta between the amount of capacity that exists today and that's needed? Are we off by 2X, 10X, some other number?

It's hard to say because right now there's no way to explore like the price elasticity of these things, right? Um, so, you know, just very specifically like the industry is kind of looking at deliveries in small quantity in September, larger quantities in December, January. Most of the large cloud providers are sold out for any scale for at least through April of next year. And so you have like really interesting dynamics like large cloud players who, you know, are the biggest consumers of- of these GPUs already, uh, like a Microsoft going and buying from other providers for near-term- near-term supply, right? So I think one question that I ask you is like, "Hey do you think this is a long-term thing? Do you think it's a very short-term thing?" But I- I think it just goes back to like the- the fundamental dynamics are do you expect the demand for these chips to continue increasing at a pace that out-increases the ability to scale a very physical like real-world process, right? Just to even be more specific, one of the challenges, like I was- I was talking to Jensen about this and a bonder, like not part of the GPU itself but like a critical tool in the manufacturing and assembly of GPUs is very specialized and so the ability to build any of these tools as well to enable these processes is- is a blocker. If you look at the demand from large labs today to continue increasing model scale and training time by magnitudes, I think it's hard to see that dynamic going away. What do you think?