No Priors Ep. 27 | With Sarah Guo & Elad Gil

1. 0:00 – 0:37

   Introduction

   1. SGSarah Guo0:05

      Hey everyone. Welcome to No Priors. I'm Sarah Gua.

   2. EGElad Gil0:09

      I'm Elad Gal.

   3. SGSarah Guo0:10

      This week on No Priors we're back with another episode where we answer your questions about tech, AI and everything in between.

   4. EGElad Gil0:16

      I think we have a lot of different questions that people have brought up this week that they were hoping we could cover and some topics that we thought were- would be kind of interesting.

   5. SGSarah Guo0:22

      I wanna go to one of our listener questions and I think a topic that's really popular with many of the companies that you and I work with i- in terms of access to computing for much smaller scale experiments.

   6. EGElad Gil0:35

      What, uh, what's going on with the GPU

2. 0:37 – 10:30

   Impact of GPU Bottleneck in the near and long term

   1. EGElad Gil0:37

      crunch?

   2. SGSarah Guo0:38

      Yeah. The companies that you and I work with, many of them are, uh, companies that, you know, they need to use very specific infrastructure to train and serve large models, right? Um, these work on GPUs and the structure of the industry is like, um, it's just not very robust, right? So you have a very small number of producers, NVIDIA and AMB- AMD generally, and then NVIDIA is very far ahead on the high-end processors that are most efficient for large-scale training and inference. Uh, then you have the pandemic supply disruption which we haven't fully recovered for. If you actually look at the supply chain, um, you go from the actual designers to, you know, the reliance on a few major foundries like TSMC. Um, you know, expansion of this capacity is not easy, right? New fabs are billions of dollars. Yield is a very complicated thing. You can think of it as a massive precision manufacturing problem where temperature, pressure, chemical concentration, tool imperfections, new processes, materials issues, like anything can make production have lower yield or lower quality, right? And- and so like if you think about the speed with which, uh, the industry, um, driven by both large and small players has decided that they want to do AI, uh, like the physical processes cannot keep up with that demand. It's as if, you know, half the companies in the world over a year-long period decided like, "Yeah, we need super computers." Not superconductors but, uh, gigantic networked GPUs.

   3. EGElad Gil2:23

      So like what is the actual gap? So you're- to your point it sounds like, uh, much of the AI world is dependent on GPUs in order to train and then, uh, do inference on these big AI models and the big suppliers are basically NVIDIA, AMD and then there's like a long tail of smaller folks. What is the delta between the amount of capacity that exists today and that's needed? Are we off by 2X, 10X, some other number?

   4. SGSarah Guo2:48

      It's hard to say because right now there's no way to explore like the price elasticity of these things, right? Um, so, you know, just very specifically like the industry is kind of looking at deliveries in small quantity in September, larger quantities in December, January. Most of the large cloud providers are sold out for any scale for at least through April of next year. And so you have like really interesting dynamics like large cloud players who, you know, are the biggest consumers of- of these GPUs already, uh, like a Microsoft going and buying from other providers for near-term- near-term supply, right? So I think one question that I ask you is like, "Hey do you think this is a long-term thing? Do you think it's a very short-term thing?" But I- I think it just goes back to like the- the fundamental dynamics are do you expect the demand for these chips to continue increasing at a pace that out-increases the ability to scale a very physical like real-world process, right? Just to even be more specific, one of the challenges, like I was- I was talking to Jensen about this and a bonder, like not part of the GPU itself but like a critical tool in the manufacturing and assembly of GPUs is very specialized and so the ability to build any of these tools as well to enable these processes is- is a blocker. If you look at the demand from large labs today to continue increasing model scale and training time by magnitudes, I think it's hard to see that dynamic going away. What do you think?

   5. EGElad Gil4:30

      I feel like there's a couple different sort of second order implications of the fact that we're- we're seeing this giant GPU bottleneck. I think the first one is that we're seeing new sort of models that are dependent on GPU access or ownership as ways to create all sorts of really interesting monetization and potentially eventually co- cloud services. So that's things like CoreWeave or FoundryML or other companies that are basically providing now GPUs in different ways. In some cases through aggregation or federating different, um, sources of GPUs, in some cases it's just having these large GPU clouds and being able to use them in really interesting ways, and one of the interesting, I think, um, side notes is that GPUs used to be very heavily used for crypto mining and while crypto is down, it may actually be more economic to just use them for- to rent out for AI training purposes or inference purposes, so I think that's one really interesting almost like sectoral shift in terms of existing GPU capacity. The second is that a lot of the different players that are startups who've built their own semiconductors specifically for AI training I think are starting to see a lot of really strong pull. So for example Cerebras, and I think we're going to have Andrew from Cerebras on, um, our podcast in a- in a couple weeks, they just signed a $100 million deal with UAE, uh, for building nine super computers using their chips which are optimized for AI.

   6. SGSarah Guo5:48

      Amazing.

   7. EGElad Gil5:49

      And so I think they and Groq and other sort of semiconductor providers are going to find really strong pull during this period where people are desperate for any solution and they're willing to do- take the extra steps to really be able to utilize other forms of silicon and so I think it creates a bit of an opening.... for other players in the market. And so it does seem like it's going to have these really interesting sort of cascading effects on members of the startup ecosystem and, you know, new players that are working against all this.

   8. SGSarah Guo6:16

      Two, um, sort of second order things are like what do you do when scaling is blocked on capacity? Like, you try to be more efficient. It's not been an area of massive focus to date because people have been chasing the state-of-the-art following chinchilla scaling as the, um, simplest path forward. But there are really interesting lines of research that are undervalued today unless the hardware supply crunch continues including in dynamically figuring out or routing to efficient models. Um, so think of like the FrugalGPT work or, uh, generally like distillation, uh, or even just a more intelligent choice of data for your pre-training or your fine-tuning training mix so you can, um, use less compute, right, um, for- for the same or for improved quality. And I- I think like everybody's been on this one path and an interesting second order effect is like does it spread people out into lots of different directions in terms of chasing performance. I personally don't think the, um, supply crunch goes away immediately and like a part of the dynamic is just, you know, um, how much more people want to scale and an- another part is like, uh, you know, if this stuff is actually useful then inference, like inference already dominates OpenAI compute usage, right? And so that demand will continue to go up.

   9. EGElad Gil7:41

      Yeah, I- I do think demand will only, um, rocket from here, at least in the short run, and so the real question is the degree to which the semiconductor industry adjust to that and the reality is that people really view NVIDIA's chips as the most advanced on the market right now and so that means that a lot of it is just a bottleneck in how much can NVIDIA scale up manufacturing. And there's other players like AMD, there's the startups we mentioned, uh, Cerebras, Grok and others, but a lot of the capacity is just gonna be how much can they- can NVIDIA and maybe AMD scale up in the short run at least and so that may just cause some ongoing bottlenecks, a- assuming again that we continue to see this very rapid growth in- in AI and AI applications. Um, I'm working on a blog post right now actually about this because it feels to me that we're still in the very, very early innings of this wave of AI adoption, right? It's not a continuum where we had CNNs and RNNs and now suddenly we have transformers. Transformers created a whole new capability set and we're only, you know, eight months since ChatGPT and a few months since, five months I think since, uh, GPT-4 and so the only people who've really adopted this technology yet are the AI native companies like OpenAI and Midjourney and a few other folks and then we had the first wave of startups come, the Perplexities and Harveys and Characters of the World as well as the first wave of incumbents adopting it, Notion and Zapier and sort of the very, very early founder-driven adopters. And so we've had zero real enterprise adoption in terms of real products at scale or close to zero and, you know, most enterprises, big businesses take six months, nine months just to do their planning cycles and then they'll spend a year prototyping and then finally they'll launch these AI apps and so we're probably a year or two years before we really start to see large-scale AI applications by existing incumbent enterprises in real products live everywhere. So from a ramp perspective one can imagine that a lot of the future ramp in AI is coming, uh, in about two years or, you know, one to two years, something like that. So there's still a lot of, um, room I think for the hype cycle, for increasing ongoing excitement, sometimes irrationally so, and then also for sort of adoption of semiconductors and other underlying infrastructure so there's still a lot to come it feels like.

   10. SGSarah Guo9:56

       I agree with you and I- I still think we're, uh, really early in let's say like the collective exploration of applications and constraints, right? Like you had the people who were bleeding edge of just personal interest. Uh, like I think ChatGPT was- i- is looked at correctly as the starting gun for people to begin developing these AI applications generally, uh, but if- if you think about h- how long it takes to ship actual interesting products to market and then the buildup of some collective

3. 10:30 – 11:50

   Timeline for existing incumbent enterprises to use AI in products

   1. SGSarah Guo10:30

      understanding of like how to make these models more useful in different applications and then, you know, turn them into workflows and then advance the state-of-the-art given a particular workflow if you have a hypothesis on value, like that all- that all takes time so I- I think we're in inning one.

   2. EGElad Gil10:48

      Yeah, it's all been demos so far, yeah. S- so I guess related to that a lot of the interest and excitement right now is around agents, you know, I spoke recently... There's a group, um, called the AGI House which, you know, hosts these different hackathons in the Bay Area and stuff like that and they had me come and, uh, help kick off like a agent hackathon they had and things like that. What do you think happens with the agent world? Like what form does that take and is it a handful of very broad agents? Is it highly specialized ones? Like what do you think is coming there?

   3. SGSarah Guo11:16

      Yeah. Uh, it's such a, um, like powerful broad idea that I think both will happen, right? Um, and- and so like the- the overall idea is, you know, you- you don't just talk to a chatbot or- or query an interface. You have some sort of planning mechanism that is model-driven that allows you to take asks autonomously, take actions autonomously, um, and like complete a more sophisticated task often using other tools and then return that result or report back

4. 11:50 – 19:33

   Vertical versus broad applications for AI Agents

   1. SGSarah Guo11:50

      on your work to an end user, right? And so, you know, I think that is going to range from, um, the pure consumer applications, so things like Inflection which is going to, you know, have personalize that do more for you, um, Minion which is working on like web agents, uh, and- and then, hm, you know, I- I think like there's been very recently more attention or just more understanding of how powerful it is to have agents that in some way, um, write executable code.... right? Um, because you can programmatically use many more tools. You can call APIs, and I- I think if that is, uh, do a task that is not a single query but requires multiple steps in, uh, in analytics or an enterprise automation, um, or even, you know, within, like, U- you know, companies that we work with, uh, like Harvey. Like, a single legal task is actually a composition of thoughts, planning, attempts of research, like writing that a- an associate might do. And so, uh, I think it's gonna be a pretty dominant paradigm.

   2. EGElad Gil12:58

      Yeah, it's kind of interesting because if you look at past technology waves and you ask about specialization versus sort of broadness, you know, are you building a broad-based platform that you can use for anything or a vertical application that really helps you with one or two things well, most of the things that really work are these vertical applications that help you really well. Now, some of them broaden and grow into the broad-based platform for everything, right? Even in consumer that's true. Like, Facebook started off as a college network, and in fact it started with, like, five colleges and then they added all colleges and then later they added the ability to add your work email as a way to register, and then they opened it up to everybody and then they started building the platforms on top of it and gaming and other things, right? But it kinda happened sequentially, and there's counterexamples to that. You know, Google would be a very broad-based thing from day one. It helped you discover information on the web, right? You needed a tool for that. But it feels like in the agent world a lot of the people that I hear talking about ideas have these very broad sort of abstract ideas, and so an idea would be, um, "I'm gonna build an agent that is gonna be your assistant." And you're like, "Okay, well what is it gonna help me with?" And they say, "Everything. It's gonna make you happy." And you say, "Well, I don't... I'm, you know, I'd love to be happy." But at the same time, you know, starting with a very targeted, focused initial use case tends to be the best way to build product, A, because you know who you're building it for, B, you can really nail the use case. And there's the old sort of YC-ism which I think, uh, which is really good, which is it's better to delight a small number of people than to have a, a very large number of people indifferent to your product. And so I think my, my bias for the agent world is if you're building an agent, start with something really targeted. If it's a assistant to help you, what exactly does the assistant do? Does it do background information searches on all the meetings you have that day? Does it specifically help with certain forms of scheduling? Does it help with other aspects of your day planning or synthesis of what you've done or follow-up, a- action items or whatever it may be? But choose one or two things and do them very well versus do everything, and then eventually you may build a thing that you start off that does one thing very well but then broadens into everything. But usually starting with everything means you're not really doing anything deeply or well, and so I think that's... that to me is one of the main patterns at least in terms of prior waves of technology development.

   3. SGSarah Guo15:12

      I very much feel like this is, like, a very classic tension between, um, what I consider to be, like, uh, I don't know, the, like, infrastructure platform engineering, like, even research agenda driven approach that is like, "Oh, you don't understand. Like, the technology is general. We don't want to be taken off the research path, uh, that pollutes our, uh, our data mix in a way that it is not a general purpose technology, eh, anymore," right? Um, or, you know, "It can do anything. Why limit it?" Um, or even getting feedback from users because you release the stuff, it is broadly capable, that they're doing everything with it, some things much more successfully than others, and I think more of a, like a product engineering, like, traditional, like, startup mindset that is like, "Actually complete the task," right? And I- I- I definitely think, um, uh, the overall exploration has been skewed to one side not as productively today. Um, and, and one of the, like, even if you think from the research agenda, one of the reasons it is interesting to, um, think about the, like, uh, the, you know, have more focus, everybody's thinking that, but have more focus on accomplishing the specific task is, like, you want to be happy a lot. All I want to do is, like, never write boilerplate code again, right? And so if you think about, like, um-

   4. EGElad Gil16:32

      That's how I define happiness (laughs) .

   5. SGSarah Guo16:34

      Oh, okay, great. Then we're, then we're still the same. Um, but, uh, like if you think about, like, okay, let's, like, complete one task. I- if I, uh, want to ask, um, you know, an agent to just, like, fix all the bugs in my software, um, then my, uh, ability to, like, successfully complete that task includes a lot of, like, bug-fixing-specific techniques, right? Like, you could do test time search and then see if all of the different things that you, uh, generated actually execute as one very simplistic example, right? And, and so, like, I- I think there are a lot of ways to advance in, um, the research in, uh, very specific tasks that are much more tractable. But maybe I'm not thinking big enough.

   6. EGElad Gil17:19

      No, it makes sense. I think, um, I would add one third piece to that framework you have which is the research-driven versus product-driven. I think there's a third approach which is infrastructure or tooling-driven, and that's where you're like, "I'm not gonna build the agents, but I'm gonna build the infrastructure that allows anybody else to build them rapidly." Now sometimes those types of businesses or approaches work really well and sometimes those things are solely an outgrowth of a vertical product that works really well that you then open up the infrastructure for everybody else to use, and it's very case tech- by case dependent. It's the difference between Stripe where it's just like we need to build payments for everybody, everybody keeps building it over and over again, and the Facebook auth platform which only existed because you got to hundreds of millions of users so you could open up auth as, like, a third-party service. And so I think as people think through that third angle of building infrastructure for others, they need to understand whether that infrastructure will be an outgrowth of an existing product area and benefit from the characteristics of the li- the market liquidity of that product or whether it's just a piece of the infrastructure everybody keeps building over and over and therefore it's a really good thing to just provide to the world. So I think it's kind of an interesting future topic.

   7. SGSarah Guo18:24

      We are on a, you know, couple month bull run at this point, 2024 tech markets, what's coming, like, will people be able to fundraise, will funds be able to fundraise, are customers purchasing?

   8. EGElad Gil18:37

      You know, I think there's going to be basically, um, four markets next year in some sense. One market is just AI, and I think AI will continue to run in different ways and it'll look very expensive at the time and a handful of companies will look really cheap in hindsight, just like with every other technology wave. And I think that's separable from the rest of tech that existed prior to the AI wave. Uh, for companies that fundraised in 2021, prior to being, like, AI companies, a subset of them, I think if I were to sort of divvy up that pie of those companies, sort of mid to late stage private tech companies not in AI and what's going to happen to them next year and in 2025, I think a third of them are just going to go under. Or a third of, I should say, unicorns, will eventually just go under, be fire sales, whatever. Um, they won't be able to ever raise money again. A third will be at the highest valuation they'll ever be at ever in the lifetime of the company. They'll reach their terminal value. And there's examples from 2014 of companies that

5. 19:33 – 23:56

   2024 tech market predictions & how founders should think about valuations

   1. EGElad Gil19:33

      went through that same wave. They, you know, raised in 2014, they went public a few years later and then they never surpassed their, their market cap again. And then I think lastly there will be a third of companies that grow past it. And so I do think there's going to be a lot of carnage next year and a lot of companies going under, and as those companies go under three things will happen. Number one, it'll be much easier to hire people. And then people are already seeing that at startups. It's easier to hire again. Second, it should have follow-on effects and ramifications for commercial real estate, and we'll see a second shoe drop there. And then third, the venture capital community will be impacted because a lot of the things that they've been using to fundraise new funds or do other things with will suddenly go to zero. Their big unicorn success will go from a multi-billion dollar or billion-dollar company to basically a company that isn't worth anything. And so I think that's gonna have knock-on effects to the venture ecosystem. But I, I think they'll, that'll take, like, two, three years to play out because all these things are a bit time delayed. Um, but yeah, I think that other shoe still hasn't dropped in private tech markets, and a lot of it is just companies raised so much money in 2021 they still have lots of money, so everything still feels like it's continuing to go. But at some point that money's gonna run out, so I think it's going to be a pretty bumpy 2024 and 2025 potentially.

   2. SGSarah Guo20:49

      Yeah. My advice to companies that, you know, raised at very healthy valuations during that period of time and then are actually building businesses is to try to completely disassociate from that valuation, um, because people will put themselves into all sorts of contortions to do a flat or up round to a valuation that makes no sense, right?

   3. EGElad Gil21:14

      Yeah.

   4. SGSarah Guo21:14

      And if you don't have the historical context of that making no sense, it's an extremely painful, um, ex- uh, sort of realization to have. But if you look at... There's this, um, one analysis of, uh, actually the very best technology companies and the ones that endured from the internet bubble and how long it took those companies to reach the valuations they were at before the bubble burst, and it's a decade, right? And it's like, startups don't have a decade to try to, you know, get to at par valuations.

   5. EGElad Gil21:48

      Yeah. I'm actually less worried about valuation. I think valuation is ephemeral, right? Effectively every, or roughly every tech company in public market's hit a down round over the last year and a half, right? They all lost, or many, many companies lost 30 to 90% of their value, right? And effectively they just did a down round in public markets because every day you're repricing a public stock. I'm more worried about the people who burn tons of cash and they don't have a lot of revenue to show for it, and then when they're going to go out to raise more money, people say, "Well, you burnt $50 million, you burned $100 million to generate $5 or $10 million of revenue." And so the issue isn't that your valuation's off, we can always reset valuation, it's the fact that you burned all this money and you don't have anything much to show for it, right? And that's where I think the real issues will happen because you can always reprice things and people will be forced to and, you know, it'll just happen, but I think it's the underlying business case and business model that's going to be the real issue.

   6. SGSarah Guo22:40

      Yeah. I, I guess, like, the, the unforced error there for companies who actually have the time to make the decision is, um... The thing you want to avoid is, like, not adjusting your cost profile or, you know, holding onto that valuation until it's too late.

   7. EGElad Gil22:54

      Yeah. Or just deciding it's the wrong business and it's not working and, you know, the most important precious thing for you as a founder is your time, and I think people forget that. You have this golden period in your life where you don't have hopefully a lot of other complications in terms of sick family members or school-related issues or whatever it is, and you can take risk and you have a low-cost basis and you can do all these things, and that's the moment when you can best take risk to start a company for many people, not for all. And, you know, you're really giving up the best years of your life working on things that potentially may not work. Thanks for the discussion. It's a lot of fun.

   8. SGSarah Guo23:34

      Yeah. Super fun. Thanks to everyone who sent us your questions. Find us on Twitter @nopriorspod. Subscribe to our YouTube channel if you want to see our faces, follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.

Episode duration: 23:57