By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
EVERY SPOKEN WORD
25 min read · 4,672 words- 0:00 – 3:03
Supply and demand
- GAGuido Appenzeller
Finding the compute capacity to run the applications is actually a real challenge
- SSSteph Smith
What really is stopping companies from going and, like, ten-xing their production?
- GAGuido Appenzeller
The crazy exponential growth of AI at the moment
- SSSteph Smith
How do I get access to the compute that I need?
- GAGuido Appenzeller
I think my number one advice would be to shop around. For a certain process which you don't wanna use, there's capacity, but for another one that you do want to use, they don't have the capacity
- SSSteph Smith
How much should founders know about hardware?
- GAGuido Appenzeller
And there's probably a certain scale where it makes sense for you. It's a whole, like, a, a new ecosystem forming, and I think there's a ton of opportunities to, to build great companies.
- SSSteph Smith
With software becoming more important than ever, hardware is following suit, and with the world constantly generating more data, unlocking the full potential of AI means a constant need for faster and more resilient hardware. That is exactly why we've created this miniseries on AI hardware. In part one, we took you through the emerging architecture powering LLMs from GPU to TPU, including how they work, who's creating them, and also whether we can expect Moore's law to continue. But part two is for the founders trying to build AI companies, and here we dive into the delta between supply and demand, why we can't just print our way out of a shortage, how founders can get access to inventory, whether they should think about renting or owning, where moats can be found, and even where open source comes into play. You should also look out for part three coming very soon, where we break down exactly how much all of this costs from training to inference. And today we're joined again by a16z special advisor, Guido Appenzeller, someone who is truly uniquely suited for this deep dive as a storied infrastructure expert with experience like...
- GAGuido Appenzeller
CTO for Intel's data center group dealing a lot with hardware, the low-level components. So it's given me sort of, I think, a good insight how large data centers work, you know, what the, the, the basic components are that make all of this I- AI boom, uh, you know, possible today.
- SSSteph Smith
Despite working with infrastructure for quite some time, here's Guido commenting on how the momentum of the recent AI wave is shifting supply and demand dynamics
- GAGuido Appenzeller
The biggest thing that is triggering that is just the, the, the crazy exponential growth of AI at the moment. I mean, AI has been booming since mid last year. I think nobody expected how quickly it would move, and that has just, uh, created a demand which at the moment the market can't fulfill.
- SSSteph Smith
As a reminder, the content here is for informational purposes only. It should not be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. Please note that a16z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com/disclosures. [upbeat music]
- 3:03 – 4:46
Competition for AI hardware
- SSSteph Smith
In a recent article, Guido even stated that some reputable sources indicate that demand for AI hardware outstrips supply by a factor of 10. Here's him commenting on how that dynamic is impacting competition
- GAGuido Appenzeller
We currently don't have, uh, as many AI chips or servers, uh, as we'd like to have. So for some of, of our portfolio companies, you know, finding the compute capacity that they need to run their applications is actually a real challenge, right? There's a whole value chain behind that. It's a combination of many things. You know, we have some bottlenecks on the chip manufacturing side. We have some bottlenecks on, on building the actual cards. Uh, you know, these development cycles take some time.
- SSSteph Smith
Maybe this is a silly question, but what really is stopping companies like Intel, like N- NVIDIA, from going and, like, ten-xing their production? Like, is that on the roadmap where we're just gonna see a lot more chips, and we won't see this discrepancy between supply and demand, or is there something more complex at play?
- GAGuido Appenzeller
It's, it's a bit more complex because if you want to make a chip, right, the way you do it is you make it in a, in a foundry, right, which are like, you know, extremely, extremely large, extremely complex. Intel build, uh, makes chips on their own foundries, but, but most companies manufacture with Taiwan, Taiwan Semiconductor, TSMC, right? And they are capacity constrained, right? You often have to reserve capacity long in advance. There's different processes, so you know, it might be for a certain process which you don't wanna use, there's capacity, but for another one that you do want to use, they don't have the capacity. And you know, you could just say like, "Well, in that case, let's just build more fabs." But building a fab is, you know, takes you a couple of years and, you know, probably, uh, a couple of billion or, or, or 10 billion of investment. So you're looking at some very large investment projects that take some time to adjust, right? And that's sort of what's, w- what prevents us from reacting more quickly at
- 4:46 – 6:28
Who gets access to the supply available
- GAGuido Appenzeller
the moment.
- SSSteph Smith
And while some countries are indeed making major multi-billion dollar investments in new semiconductor production plants, AKA fabs, these will take time to scale, and there are also no promises given that expertise is concentrated in a few companies. So with demand not subsiding, what does this mean for who gets access to the supply available? I mean, it doesn't sound like the demand is gonna subside, especially because we see this really, what seems like intrinsic relationship between the power of these models and then the compute that's thrown at them. So how does a company, let's say if I'm a founder today, how do I get access to the compute that I need, and who decides? Is it just who's willing to pay the most, or how is that supply being distributed?
- GAGuido Appenzeller
Yeah, there, there, there's some of that, right? At the moment, uh, capacity is expensive wherever you go. You know, I, I tried just to run some, some personal experiments, you know, tried to, to reserve an instance with one of the cloud service providers a few days ago, and they just didn't have any. [laughs] It was like, "No, not available." What we are seeing is that often in order to get access to the, the newer cards, the newer chips, if you want it at scale, you have to pre-reserve capacity. So often these are negotiations between a company and a large cloud where you say, "Okay, I need this many, this many chips for this amount of time." What they'll often ask for is they ask for a, a certain time commitment. So they'll be like, "Okay, we can give you this many chips, but we want you to sign, uh, basically that you get them exclusively for two years, and you pay for, um, uh, for that amount," right? But we've seen, um, I mean, I think OpenAI was in the news with that, right, where you, you have investment deals where, for example, a cloud provider comes in and invests in a company. As a result, the company gets capacity. So we're seeing all kinds of deals being struck. As with any, any scarce, uh, scarce resource, right, there's a lot of, uh, deal-making going on
- 6:28 – 8:59
How to select which hardware to use
- SSSteph Smith
And it's not just a matter of getting access to compute, it's about ensuring you get access to the compute tailored to your needs, and cost is not the only factor. What would you say in terms of the considerations that they should be keeping in mind? Really, how much should founders know about hardware and, again, selecting which hardware to use?
- GAGuido Appenzeller
Fantastic question. I mean, the-- I think the, the first question, honestly, I would ask is, do you really need to consume the hardware directly, or do you really just want to consume something that runs on top of the hardware, right? Today, um, let's take an example. If I want to generate images with stable diffusion for my, for example, for my mobile phone app or something like that, um, it might be easier to go to a SaaS company like Replicate, for example, that essentially will host the model for you, where you just pay for, for access to the model, and they send you, like, the, the, the generated images. And they will manage all the provision of compute infrastructure and, and will find the GPUs for you. If you do n- want to run your, your own model, I think my number one advice would be to shop around, right? There's a, there's a fair number of providers. The large clouds, in my experience, are not always the best option, right, if, if you price it out. Uh, you know, we've seen that the, uh, startups typically are more likely to go with specialized clouds, you know, like CoWeave or Lambda, right? That, that obviously specialize in providing AI, uh, infrastructure to startups. Shop around, look at the different offers, compare prices.
- SSSteph Smith
Yeah. And when you're shopping around, in addition to price, which I feel like is a major motivating factor, what other factors are there in terms of these other companies who maybe aren't the big clouds? How are they differentiating relative to one another? How are they standing out in that market?
- GAGuido Appenzeller
Yeah. There, there's a whole sort of decision tree there. You know, I mean, the first thing is, uh, one thing that often drives the decision is how much memory do I need in my cards, right? If I have a small image model, right, uh, I, I might be able to, to work with a, a more consumer-grade card, which is much cheaper, right, per hour if I reserve it in a cloud. Versus if I, for example, train a large language model and not only need a card with the, the most memory I can find, but I probably want to have as many cards as possible in one server because communication between them matters, and I may even care about the networking fabric behind it, right? Some of the very large models, you actually network constraints in, in terms of how quickly you can, can train them. So it really becomes a question of what's your objective? Is it inference? Is it training? If it's training, how big is your model, right? And based on that, you figure out what the card is, what kind of server you need, what kind of fabric you need between those servers, and then you sort of can, can decide what the right fit is
- 8:59 – 13:26
Cloud versus bringing infrastructure in house
- GAGuido Appenzeller
for your application.
- SSSteph Smith
Even prior to this AI wave, compute was a major line item for many software companies, and the calculus of leaning on the easily accessible cloud versus bringing infrastructure in-house was becoming an increasingly important consideration. Here is Guido touching further on that very calculus in today's era and where scale comes into play. Compute is expensive. It's a major line item for many companies, and this is even before the AI revolution-
- GAGuido Appenzeller
Yeah
- SSSteph Smith
... you could say.
- GAGuido Appenzeller
Even more so today, yeah.
- SSSteph Smith
Yeah. So do you think-- How do you think about, again, like, how that impacts different companies' bottom lines and whether they really factor that in to having their own, I guess, like, allocated GPUs versus using something more like Replicate?
- GAGuido Appenzeller
You really have to figure out, um, you know, what is the right fit for you, and it, it probably depends a lot on the scale at which you need them, right? If you need a lot, you-- frankly, you have to pre-reserve them, right? [laughs] You have to have your own. There's just no, no way around that. If you need a smaller quantity, you may be able, um, to reserve them on a more short-term, uh, basis, or, you know, you have, um, various models where you can consume only while your application runs, but at a higher price, right? And so this really comes down to what kind of load do you have, right? What we're typically seeing if somebody is training, they're more likely to, to do a long-term reservation for a GPU because you wanna make sure you have access to it. If somebody has, uh, basically more continuous workloads where availability is important, like if, if I just do inference, but, you know, I wanna make 100% sure that if a request comes in, I can service it, I can never be down, right? I probably need to reserve capacity as well, right? On the other hand, if I have more batch jobs where it's like, "Well, if this job runs an hour later, that's, that's not the end of the world," then you probably can go with variable capacity and, um, you know, just reserve it ad hoc. But it's, it's really a conversation of what is your usage pattern, what is your demand pattern, and from that comes, you know, the, the best pick for, for your, the partners that you work with.
- SSSteph Smith
We've seen that companies, even prior to AI, have benefited from building their own infrastructure by basically bringing that in-house because before that they were renting, and they were paying a lot to rent that compute. Do you think that will be a differentiator for companies moving forward, or how should founders be thinking about that relationship between owning the infrastructure and renting it?
- GAGuido Appenzeller
Owning the infrastructure comes with cost as well, right? Because you need to, you know, now hire people that run it, right? You need to, uh, uh, get money for the CapEx and so on. So my guess is that most early stage founders, and, and probably even most mid stage and late stage founders, uh, you know, are better off by, by renting capacity, renting a cloud or, or using-- consuming a SaaS service, right? Um, there's a couple of exceptions. If, if you have really, really specialized needs, right, you may just not, not find anybody who has exactly the kind of hardware that you need, right? There might be some cases where you have geopolitical concerns, you know, your, your data is just too sensitive, uh, you need, you need to run your own data center. Um, and there's probably a certain scale where it makes sense for you to run your own data center, but it's a pretty large scale, right? If you're spending ten million dollars a year, you're probably still under critical, right? If you're spending $100 million a year on, on infrastructure, that, that may be a reason to look into, uh, into options for your own data center.
- SSSteph Smith
But if everyone is competing for the same compute, are there other ways to stand out? Where's the moat here? Or you could say a moat is getting access to different training data, but that actually doesn't necessarily have to do with compute or money being thrown at the problem. It's getting access to differentiated data.
- GAGuido Appenzeller
If you have access to differentiated da- differentiated data, that could be a moat. I mean, there's- It's, it's a bit more subtle, um, because look, if, if you had an area where there's just not much public training data, that's probably right, right? You know, and, and there might be areas like in finance or so where that's the case. Um, but for, for example, for a large language model, it turns out that just making a larger model and training it on more data has more benefits than just absorbing more knowledge. It also means that it's better in reasoning and, and, you know, understanding abstract context and then, you know, answering really complex multi-stage questions and so on. So, so probably, you know, if I have to guess, I think the future will be that we'll, we'll still train on all the data we can find, right? [laughs] And, and then maybe you fine-tune, meaning you, you sort of do some additional training on a particular problem domain with your, with your private data, if that makes sense, right? So you, you first go to elementary school to learn reading and writing and, you know, then you go to your, your, uh, you know, vocational training for the, for the specializ- specialized job that, uh, you know, you, you're going
- 13:26 – 16:48
What role does open source play?
- GAGuido Appenzeller
to do in the future.
- SSSteph Smith
Another important question worth addressing is who can realistically compete? If compute is expensive, will all the largest, most heavily capitalized companies win since they can build the largest models with the most data, or what role does open source play? As one of many emerging examples, Vicuna was created by fine-tuning Meta's Llama 1 model for chat. The cost of fine-tuning added only an additional three hundred dollars, but the result is competitive with much larger models like ChatGPT or Bard. So what might this example and a growing number of open source projects tell us about the future of open LLMs?
- GAGuido Appenzeller
So first of all, the, the, in general, larger models, if they're, you know, everything else being equal, perform better, right? So the, the really small open source models that we're seeing out there today, they're not yet at a level of a GPT 3.5 or GPT-4, and, um, there's actually a, a website that runs sort of regular bake-offs where they basically ask users to compare answers and, you know, it, it seems to be pretty clear that the, the, the large ones are still a little bit ahead. That said, w- we're making big advances there, and we're figuring out a couple of things. So one thing we've learned is, um, there's something called, uh, the Chinchilla scaling law that basically give us an idea how does data, uh, correspond to model size. And if we over-train, so don't train as, as efficiently as, as we could, we can actually get potentially a smaller and better model, right? So you can, can match the performance of a large model with a smaller model if you train it more, right? So that's interesting. That, that reduces model sizes, and the trend at the moment is actually to make slightly smaller model, models, uh, be- and, and train them more to get, to get equal performance, right? Um, the, the second thing is that, um, when we talk about models, there's models for slightly different purposes, right? You have the base large language models. You know, all they're trained in, practically speaking, is completing text, right? Literally, how, how you train them is you give them text and say, "Guess the next letter," and then you tell them, "Nope, that was wrong," or "Yes, that was right," right, and, and basically by, by propagate the, um, you know, how, how they predict. Um, and they're really good at that, completing text, right? That's not quite the same that you want from a chatbot or from, you know, a model that you can tell to do something. So there's usually another step afterwards, which is called, uh, you know, fine-tuning for instruction following or for, for, for chat specifically, right? Where basically I tell a model, "Look, if somebody, somebody asks you to, uh, you know, come up with a list of, of to-do item- like a list of steps how to make pizza, right, this is roughly what I expect you to, to answer," right? These models are very good in learning these things. So, so basically you first train them just complete text, and then you train them how to, uh, you know, react to, to human requests and, and instructions, right? It's called instruction fine-tuning. And so, so, uh, you know, Llama, for example, that was a Facebook model where they published the weights for, for researchers, and then some people took that and they fine-tuned it, meaning they, you know, they, they took a bunch of instruction following things to turn it into, uh, Paco or Vicuna, which is a much, much nicer model in terms of how it interacting with it, right? For a human, it's much, much more useful, right? And so the, the biggest challenge at the moment we have on the open source side is there's currently no large open source LLM out there, right? The, the, you know, the, the, uh, uh, GPT-3, uh, was 175 billion parameters. There's currently nothing in that weight class, uh, that's, that's open source and that people could use to fine-tune or to, to, to play with
- 16:48 – 20:16
Cheaper and decentralized compute
- GAGuido Appenzeller
or to modify.
- SSSteph Smith
It's worth noting that since this recording, several more open source models have been released, including Llama 2 with 70 billion parameters and an open license, unlike its predecessor, Llama 1. Another 40 billion parameter open source model, Falcon, was released as well. But both of these are still dwarfed in parameters compared to closed models like OpenAI's GPT-3 at 175 billion parameters or GPT-4 at an estimated 1.8 trillion parameters, although the latter is speculated to be a collection of multiple smaller models. However, parameter count isn't the only driver of performance. For example, while Llama 2 has fewer parameters than GPT-3, its performance is actually much better due to being trained on more data. In fact, Llama 2 is currently comparable to GPT-3's successor, GPT-3 3.5, the current default for ChatGPT. And as these models continue to get larger, we may actually see some models compress, becoming more efficient and enabling inference on your device. Stable diffusion can, you know, run on your computer's GPU. Do we expect to see more of that? Because right now they are all hosted by these companies, right? They're trained by these companies on their dedicated servers, and then even, you know, if you interface with ChatGPT, it's running that inference for you. Do we expect to see that change at all as compute becomes cheaper, maybe more decentralized, or how would you think about that?
- GAGuido Appenzeller
That's a really good question, and, uh, we're, we're speculating a little bit here, but my, my guess is we will, right? And the, we- we're seeing some of these smaller models getting pretty good. They run on, you know, your laptop or, or even your phone. Uh, you know, we're starting to see stable diffusion implementations that run well on phones, right? Which I would have never thought, right? And they, they take couple of 10 seconds to create an image, uh, you know, which is comparatively slow, but, uh, you know, there's, there's certain applications where that's, where that's access- a- acceptable. So my guess is as- both the devices get faster and the models get more optimized, right? We'll, we'll, we'll... This will be a trend that we see more and more. And, you know, in the future it might just be part of the operating system to have a basic, uh, large language model or basic image generation model.
- SSSteph Smith
We've talked about how expensive compute can be and how ultimately that can be a major line item for companies. And I guess probably the model training will remain, you know, with those companies and not necessarily on folks' devices. But in terms of the inference, I assume that's still a, you know, pretty significant cost. And in a way, if, if someone is able to run that locally, doesn't that disjoint the company from having to pay for that compute because it's running on, let's say, you know, someone's MacBook GPU?
- GAGuido Appenzeller
Oh, yeah, totally. I mean, like, look, if I can generate an image on my phone directly, you know, all it takes is, is, is some battery power and it gets, it gets a little warm, right? And that's, and that's it, right? So that's a huge advantage. Um, you know, at the same time, there's probably gonna be a little bit bifurcation there around quality and, and parameters, right? You can run things locally, uh, but you can probably run them a lot better in the cloud, right? Because you have a, have a much bigger server there. So it probably depends a little what you want to do, right? If I just wanna have a better spellchecker, uh, you know, that, that checks my email or maybe, you know, just some, some simple, uh, completion, that's perfectly fine. I can run that on my phone. On the other hand, if I want something, you know, that is more, you know, write a good speech or, you know, summarize a, a, a complex text, they might be like, "Oh, that I gotta run in the cloud because it takes so many more operations."
- 20:16 – 21:31
Rebuilding the stack
- SSSteph Smith
Hopefully this is getting your wheels spinning in terms of what can be built, and here is Guido speaking to how this presents a fundamentally new stack and what that means in terms of opportunity. It feels like this really is, like, this massive wave, this renaissance of, of innovation.
- GAGuido Appenzeller
It's full of opportunities, right? I mean, we, we, we're rebuilding a stack. You know, it's, it's... You can look at AI just as a new application, but honestly, I think it's, it's probably a better way to look at it as, as a different type of compute, right? Uh, we, we, we traditionally build software by composing algorithms in a way that we understand well, and where, you know, the end result was, uh, you know, a program that was from bottoms up constructed. Now we have a second type compute, uh, you know, where we just train a large neural network. And the big advantage is we don't actually need to know how to solve a problem as long as the network can figure it out, right? The neural network can figure it out, we're fine. And that opens up a, uh, a bunch of new applications, but it also means, you know, you need a, a completely different stack in terms of all the different pieces, right? You probably want vector DBs to retrieve, uh, context. You know, you, you want different types of hosting, uh, providers that are, that are good in, in hosting these models and providing them to you as a service, right? It's, it's a whole, like, you know, Cambrian explosion of creativity. There's a whole echo... uh, uh, new ecosystem forming, and I think there's a ton of opportunities, too, to build great
- 21:31 – 22:39
Upcoming episodes on cost of compute
- GAGuido Appenzeller
companies.
- SSSteph Smith
I think that paints a pretty incredible picture of opportunity across the stack. And as many of these trends continue to progress, like supply and demand, the calculus of renting versus owning compute, closed versus open source models, we look to part three of the series to answer a very important question: How much does all of this cost? We'll explore all this in depth, including how much startups are really spending on AI compute and whether that's sustainable, how much it really costs to train a model like GPT-3, the difference in cost between training and inference, and how all of this will change with time. We will see you there. Thank you so much for listening to the a16z podcast. What we're trying to do here is provide an informed, clear-eyed, but also optimistic take on technology and its future, and we're trying to do that by featuring some of the most inspiring people and the things that they're building. So if that is interesting to you and you'd like to join us on this journey, go ahead and click subscribe and make sure to let us know in the comments below what you'd like to see us cover next. Thank you so much for listening, and we'll see you next time.
Episode duration: 22:40
In the uListen app, the transcript lights up word by word as you listen. Get uListen
Transcript of episode IPre5287P3I
