By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
- 0:00 – 3:03
Series Overview and the AI Demand Surge
Steph Smith frames part two of the AI hardware miniseries for founders and introduces Guido Appenzeller and his infrastructure background. Guido says the exponential growth of AI since mid last year has created demand the market cannot meet.
- •Part two covers supply versus demand, renting versus owning, moats, and open source
- •Part three will cover the cost of training and inference
- •Guido was CTO for Intel's data center group
- •AI demand has grown faster than anyone expected
- 3:03 – 5:18
Why Chip Supply Can't Simply Scale Up
Guido describes the gap between demand and supply and the bottlenecks in chip manufacturing and card building. He explains that foundries such as TSMC are capacity constrained and that building new fabs takes years and billions of dollars.
- •Some reputable sources say demand outstrips supply by a factor of 10
- •Portfolio companies struggle to find compute capacity
- •TSMC capacity must often be reserved long in advance and varies by process
- •A new fab takes a couple of years and billions of dollars
- 5:18 – 6:28
Who Gets Access to the Available Compute
Steph asks how a founder gets compute and who decides. Guido explains that capacity is expensive, that newer chips at scale usually require pre-reserving, and that cloud providers often ask for long exclusive commitments or strike investment-linked deals.
- •Guido could not reserve an instance with a cloud provider for a personal experiment
- •Large-scale access to newer chips often requires pre-reserved capacity
- •Providers may ask for two-year commitments
- •Investment deals can come with capacity, as in the OpenAI example
- 6:28 – 9:00
Choosing Hardware and Shopping Around
Guido advises first asking whether you need to consume hardware directly or can use a hosted service like Replicate. For those running their own models, he recommends shopping around, noting specialized clouds are often preferred by startups. He then explains how memory, server size, networking, and training versus inference drive the choice.
- •Hosted services can manage GPU provisioning for you
- •Large clouds are not always the best option when priced out
- •Startups often choose specialized clouds like CoreWeave or Lambda
- •Memory needs separate small models from large language models
- •Networking fabric can constrain very large model training
- 9:00 – 10:48
Matching Reservations to Your Workload
Steph raises the cloud-versus-in-house calculus, and Guido says the right fit depends on scale and usage pattern. Training and always-on inference tend to need long-term reservations, while batch jobs can use variable capacity.
- •Large needs require pre-reserving
- •Training typically favors long-term GPU reservations
- •Inference that can never be down needs reserved capacity
- •Batch jobs that can wait can use ad hoc capacity
- 10:48 – 12:06
Renting Versus Owning Infrastructure
Guido argues that owning infrastructure carries costs like hiring staff and CapEx, so most founders, even later-stage ones, are better off renting. He lists exceptions such as specialized hardware needs and sensitive data, and suggests scale becomes relevant around $100 million a year.
- •Owning requires people to run it and upfront CapEx
- •Most early, mid, and late stage founders are better off renting
- •Exceptions include specialized hardware needs and geopolitical or data sensitivity concerns
- •$10 million a year is probably still under the threshold, while $100 million may justify a look
- 12:06 – 13:26
Moats and Differentiated Data
Steph asks where the moat is if everyone competes for the same compute, suggesting differentiated data. Guido says it can be a moat in areas with little public data, but larger models trained on more data also gain reasoning ability, so he expects training on everything and then fine-tuning on private data.
- •Differentiated data can be a moat where public data is scarce, such as possibly finance
- •Larger models trained on more data also reason better
- •Expected future: train on all available data, then fine-tune on private data
- •Guido's analogy is elementary school followed by vocational training
- 13:26 – 16:48
Open Source Models and How They're Built
Steph asks who can realistically compete, citing Vicuna's $300 fine-tune of Llama 1. Guido says larger models still perform better, but over-training smaller models and instruction fine-tuning are closing the gap, though no large open source LLM exists in GPT-3's weight class.
- •Small open source models are not yet at GPT-3.5 or GPT-4 level
- •Over-training smaller models can match larger ones, per the Chinchilla scaling law discussion
- •Base models complete text, then instruction fine-tuning makes them useful for chat
- •The biggest open source gap is the lack of a large model near 175 billion parameters
- 16:48 – 17:49
Update on Open Models and Parameter Counts
Steph notes that since the recording, Llama 2 and Falcon have been released. She explains that they are still smaller than closed models, but parameter count isn't the only driver of performance, as Llama 2 is comparable to GPT-3.5 due to more training data.
- •Llama 2 has 70 billion parameters and an open license
- •Falcon has 40 billion parameters
- •GPT-3 has 175 billion parameters and GPT-4 an estimated 1.8 trillion
- •Llama 2 outperforms GPT-3 through more training data
- 17:49 – 20:16
Running Models Locally on Devices
Steph asks whether inference will move onto devices as models compress and compute becomes more decentralized. Guido expects so, citing Stable Diffusion on phones, and says local inference removes server costs while the cloud will still do higher-quality, heavier tasks.
- •Smaller models already run on laptops and even phones
- •Faster devices and optimized models will drive the trend
- •Local inference costs only battery power and some heat
- •Simple tasks like spellchecking can be local, while complex ones like speeches or summaries need the cloud
- 20:16 – 22:40
Rebuilding the Stack and What's Next
Guido describes AI as a second type of compute, in which neural networks figure out solutions, requiring a new stack with things like vector databases and model hosting. Steph closes by previewing part three on the cost of training and inference.
- •AI is better seen as a different type of compute than just a new application
- •Neural networks can solve problems without engineers knowing how
- •New stack pieces include vector DBs and model hosting providers
- •Part three will cover startup AI spend and training versus inference costs
