Alex Wang: Why Data Not Compute is the Bottleneck to Foundation Model Performance | E1164

At its core, this AI technology has the potential to be one of the greatest military assets that humanity has ever seen. Potentially even more of a military asset than nukes. Let's say China or Russia had AGI today and the United States didn't, I would imagine they would use that to conquer. The CCP system is incredibly good at taking very aggressive centralized action and centralized industrial policy to drive forward critical industries. They have a clear shot at racing forward.

Ready to go? Alex, I am thrilled that we could do this in person. Thank you so much for joining me today.

Yeah, great to be here.

Now, listen, it's funny. I told you, I tweeted before like, we should skip the founding stories because there are many, many great times you've told it before. But I want to dive straight in, and I want to ask you the question of when we look at model performance today, let's just start high level, do you think we're seeing a case of diminishing returns where more compute doesn't lead to better performance?

Yeah, I think it's pretty fascinating. I mean, I think there's been, um, this especially coming up now where GPT-4, uh, you know, OpenAI has had GPT-4 since fall of 2022. Uh, and since that timeframe, we haven't yet seen a new base model or new, a new model that's, you know, um, jaw droppingly better than GPT-4. You know, we haven't seen the GPT-4.5 or the GPT-5 or, you know, the other labs haven't yet come out with models that are, you know, leagues and leagues better than GPT-4, despite just like way, way more compute expenditure. You know, since, since when ChatGPT came out, you know, you can look at the graph of NVIDIA's revenue, it just inflects. It's just like, it just goes straight up after GPT-4 came out, and it goes from, you know, I think the NVIDIA's data center revenue, they were doing, uh, roughly about five billion a quarter, and then it shoots up to now it's, you know, north of $20 billion a quarter. So there's been, you know, tens of billions going to a hundred- more than a hundred billion of spend on, uh, high-end NVIDIA GPUs all in the same timeframe. We haven't yet seen the big breakthrough since GPT-4, which actually that model was, came out before this huge inflection in NVIDIA, um, expenditure. So, uh, I think overall it's very, it's this interesting thing where we're seeing investment into compute go up dramatically, go up exponentially right now. But we're still, I think as a community, as an industry, kind of waiting for the next great model.

Do you think we've reached this kind of asymptote of performance where actually we'll see this kind of plateauing in performance while we wait for that? And do we think that's like a monthly thing or do we think that's kind of like self-driving? Do you remember with self-driving we saw kind of the plateau in performance actually for kind of several years, and actually it was only recently where we see that inflect again.