The 10 Trillion Parameter AI Model With 300 IQ

If O1 is this magical, what does it actually mean for founders and builders? One argument is it's bad for builders because maybe O1 is just so powerful that OpenAI will just capture all the value.

You mean they're going to capture a light cone of all future value?

Yeah. (laughs) Yeah. They'll capture a light cone of all present, past and future value. (laughs)

Oh my god.

The alternative, more optimistic scenario is we see ourselves how much time the founders spend, especially during the batch, on getting prompts to work correctly, getting the outputs to be accurate. But if it becomes more deterministic and accurate, then they can just spend their time on bread-and-butter software things. The winners will just be whoever builds the best, like user experience, who gets all these like nitty-gritty details correct.

(instrumental music)

Welcome back to another episode of The Light Cone. We are sort of in this moment where OpenAI has raised the largest venture round ever, $6.6 billion with a B. Here's what Sarah Friar, the CFO of OpenAI said about how they're going to use the money.

It's compute first, and it's not cheap. Uh, it's great talent second. Um, and then, of, uh, course, it's all the normal operating expenses of a more traditional company. But I think there is no denying that you are, we are on a, a scaling law right now where orders of magnitude matter. The next model is going to be an order of magnitude bigger and the next one on and on. And so that does make it very capital intensive.

So it's really about orders of magnitude. Let's live in the future. There's 10 trillion parameters out there, 10 trillion parameter large language models, two orders of magnitude out from the state-of-the-art today. What happens? Like are people actually going to be throwing queries and actually using these 10 trillion parameter models? Uh, seems like you'd be waiting, you know, 10 minutes per token.

Yeah. For a bit of context, right now, the frontier models, I mean they're not public exactly how many para- parameters they have, but they're roughly in the five hundreds of billions-ish, like Llama-3, 4.0, 5 billion. Anthropic is speculated to be 500 billion, GPT-4.0 roughly a- around that much. Getting to 10 trillion, there's a two order of magnitude, right? I think the type of level of, uh, potential innovation could be the same leap we saw from GPT-2, which was a- around one billion parameters that was released with the paper of a scaling laws, which was one of the seminal papers that people figured out, okay, this is transformer architecture that we figured out, what if we just throw a bunch of engineering and just do a lot of it? Where does this scale on this logar- logarithmic type of scaling? Then this was proofed out when GPT-3.5 or 3 got released, that was about 170-ish billion parameters. So that's like that two order of magnitude. And we saw what happened with that, that created this new flourishing era of AI companies. And we saw it, we experienced this back in 2023 when we started seeing all these companies building on top of GPT-3.5 that was starting to work and it created this giant wealth. So we could probably expect if this scaling law continues, the feeling will be similar to what we felt from that year of, uh, transition from 2022 to 2023.