No Priors Ep. 14 | With Sarah Guo and Elad Gil

Hey, everyone. Welcome to No Priors. Today, we're gonna switch things up a bit and just hang out and answer listener questions about tech and AI.

The topics people wanted us to talk about include everything from the evolution of open source models to the Balkanization of AI, Elon AI, which I think will be super interesting to cover, regulating AI, and AI hype in the investing environment. Let's start with the march of progress for, uh, open source models. Um, I guess, Sara, what have you been paying attention to, and what are some of the more interesting things that you view happening right now?

Yeah. So there's nothing out there today in open source that is like GPT-4 3.5 or Anthropic Claude quality, right? So there is a, there's one player out in front, and that's OpenAI, but I think the landscape has changed a lot over the last couple of months. Like, Facebook LLaMA's quite good. Um, many startups are just using it despite its licensing issues, assuming Mark won't come after them, and then you have a number of other releases that have happened, right? Uh, Trmro just released a pre-training dataset, which seems quite good. Stability just refused, um, released DiffusionXL in the image gen space. Um, and so I, I think the, like, larger dynamic is that there's been an increasing number of people and teams that now know how to train large models. The cost of a flop is only gonna go down. Um, there's a lot of investment in, like, distilling models, and, uh, a lot of researchers would claim that you and I know that it's gonna be 5X cheaper to train the same size model the second time around, like, once you've made your mistakes and know what you're doing. And then you have these other accelerants, like you can use these models to annotate your datasets and increasingly do advanced, like, self-supervision. So if VCs are going to continue to fund foundation model efforts, including open source, um, foundation model efforts, like... If I were a betting woman, and I am, I'd bet there's a 3.5 level model in the open source ecosystem within a year, um, and I, I didn't personally believe that would be true, like, a few months ago.

I guess that puts it about two to three years behind when GPT-3.5 came out, though. And so do you think that's gonna be the ongoing trend, that there'll be a handful of companies that are ahead of open source by, you know, one or two generations?

Yeah. I, I think that's, like, the status quo, so if we just straight line project, I imagine that will continue to happen, and the real question is, like, can you stay in the lead if you are OpenAI and, um, and, like, get, uh, get paid for it? Or is that the, is that the objective of the organization anyway? Like, I, I think, you know, if you have a, um, a great leader and a lot of resources and a lot of really talented people, that's not something I wanna bet against.