a16zCan Open Models Keep Up With OpenAI and Anthropic? | Nathan Lambert
EVERY SPOKEN WORD
65 min read · 12,931 words- 0:00 – 0:58
Intro
- NLNathan Lambert
When I think about model progress, it comes down to different forms of capital, and that is compute data and talent.
- LLLisha Li
How do you see that playing out in terms of, like, who's gonna win in the next couple of years?
- NLNathan Lambert
The Chinese are so good at building models. To say that China is anything but equal on talent, you're gonna have a hard time. What they don't have is compute.
- LLLisha Li
Most people regard distillation as sort of like the main lever for them to do this.
- SPSpeaker
People distill my products every single day. They take it, they strip it down to its raw parts. That's called competition.
- NLNathan Lambert
Previous post-training was much more reliant on the supervised fine-tuning. Now this has morphed a bit into environments where it's RL heavy. But it's just, like, hard to map this type of post-training, which is you get the completions and the score goes up to what is going on in reasoning today.
- LLLisha Li
The discourse now is just so much fear on, like, how much the frontier is just hidden and also breaking away.
- NLNathan Lambert
The AI capability plot doesn't necessarily worry me, but the AI capability plot in the pretty much non-existing preparation of cross public cybersecurity and all of these things, like-
- 0:58 – 8:48
Compute, data, and talent
- LLLisha Li
So Nathan, really happy to have you here. Uh, thank you for joining in person. Super timely given you're launching Trillium, which I'd love to talk about. Um, but more sort of on a, on a macro level, um, I think we chatted a lot about open source, and that's, you know, a lot of what you care about. And, uh, what I sort of, um, or what, what we've been hearing a lot in the public discourse is like, "Hey, we have to pace the frontier," you know, the frontier this, the frontier that. And I think an arc of the stuff that I'm really trying to ask e-experts about are-- is, like, what is in the frontier? [chuckles] What is it even... Like, what are we pacing? [laughs] What's there? And then more importantly, kind of threading it with open source and open weight models, like, can they keep up with the frontier? And I think this matters very much, um, and is, is something that you're an expert on. So maybe to, to sort of, uh, you know, start on that arc, what's gotten to the f- uh, well, sort of how you would break down what's gotten us, uh, to the frontier for both closed, uh, labs and open, um, source models. You know, how does it break down in your mind, maybe first at a higher level of pre-training, mid-training, post-training, how post-training has evolved, which is something you've, you know, done a great amount of education, uh, to the public on.
- NLNathan Lambert
When I think about model progress, it comes down to, like, different forms of capital that you put into them, and that is, like, compute data and talent. And all of the labs have various amounts of this.
- LLLisha Li
Yeah.
- NLNathan Lambert
And I think the, the closed frontier labs of OpenAI, Anthropic, Google's have clearly had the most of these. And what we see-
- LLLisha Li
Yeah
- NLNathan Lambert
... is, like, the different abilities of the labs to organize these forms of capital. And, like, making the org chart work for building models is really hard, which is why you have-- companies have to do complete resets from the bottom up. And then it actually kind of works sometimes. Like, company [chuckles] like Meta has, like, totally reset-
- LLLisha Li
[laughs]
- NLNathan Lambert
... and they're, like, on a positive trajectory again.
- LLLisha Li
Seriously, yeah.
- NLNathan Lambert
It's like, I think this-- I'm very interested in these human aspects of, like, why competition drives the frontier-
- LLLisha Li
Yeah
- NLNathan Lambert
... and, like, why the org chart is so important because, like, we see these things, and this comes into play in the open/closed dynamic as well, where it's like-
- LLLisha Li
Yeah
- NLNathan Lambert
... the closed companies, the, the fron- the true frontier labs, which I say if somebody wants to work at the frontier of AI, I mostly tell them to work at Anthropic and OpenAI, which is the easiest bet. I think-
- LLLisha Li
Yeah
- NLNathan Lambert
... um, Meta, Google, um, SpaceX AI, like, I think you still might be there, but it's harder to see from the outside.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And, like, these companies have just taken the most ambitious bets the earliest in order to get there, and that's why OpenAI has done things like kickstart the reasoning model revolution. And I think that kind of breakthrough and, like, that beautiful plot with the inference scaling laws is, like, peak OpenAI. It's like they're really good at getting this plot that just changes how everyone thinks about everything.
- LLLisha Li
Yeah.
- NLNathan Lambert
And on the other side, it's like Anthropic has been, like, very focused on the types of things the models are going to be able to do, and then their reward was this Claude code explosion with their focus on coding for so long. And there's these subtly different cultures, and I think those two companies in particular have been the most forward-looking and the most willing to spend egregious amounts to make a step towards a direction that people think is impossible. The Chinese companies came into this because they want to have a company that matters, and if they were to build their models and try to sell an API from China, nobody would know about them. Like, nobody would know about Moonshot AI. Very few people would know about Zhifu, and there's a very long list after this. And this is a very practical decision by them. They're following the discourse. They know there's gonna be a huge market. They know there's a huge demand in the US for paying for these things, and it's very practical. So therefore, like, releasing open models is a way to-
- LLLisha Li
Yeah
- NLNathan Lambert
... shortcut this and get into the conversation. And I, I think there's a lot of complexity in, like, why do they do this? Is the government involved? And, and, and, [chuckles] and not. That was, like, not the case.
- LLLisha Li
[laughs]
- NLNathan Lambert
They're just, like, trying to build a startup, and it's not to s-
- LLLisha Li
So practical.
- NLNathan Lambert
It's not to say that they will not get there. Like-
- LLLisha Li
Right
- NLNathan Lambert
... the Chinese government is-- does things that would blow people's minds in the US, and I think that, like, that makes me think that the Chinese labs have a much more serious approach to security because, like-
- LLLisha Li
Mm
- NLNathan Lambert
... things happen in China that are much more drastic responses to companies doing things in the US. And I don't need to go down this rabbit hole, but this is just, like, between the balance of open and closed, it's just, like, these companies in China are trying to catch up. And this is a very different dynamic than what I said about OpenAI and Anthropic that are trying to create the new paradigms, that are trying to, like, make true RSI happen so that their company wins with the best models forever if you get there first.
- 8:48 – 14:57
Compute-starved, and the distillation debate
- LLLisha Li
Um, and where do you see the levers kind of accelerating?
- NLNathan Lambert
Yeah. So the Chinese labs are very technically excellent. They are more compute-starved, I think.
- LLLisha Li
Yes.
- NLNathan Lambert
Like, without a doubt, they have a comparable talent pool to draw on. I think there might be like a slight edge based on who was where when AI really got going. But to say that China is anything but equal on talent is gonna-- you're gonna have a hard time because they just have so many people being trained at such a high level. And what they don't have is compute. You talk to people there, and you're like, "Oh, I heard an OpenAI, average OpenAI researcher has 5,000 H100 equivalents that they can access at any time," and like their jaw just falls to the floor. They're just like, "I can't comprehend." And this is like a top, like a, a top pre-training researcher at-
- LLLisha Li
Right
- NLNathan Lambert
... one of, one of the labs that you would know, and then you have this interaction a few times. And, like, the compute thing, I think, is the biggest difference, and distillation plays in as it is a way to effectively spend more money to get what is compute equivalent. Like outputs from a language model-
- LLLisha Li
Mm-hmm
- NLNathan Lambert
... are a function of compute, and if you know the outputs you want, it's the same as having compute to generate with a, with a model. And I've seen the stages of distillation, and how I see it is that previous post-training was much more reliant on the supervised fine-tuning and other t- like this RLHF era, which is, um, where post-training started, and you could get most of your performance just by you do this next token prediction loss on the outputs of a very strong model. It was very established that you take the best model you can, you generate completions, and you train on it, and your model is gonna be pretty good so long as you have access to the right prompts. Prompts have always been a key to post-training. I remember at AI2, like with Hannah, who I co-led the post-training there with, we would always say, like, "Prompts are the key unit of post-training." It's like you need to get the prompts, and you can do different things with them at different stages of training, and now this has morphed a bit into environments where it's RL heavy. But it's just, like, hard to map this type of post-training, which is you get the completions and the score goes up to what is going on in reasoning today.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And based on the public literature, it's very hard for me to be like, "Distillation is a major factor," because, for example, in the fully open literature of people trying to do open post-training recipes, all the open model builders are stuck using like a really outdated Qwen model-
- LLLisha Li
Hmm
- NLNathan Lambert
... because you can't figure out how to get the teacher right. My hypothesis is that there's some like mid-training and something in the common pre-training data that's open that makes it hard to do this thing, which is get reasoning traces from the absolute best model and do SFT, and the numbers go up. So like if I were to have access to Claude's reasoning trace when I was building Olmo, I'm actually-- like Claude Opus, Claude Mythos, whatever, I'm not sure that it would have actually made our model a lot better.
- LLLisha Li
Hmm.
- NLNathan Lambert
So this is like my evidence grounded [chuckles] in the literature why I think distillation is hard. Um-
- LLLisha Li
Mm-hmm
- NLNathan Lambert
... and then, like, how do you take the outputs of a model and build an RL environment to do a big RL run? Because then you have the Xiaomis of the world doing this week-long open RL run where they're like, "Here's what we did," and then they've dropped the model. And it's like, where exactly does distillation go into this? They're, they're clever. They know how to do marketing well. There are some questions of like, did they do this ahead of time because they did their RL run, and then they dropped the model, and then their tech report said they had an extra post-training stage after RL? So there's some questions. But like Mapping the established distillation literature to that is very hard.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
But there are question marks that make me support the labs a bit, which is like these Claude proxies and these model proxies in China, where there's a ton of tokens that are used, and there's intermediaries, and they have tons of accounts, and even cases where Kimi was serving Claude in their products to users. And I'm just like, don't really understand how this fits into distillation as a common technical tool in post-training. And like, yes, we should watch it too. Like, there's other weird stuff with Chinese companies using the outputs from OpenAI and Anthropic's models at a vast scale.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
So I think we have a lot more questions we need to answer, and I think most reasonable people when they're like, "What's the gap from distillation?" are like, "There's very big error bars."
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
But when I say that distillation helps them by a few months, it's based on the scientific literature that we just, like, that's the information we have. We should do more open research and-
- LLLisha Li
Mm-hmm
- NLNathan Lambert
... enable this so that we can study it. But like, I don't know how you can come to a different conclusion that it's, like, not a huge effect in current post-training based on this.
- LLLisha Li
Yeah.
- NLNathan Lambert
Unless it's like, the Chinese are so good at building models as well. I think there's a bit of cope where people are like-
- LLLisha Li
Oh, yeah
- NLNathan Lambert
... "Only we are good at building models," and it's like-
- LLLisha Li
[laughs] How is that-
- NLNathan Lambert
There are so many companies. There's like 20 to 30 companies that are building good models.
- 14:57 – 19:24
Is distillation just competition?
- LLLisha Li
Yeah
- NLNathan Lambert
... many things in your head. [chuckles]
- LLLisha Li
I, I wonder what your response is. So I, I heard, I think Jensen said in some literally either tweet or interview, maybe even-
- NLNathan Lambert
It was like a great, it was a great quote.
- LLLisha Li
Yeah, he was like such a Chad quote.
- NLNathan Lambert
Put it, put it in the, put it in the show. [laughs]
- LLLisha Li
[laughs]
- SPSpeaker
People distill my products every single day. They take it and they sh- they strip it down to its raw parts. They test everything. That's called competition.
- NLNathan Lambert
Do you think that's fair game? You think distillation is not robbery?
- SPSpeaker
Uh, that's called competition.
- NLNathan Lambert
I'm very sympathetic.
- LLLisha Li
[laughs]
- NLNathan Lambert
'Cause I think there's a lot of calls from the AI safety community that we want to have, like, US-China dialogue as we're getting to such capability levels.
- LLLisha Li
Yeah.
- NLNathan Lambert
And if you're gonna ask the company to s- just like start hitting their AI industry with slaps on the wrist and more-
- LLLisha Li
Yeah
- NLNathan Lambert
... downstream of distillation. So you won't regulate distillation with policy from Congress. You will probably just do federal action to make their life annoying.
- LLLisha Li
Yeah.
- NLNathan Lambert
And at the same time, when we're talking about how do we manage frontier risks, where you have closed models in training going all over the web and accessing things that they shouldn't, and we have valid worries about, like, what will happen with open weight models, where there's, like, hasn't been harms yet, but it's like, why? Like, wh- why aren't people using open weight models? Like, why do they go to closed models and get around the safeguards? And like-
- LLLisha Li
Yeah
- NLNathan Lambert
... pe- we want to be able to communicate with the Chinese tech companies and everything about this, and I just think, like, you're gonna cause more harm than good if you do anything about distillation, because they're not gonna stop. It's all, it's like they're gonna keep doing it. Like-
- LLLisha Li
Right, right. And then, and, and the open source community benefits, and therefore the startup community benefits a lot. We can go into, you know, a lot of it. It's not just like, "Hey, China, USA." [chuckles] It's actually like there's... You know, and we can actually talk about, like, I am sympathetic obviously, or try to steel man the, um, frontier lab position, which is like you have just unprecedented amounts of capital that's required to build these frontier models, right? And it's, it's kind of alarming/not very, um, uh, conducive to, to this, uh, flywheel continuing if, if the, um, through distillation or other kind of distillation related methods, um, open source can catch up so quickly. And so, you know, and what I mean by this flywheel is just kind of a very simple picture of, you know, you, you train the frontier model by putting in a lot of money to get compute, and then they park this compute, of course. OpenAI and Anthropic are very guilty/successful at this. [laughs]
- NLNathan Lambert
Successful.
- LLLisha Li
Whichever way you wanna, like, look, look at. And there is a huge compute, you know, uh, kind of, uh, um, sort of, um, lack of compute available because there's actually, like, not just compute and the chips, but there's also power, et cetera. So, like, all of this scarcity is kind of, you know, bottleneck until 2028. So this is all to say that it's a successful strategy to park that which leads to actually a lot of, um, revenue, [chuckles] because once they get to the frontier, because they understand, you know, scaling is the one thing that kind of gives them, or the science of scaling is what gives them a handle over how, how well, um, that can translate, like, into a predictable revenue stream. So they get the revenue, and because of that revenue, they're able to raise more money. It's just this unprecedented, like, beautiful flywheel, and this happens for both OpenAI and Anthropic. Um, of course, we need exponential [chuckles] amount of resources. And so we've, we've gotten to the point where we're like, "Okay, we need to IPO. [laughs] We need to go to the public markets." And so anything that threatens this flywheel from functioning well, um, is, is a huge threat to, to Just the entire, like being at the frontier. Um, and so-
- NLNathan Lambert
For the entire AI industry, realistically.
- LLLisha Li
[chuckles] And that, and exactly that actually. So like, you know, maybe not just kind of being sympathetic to their version of like trying to, you know, uh, eke out capitalism for themselves, it's, it's also just like, hey, if, if they don't do well in the IPO, that's actually bad for everyone else. And so I, I, I want everyone... You know, I want them to do well. Um, on the other hand, you know, it's like the Chinese open source models are a huge benefit to the startup ecosystem. I mean, we see this all the time, um, just kind of metrics I remember from like Decagon and Harvey was trying to do a lot of fine-tuning. Um, there's s- like a huge amount of, uh, startups doing post-training as a service of various degrees and trying to productize that and, and it actually works because you can get to the Pareto frontier with these, you know, Kimi K3 or pick your favorite like GLM, um, uh, open source model. So I guess like weighing these two, uh, things, how do you, um... Oh, f- uh, I, I guess there's so many questions can fall from that, but I guess, one, um, how do you see that playing out in terms of like who's gonna, who's gonna win in the next couple of years versus, you know, what,
- 19:24 – 25:00
Open and closed on different timescales
- LLLisha Li
what, uh... And, and then also relate that to, you know, why you're doing obviously an open source.
- NLNathan Lambert
So I think open and closed models are on different timescales of exponentials, and the closed labs are so integrated where they have efficient inference, they can serve the model at scale when they are ready to because they have all these resources, they can plan, and they get to these markets first that are very high value. We have seen this with coding, and this has been the story of the year, where the revenue has just gone zero, 100 billion and likely to grow. But the open-model labs, partially this is, this is like an economic issue being the fast follower, is that you don't have this flow of information and you don't have this flow of revenue. So the open model, you have an open model builder, then you have the open model inference companies, and you have the open model post-training companies, and like there's information from those customers, which are like pr- things you need the models to do, and it's like you can't... Like that information doesn't just flow back to the model builder-
- LLLisha Li
Yeah
- NLNathan Lambert
... very easily. And they also don't have the economies of scale to serve as many users, to serve it as efficiently, to have as much margins on influen- uh, on inference, and these are just like it's gonna take a lot longer. But to the extent that things can be commoditized, it seems like companies really want to have private data. There's a lot of things that they just cannot send over an API, and we can talk about the like, um, zero data retention or not and so on. But like these, it takes a long time to figure out if a company has internal expertise, or they figure out how to build an internal evaluation that actually matches to revenue.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And then they figure out how to fine-tune, or then they go find, they go find thinking machines or prime intellect, and they're like, "We can make money en masse if we figure out how to fine-tune a model that solves this."
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And like they have these customers, but there are just so many more companies involved that it takes so much longer. But at the end of this process, they have real ownership, which is model weights, no terms and conditions that they can run on their hardware, and they've figured out they can make an ROI on. So this is why I think the open model economy is just it takes way longer to build, and I'm less concerned about this being kind of bubbly, where it's like this could eat into the market of the really fast takeoff as people figure out how to... what is the cheapest model that accomplishes a task and how do you compete on margins.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And I don't really know, like I don't really know what's gonna happen. I think there's-
- LLLisha Li
Yeah
- NLNathan Lambert
... it's, it's, it's too early to know.
- LLLisha Li
It's, it's, it's so... Yeah, it's, it's so hard. I mean, it's definitely something that's always top of mind thinking about, like there's thi- there are things where scaling up general reasoning just as a general model is so, uh, it, it's going to beat out any specialized model, and we see obviously examples in math. [chuckles] And that's probably the most pure domain that you can see this. Um, but also, you know, even in, say, robotics, you know, with, with Astra, um, and, and I for maybe, uh, however degrees of like, uh, tru- truthy the signal is, it just doesn't sound like they've actually tried to ascend in robotics in general.
- NLNathan Lambert
Oh, I think they did.
- LLLisha Li
[laughs]
- NLNathan Lambert
I think this is a cycle of like six to eight months ago, they started hiring people and buying a lot of data on robotics.
- LLLisha Li
They did.
- NLNathan Lambert
And then the model magically six months ag- later is really good at it. But this is why I think that the-
- LLLisha Li
But I actually-
- NLNathan Lambert
[laughs]
- LLLisha Li
Well, the other, other, you know, more contrary things I've heard is that it's actually coming out of computer use, um, because it's, you know, 'cause computer use is like some level of spatial reasoning and then getting to have good Blender, [chuckles] you know, demos is, is another extension of that. Um, so you know, uh, we can believe some version of that. But either way, I guess it's like it... Maybe if it's like really specialized, you see it's, it's the capability gain is really from like trying to get better in a certain area. But like in math, I don't even know if that fell out of trying to specialize in math. I mean, I think the whole thesis there was just like, let's just scale up reasoning and actually got better in, in, in natural language and got better than any specialized model in math. I mean, Anthropic wasn't even trying to be good at math. You know, OpenAI, [laughs] you know, certainly I think, you know, they hired a bunch of mathematicians, but they also weren't trying to like use specialized, you know, kind of like lean data set to get better at it. Um, or at least that's not, um, the story. So anyways, this is, this is all kind of fodder for the view that like it's kind of hard to compete with the frontier models because eventually they start ascending so much that, you know, you, you get, uh, good enough in respective, um, categories. With that said, I think actually there's also very competing, like a counterforce of data that you're seeing, which is like the open source models that are already good enough, uh, allow for fine-tunes, you know, to your point of like these slow percolation diffusion of these, uh, um, very specific benchmarks and, and data sets that are only available in these, you know, companies that are, you're being applied to. And it's just, it's just good enough, and it's actually we're starting to see, um, people adopt that. I mean, in the most maybe pure regime where you would expect frontier models are good is, um, I think Cursor published something over the summer where like, "Hey, we just, you know, we, we had this long horizon task of writing, rewriting SQLite," and turns out using two frontier models, you know, one for planner, one for, um, execution, is not as ... good or is equivalently as good, but far more costly than just using Composer with maybe Opus or, you know, Fastra, whatever, as planner. And, and that, I think that's actually great evidence that the open source fine-tunings are very good or good enough and has surpassed, um, intelligence-hungry tasks. Um, is a long way of asking what evidence do you find is most compelling or, you know, pieces of evidence for various versions of this stable state [chuckles] of where the frontier wins
- 25:00 – 31:29
Frontier progress is really about data
- LLLisha Li
versus open source?
- NLNathan Lambert
I'm a big proponent that a lot of the frontier progress in domains that are not fully verifiable is by buying data that is high quality and giving the models worlds to act in and hill climb on. And I think there's two ti- there's two things. We have the math and code thing, and we, we, we don't know how big the coding market will be, and then there's kind of everything else, which is still they're targeting this knowledge work, this biology, healthcare, legal. And all of these things are not as verifiable, and I think it is much closer to the story of OpenAI bought a lot of robotics data. They're probably gonna tell us it's generalization, magic, poof. People love poof generalization.
- LLLisha Li
[chuckles]
- NLNathan Lambert
Um, and, but realistically, it's probably some of A, some of B, but a lot of it comes down to data. When I talk to people at Frontier Labs, there's some people that have amazing jobs. There's some people that contribute by doing a very narrow thing, which is you make a dataset and you-
- LLLisha Li
Mm-hmm
- NLNathan Lambert
... figure out what the behavior should be for your model in that area, and you figure out how to make the data that helps get that data-
- LLLisha Li
Important work [chuckles]
- NLNathan Lambert
... to get that behavior, and you figure out what data you need to buy. And that's gonna take, I think we're gonna have years of this-
- LLLisha Li
Yeah
- NLNathan Lambert
... across different sectors, and that's very capital intensive to buy all of this data. But that's not just like poof, magic, RSI happens. Where I do-
- LLLisha Li
Yeah
- NLNathan Lambert
... see in this math and code and some, like, em, distributed ML engineering, like it's very verifiable.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
You know what your cost is, and the AI models are gonna drive that down at a really magical rate.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And that's gonna just, just have a lot of surplus benefits, and we can do research faster. You could lo... I'm very interested in, like I think the efficiency of models is gonna drop multiplicatively over years. So we've seen OpenAI had drastic price cuts, and it's like what does it mean for GPT Sol or Astra and these bigger models to reduce in price 2X every eight months for the next three years? [chuckles] Like you start getting-
- LLLisha Li
Lucky us, right? [chuckles] What a time
- NLNathan Lambert
... you start doing this math and it's like it's 1% of the cost in whatever, like four years.
- LLLisha Li
Yeah.
- NLNathan Lambert
And it's like, are they gonna be able to have higher margins due to Jevons paradox there, or their revenue's gonna decrease because the market fills up? There's just like all these really weird dynamics, but fundamentally, like AI progress is about data and about writing better software to serve your models, and those are very different worlds. [chuckles]
- LLLisha Li
Yeah. I mean, at least definitely, I agree with you that observing from the outside, there's a lot of, um, data RL environment companies that are, you know, doing very, very well, um, selling both to the, the top two frontier labs, but an increasing kind of longer tail of, uh, labs, uh, as well. Um, so to your point, I think a lot of the capability gain has really come from just scaling that up. Um, and it's not an easy task. I mean, maybe there's... I'm actually very curious on your opinions between, uh, the, the data collection and environments that are built by hand versus the ones that are, you know, done in sim. There seems to be like different [chuckles] versions of like what could be, you know, what, what kind of price they command, et cetera.
- NLNathan Lambert
They all have good things and bad things.
- LLLisha Li
Yeah.
- NLNathan Lambert
And I just think there's a lot of talk of like, like we've talked about distillation, but the other side is like the Chinese company buys the RL environment-
- LLLisha Li
Yes
- NLNathan Lambert
... that they use six months later and like-
- LLLisha Li
Exactly. Oh, Anthropic or OpenAI bought this. Now there's a premium. [chuckles] I've heard that so many times
- NLNathan Lambert
... the good, the good data will do this.
- LLLisha Li
Yeah.
- NLNathan Lambert
But the data that people are buying is pretty flawed. Like-
- 31:29 – 38:05
What Trillium will build
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And I, I think this is gonna just take years. Uh-
- LLLisha Li
How do you think about then, uh, using environments and, like, your purchasing decisions as part of the nonprofit that you're starting?
- NLNathan Lambert
I think most of our work will be making open source environments for tasks that are very high leverage.
- LLLisha Li
Mm.
- NLNathan Lambert
So I think of this work that I advise a UW or a UW and AI2 student, Hamish Ivison, on this TMax work, which is, you use a few... It's for terminal tasks, which are, like, super popular in Terminal Bench, whatever number we're on. And this is fully synthetic data, where you prompt a frontier model to generate a type of problem, and you use a few different prompts to generate diversity, and then you use model rollouts and other things to find bugs in them and eliminate the ones with bugs.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And this process is not magic. It's like, you read every data point, your data will get better, and you can do this. But there's just not that many institutions that are investing what it takes, thousands or tens of thousands of dollars, to make these environments well and then release it.
- LLLisha Li
And release it. [chuckles]
- NLNathan Lambert
Because the profit, the, because the profit motive is so high.
- LLLisha Li
It's so high. [chuckles] Yeah, it's like exactly-
- NLNathan Lambert
And you'd be like... I was joking with the author, like, "Yeah, you could probably sell this data for a million dollars and then release the paper." And they're like, "No," they're like, "We care about doing the right thing."
- LLLisha Li
[chuckles]
- NLNathan Lambert
But it's just like-
- LLLisha Li
Oh my God, yeah
- NLNathan Lambert
... these open resources are the fun- foundation of-
- LLLisha Li
Yeah
- NLNathan Lambert
... so much study. And it's like, you release some RL environments with some RL rollouts. People can do research on, say, like, eval awareness. You, you have the generations from the model. People can look for different behaviors. If you've done RL on it, they can see how it goes over time. And this is just the, the units of infrastructure that PhD students-
- LLLisha Li
Mm-hmm
- NLNathan Lambert
... spend a year looking at and produce a increased knowledge. And the tech industry is the place where value is captured from CS research.
- LLLisha Li
[chuckles]
- NLNathan Lambert
All CS research-
- LLLisha Li
Thank you
- NLNathan Lambert
... the benefit accrues to the tech industry in, like, 10 years. And I just think, like, this is why more people supporting open, that if they're gonna work, if you're gonna work in the industry for 10 years-
- LLLisha Li
Yeah
- NLNathan Lambert
... like, your peers are gonna be the ones that benefit from it.
- LLLisha Li
More fossil fuel. Let's not deplete it all. [chuckles]
- NLNathan Lambert
But it's just like the opportunity to for short-term win is so high.
- LLLisha Li
Yeah.
- NLNathan Lambert
The opportunity to make, what is it, $100 million, like, nine-figure returns for your business within five months of founding a startup with your friends is just like, these are people who may have done more open source work, or these are people who left their PhD-
- 38:05 – 44:32
Demystifying RSI
- LLLisha Li
Um, another area I think I read on your announcement post is RSI, which is, you know, quite, um... I think, I mean, we can first talk about what is actually recursive about it or how scary it is or whether we're gonna have takeoff. I think you have a very grounded opinion on that. Um, uh, but, you know, another thing I wanna kind of talk about is, I think RSI, maybe related to how much it's kind of hyped is, is a reason why we are afraid the frontier labs will pull away. Um, in fact, it's also a reason why I've heard, not just once, but several times, that people are, one, convinced that their own jobs are gonna be automated, a- as a researcher. [chuckles]
- NLNathan Lambert
Yeah. Yeah, like-
- LLLisha Li
Yeah. And that's-
- NLNathan Lambert
I've gone through the last few years, some of my best friends, I'm like, "You are some of the best people at the world at this."
- LLLisha Li
Yes.
- NLNathan Lambert
And then they go to the work and they're like, "Nathan, I'm, I'm struggling." Like, "I'm not-"
- LLLisha Li
Yeah.
- NLNathan Lambert
"Like, what am I gonna do?" [chuckles]
- LLLisha Li
'Cause, 'cause to your point, like, a lot of what they do is verifiable, right?
- NLNathan Lambert
Yeah.
- LLLisha Li
For, um, but because of that, you know, there's actually quite a lot of other sort of non-verifiable or harder out of distribution tasks, such as, um, uh, uh, what is it? Alignment and things like that. And so actually because, I mean, one argument I heard was because there's such a difference between how well RSI is doing for model capabilities versus alignment, which is much more of out, out of distribution, maybe creative work, that gives people an ease on how, you know, maybe there is actual dangers, et cetera, et cetera. Um, so anyways, because of this, I'm very interested in hearing from you, like how... 'Cause I think it's really important that we do have an open source, um, kind of front in, in showing how RSI is being done, how much maybe then we can, you know, understand the capabilities of, like, if in within a lab they have a better model, what, what are the chances they actually pull away? Just better understanding and calibrating. Um, so yeah, I would love to hear more about that, that.
- NLNathan Lambert
Yeah. So I think this, we won't be the only people doing this, but essentially, like RSI has so much mystique.
- LLLisha Li
Yes.
- NLNathan Lambert
Where like we're using models for everything. And like-
- LLLisha Li
[chuckles]
- NLNathan Lambert
... what we can do as a open lab and a nonprofit, and I think other companies like Eleuther or Hugging Face or Open Athena, which is Marin and Stanford, like they will get into this with this good marketing line for them, is, and me, that's why I'm doing this. And it's like we can share what tools we're using. We're gonna build an open source stack for SFT, for mid-training. We're gonna use SkyRL, Miles, all these things that you know. And we have our task, we have our configs, we know what type of compute we're running. We can work with Claude and GPT and all these models. And we'll just be like, we can store the trajectories for what we did. We can tell you what our tasks, whatever, use Linear or whatever these popular things are. We'll say what our tasks were and why we chose them and whether models work on them or not. And you just do this for a year of a lab, and you're like, "Here's what we did."
- LLLisha Li
Yeah.
- NLNathan Lambert
And then you just check in every six months and we're like, "Has it changed?"
- LLLisha Li
Yeah.
- NLNathan Lambert
And, and it's just like to be able to have the full stack and not need to go through these classifiers and to like try to s- like say what the task is or vaguely and so on.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
It's just like people that can share what automation is happening should do this. And we have a different distribution of tasks, but I think that the research is similar. [chuckles] Like, I think, like-
- LLLisha Li
Yeah
- NLNathan Lambert
... the AI research that I've done at AI2 and will do at Trillium versus what people do in some sects of academia and the labs, it's not that different. You have more resources, you scale it up, and it's just like so good for the world when everybody's talking about this to just, just show people like-
- LLLisha Li
Yep
- NLNathan Lambert
... this is what Nathan asked his model to do, and this is what it got stuck on. And I think that will help a lot of, like, literal policymakers-
- LLLisha Li
It's like TikTok. [chuckles] Exactly
- NLNathan Lambert
... like be like, "Oh, okay."
- LLLisha Li
[laughs] It's like, "Oh, that's why it's very logical."
- NLNathan Lambert
[chuckles] Like, "That's what it got stuck on."
- 44:32 – 49:32
Can open source pre-train?
- LLLisha Li
Yeah. I... Do you think that there's, like, a chance for any pre-training open source effort? I know the capital requirements are just, like, order, orders of magnitude larger, so that is what's difficult. Um, but maybe what are the values of that, and then maybe we can kind of discuss potential commercial strategies and make it viable.
- NLNathan Lambert
Pre, pre-training science is pretty... Like, the, the tools are pretty accessible.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And you have what is a pre-training ladder, which is, like, the decision-making tool you have to test if your ideas scale, and the pre-training data isn't growing at, like, at, like, the environment scene. Like, it's essentially... So first approximation, you train on the internet.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And, like, there's a lot of, there's little, there's little details and secret things, but-
- LLLisha Li
Yeah
- NLNathan Lambert
... but, like, that's not changing that much. The state-of-the-art architecture is, and the, the GPUs are, and that's, like, th- that changes. And, but it's not astronomically expensive to make a pre-trained model that is solid. So I think, like, you don't have to spend a billion dollars to get a very good base model. The more you spend, the better you will get, but it's like, people keep making progress, and I'm constantly surprised by a new lab just goes from scratch because they can. And some of this is on open tools, so, like-
- LLLisha Li
Yeah
- NLNathan Lambert
... you can go off Megatron. Like, a lot of new startups, they're like, "Oh, what, what should we train? We train on..." [chuckles] They train off the open source tool, and they fork it-
- LLLisha Li
Mm-hmm
- NLNathan Lambert
... and then they never release it again. This is what all the, all the, like, even prominent open model labs have done. They take the open source code, they make it their own thing, and they just, a lot of them don't push upstream.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
So I've always been surprised by this. So I'm, like, not betting against new entrants because, like, like, it's not that you need $10 billion to, like, have a relevant model. The hard thing is, like, get a model that's good enough for people to use.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And to get a story where if you release it, you can do your next fundraise. And just, like, you have to get past this lock of, like, we need our first model to land.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
If, like, your whole company is based on models are the best, and you're not in the ballpark, and you're not releasing, like, you're on the clock. And it's like, I think of, like, Thinking Machines getting Inkling out, where it's like they were very transparent. They're like, "This wasn't the best model. We're proud of it. It has some cool things."
- LLLisha Li
Yeah.
- NLNathan Lambert
And they're like, they're now, they've released their second model, and they're getting going, and they have Tinker that has this. Like, that's very good for them. They're, like, off the ground.
- LLLisha Li
Yeah.
- NLNathan Lambert
And it's just like, that, I don't think that's what they were do- Like, they were building models, but open models wasn't their first plan, and then they got there. And there's a lot, there's rumors that Me- like, Meta said they'll release a new open weight model again. Like, that'll come. There's a lot of companies out there that couldn't afford models.
- LLLisha Li
Yeah. Yeah. No, I'm definitely, you know, glad that Thinking Machines is, and it makes so much sense for their business model, um, 'cause they do post-training. Um, and, you know, with Meta, I think you mentioned this as well in, in some previous writings, it's just for them, it's just such a good strategy. I think it was also just kind of obvious when they, [chuckles] when they do it, it's like, "Hey, it's complementary to my, you know, revenue streams."
- NLNathan Lambert
And Meta-
- LLLisha Li
"Why not? Let's just drop a bomb"
- NLNathan Lambert
... Mark wrote the best blog post on this. Like, when Llama 3 came out, he wrote, like, the best, like, why open is the path forward.
- LLLisha Li
[chuckles] Right, right.
- NLNathan Lambert
The guy got, guy got a little distracted. Seems like he's coming back to the light. [chuckles]
- LLLisha Li
Right. Right. Yeah, yeah, yeah. So I hope they, they, they keep on being motivated to do that. Um, getting to the frontier with pre-training, you know, I think there's a lot of help given that there's such good open source models to maybe, you know, somehow still kind of bootstrap from. I, I keep on getting kind of conflicting signals as to, like, whether pre-training is actually as saturated or well understood. I mean, maybe, maybe to your point, like, there's a lot of, like, well-known techniques at work, but what's hard is just actually having access to the 20,000, [chuckles] 30,000 chips to run.
- NLNathan Lambert
I think people are turning the crank on pre-training, and pre-training became adjusted when super long context generation happened, so there's just way more return if you're, if you can, for example, like, get a sparse attention architecture to work as well, and then your 100,000 token generation is 40% faster.
- 49:32 – 58:34
Harnesses and agent swarms
- LLLisha Li
Yeah, yeah
- NLNathan Lambert
... it doesn't take as much capital to make very big compounding gains-
- LLLisha Li
I see what you're saying
- NLNathan Lambert
... on the state of the art-
- LLLisha Li
Yeah
- NLNathan Lambert
... in the open source, 'cause the bar is very low. It's like-
- LLLisha Li
Yeah, yeah. [laughs]
- NLNathan Lambert
... it's not, like, it's not close.
- LLLisha Li
Yeah, yeah, yeah. No, that, that makes sense. It's just like a, a kind of efficiency where you can put capital and make a huge-
- NLNathan Lambert
Yeah
- LLLisha Li
... difference. Yeah, yeah. Kind of related to the whole, you know, post-training side, maybe, you know, pulling away from closed and open, is the fact that the agents, you know, the type of harnesses that are being used in the closed labs are very different and slash, you know, actually trained and, you know, post-trained, uh, onto to, you know, uh, to, to be specifically good for, um... Their models are specifically good for those harnesses. And so that in- increases the mystique even more, makes it very hard. Um, how do you, I guess, one, do you agree? And then two, like, how are you sort of hoping specifically to kind of address that? And w- things I, I mean are really examples like, hey, spin up like thousands of agents, [chuckles] tens of thousands of agents, and just, like, a very different workflow than what people are, are obviously, um, exposed to.
- NLNathan Lambert
I think it'll be okay for open because I thought years ago that, say, the closed labs would be like Apple. They'd be the integrated company where they have the model designed for the harness, which was evolved. It gives them, um, like, kind of economies of scale, efficiency. But it turned out that so many people have different harnesses for different things that they like to do, that the models companies have been pressurized to make it so that their model can work in all sorts of different harnesses.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And I thought it would be a weakness of open models that they have to be designed to train on all sorts of types of harnesses. But it ended up being pretty equilibriated, or, like, it's at equilibrium where they all have to do it to some capacity. And then what we were seeing with this, like, agent swarms is that OpenAI and Anthropic are gonna be earlier to it. They're gonna know how to design their models. They're gonna know how to design, what do we call, what do we call, wanna call a harness for a swarm? I don't know. Like, [both chuckling] I, I, I f- I don't know, but there's gonna be a word for it. It's not a harness.
- LLLisha Li
Yeah.
- NLNathan Lambert
It's like a coordination system.
- LLLisha Li
[laughs] Yeah.
- NLNathan Lambert
But then eventually other people are gonna wanna use them, and then I think it's gonna be like these cycles where the frontier labs-
- LLLisha Li
Yeah
- NLNathan Lambert
... like, they push this out, and they have this... To date, they've had huge returns on being first. But, like, what is the returns on Navier-Stokes and all these things? Like, we don't really know.
- LLLisha Li
[laughs]
- NLNathan Lambert
Like, we don't know. They push the frontier-
- LLLisha Li
Yeah
- NLNathan Lambert
... in the assumption that there will be high returns, and that this marginal intel- marginal intelligence has, like, disproportionate returns.
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
And we don't know if that's always true. But I kind of expect these cycles of, like, once you know the shape of how do you control 10,000 agents in one harness, then the open model companies will be like, "Oh, we can do this too."
- LLLisha Li
Mm-hmm.
- NLNathan Lambert
But it's not like Timmy is gonna do that first. For one, they don't have the compute.
- LLLisha Li
Yeah.
- NLNathan Lambert
It's an astronomical amount of compute. But these things go through cycles. Nvidia's chips will get way faster. There's more compute built out.
Episode duration: 58:48
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode BKuK0M6oDTI
