By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Latent SpaceRecursive Language Models — Alex Zhang, MIT PhD
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
EVERY SPOKEN WORD
110 min read · 21,898 words- 0:00 – 1:56
Hook
- AZAlex Zhang
Oh, like I love Claude Code, I love Codex, I love Pi. Like, no, I love Oh My Pi, I love Prime Agent. It- it- to be honest, I think all of them are the same
- SWswyx
You know, your- your models actually are capable of a lot more if you try harder, so this is a skill issue.
- AZAlex Zhang
Yeah, yeah, yeah, basically. I- I-
- SWswyx
[laughs]
- AZAlex Zhang
I mean, I- I- I think there's one thing I want to see. I- I appreciate that there's a big focus on like jagged intelligence because it paints a big picture of like we can do this if we really set our minds on it. But I kind of wish... And- and maybe someone in academia should do this. Like really just sit down and think about like if I took Astra, even the current frontier models are not good enough at like doing a particular job over, let's say, the- the span of a month consistently and well
- SWswyx
It's, uh, 10,000 agents in 88 hours.
- AZAlex Zhang
Oh, yeah.
- SWswyx
Um, 130 billion to- output tokens, which is estimated to be about $40 million in public pricing.
- AZAlex Zhang
May- maybe you'll have to pay like $40 million to get a result.
- SWswyx
[laughs]
- AZAlex Zhang
And it's like, well... But it- it's exciting. I- I- I will say like it is-- it's very exciting that we even have the option to point $40 million at a problem and solve it.
- SWswyx
Yes. Yes. Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring The Latent Space to you each and every week. If you do it, I promise you, we'll never stop working to make the show even better.
- 1:56 – 2:45
Introduction
- SWswyx
Now let's get into it. [upbeat music] All right, we're here in the studio with Alex Zhang, uh, I guess most famously of, uh, RLMs, uh, but you have, you have a lot- a few other affiliations. Welcome to the show.
- AZAlex Zhang
Yeah, thank you for having me.
- SWswyx
Yeah. I guess GPU Mode as well?
- AZAlex Zhang
Yes, GPU Mode as well.
- SWswyx
Uh, you were shepherded in by Mark Serafini. Not- not everyone gets that kind of welcome.
- AZAlex Zhang
Yep, yeah.
- SWswyx
Uh-
- AZAlex Zhang
Yeah, I'm very, very close to all the people in GPU Mode. So yeah-
- SWswyx
Yeah
- AZAlex Zhang
... we often end up working together in various capacities, like even beyond just GPU Mode itself, so.
- SWswyx
Yeah. Can we explain, uh... So people who are not that close-
- AZAlex Zhang
Mm
- SWswyx
... don't know about this. It's just a, it's a Discord. It used to be focused on, I- I guess, CUDA mode, and then- then- then generalized a little bit. Uh, it was started by Mark.
- AZAlex Zhang
Yep.
- SWswyx
It was basically like... To me, it's like the hiring pipeline of the PyTorch team.
- AZAlex Zhang
Mm-hmm.
- SWswyx
And then he left PyTorch. [laughs]
- AZAlex Zhang
Yep, yep, yep.
- 2:45 – 9:34
GPU Mode, KernelBench, and AI-Written Kernels
- AZAlex Zhang
Yeah. So it used to be, I think, uh... It started actually around when I was in college, um, in like 2023. Uh, I think it was started by Mark Andreas and Jeremy Howard. The original premise was just like it was a GPU or it was a Discord dedicated to learning how to write GPU kernels, and they had like lectures. That was basically the extent of it. Um, and I got interested in it because I was writing GPU kernels. It was actually out of like pure chance. I was interning at Snapchat at the time, and-
- SWswyx
Rexxars.
- AZAlex Zhang
Yeah, I was very bored [laughs] with Rexxars. So, um, they had a project where like they were interested in writing... It was this paper called Infinite Attention. It was like a Google paper.
- SWswyx
Yes, we've covered it on Paper Club.
- AZAlex Zhang
Yes, yeah. So I was interested in- in whether or not you could write specialized kernels for it at Snapchat. Um, it didn't... Nothing really came of it, but I joined GPU Mode. At the time, it was called CUDA Mode, uh, I think for like legal reasons or something, they changed the name. But I met Mark, uh, I met Matei, I met a bunch of other people that were very involved in the community, and then Mark had pitched this idea called Popcorn, which was now what you see as the leaderboard today. But the general idea was like I think all of us had this like intuition that GPU programming is like very similar to if you guys have done like competitive programming. It's a, it's a s- not-- I d- I don't mean to say like they're transferable skills.
- SWswyx
You have constraints. You code golf a little bit.
- AZAlex Zhang
Yep.
- SWswyx
Yeah.
- AZAlex Zhang
Yeah, yeah, yeah. And there's like, there's actually a surprisingly small space of optimizations that people do. Uh, and there's actually not that many kernels per se that people are interested in optimizing. And so we kind of had this thought that like if you had enough data, like in the same way that code forces, there's millions of problems. If you could do this with GPU code, like you could scale and automate kind of GPU kernel development, which for researchers is a huge deal because I think one of the bigger bottlenecks... Like if you look at like Mamba, for example, like they released the paper with kernels because otherwise, like you can't really use it in any meaningful way, you know? And not everyone has like a tree dal on their team. So we're very interested in this. Uh, KernelBench kind of spawned from that too, of like can we get LLMs to automate, um, GPU kernel code? And I think that was like a... It was a very, very fun time. It was like between college and my PhD, and yeah, I- I had a really pleasant time doing stuff with GPU Mode. Now I kind of just help with the lectures sometimes. Uh, I'm not as involved, and I think in general, like we don't have as many competitions as we used to. But, um, yeah, I still keep in touch a lot with- with everyone there.
- SWswyx
Is there a friendly rivalry? Because I- I think the previous community that used to do this was like MLsys, MLPerf-
- AZAlex Zhang
Yeah
- SWswyx
... kind of thing. Is there a friendly rival- rivalry? Is this like the just new generation MLPerf, or what's going on?
- AZAlex Zhang
The nice thing about GPU Mode is that it is also a community in the sense that like a lot of the lectures are very easy enough for a beginner to follow and ask questions and things like that. And like the competitions are like somewhat, not secondary, but like you can participate in them to learn. I think with a lot of, like MLsys, MLPerf kind of benchmarks, like for the most part, like only serious labs and companies participate, like- like seriously in them at least. Like that's, that was my understanding of it. Um, I- I could be wrong. But I- I think also Beyond GPU Mode now, one thing that has been really exciting is there's a lot more websites and, like, people that work on hosting competitions. Like, I think there's this, I think there's this website called, like, LeetGPU or something, and it's like LeetCode for GPU problems.
- SPSpeaker
Wow.
- AZAlex Zhang
Um, there's, like, other ones too that I've seen-
- SPSpeaker
Le-
- AZAlex Zhang
That, like, we've, we've seen... Like, there's many that have kind of spawned and, like, talked on GPU Mode, and, like, um, it's very exciting in the sense that I think GPU programming used to be super niche. Like, when I was interested in it, and the only reason I got interested in it was Tri Dao gave a talk at Princeton because he was applying for faculty. I mean, he is faculty there now, but I listened to his talk on flash attention in, like, 2023, and I was like, "Wow, this is, like, the coolest thing ever." [laughs] Um, and I was like, "This is, like, this is what everyone should be working on." I guess, like, vLLM and stuff had come out too, and it was like, "Oh, you know, we should be writing kernels." But now it's like, you know, everyone writes kernels. Like, everyone... It's, it's-- I think it's actually almost saturated in some sense, uh, as a field.
- SPSpeaker
Any, any interesting takes for people that wanna get into it? So I think one of the biggest news is GPT 5.6-
- AZAlex Zhang
Mm-hmm
- SPSpeaker
... wrote more efficient kernels, so Terra and, uh, Luna could be 80% cheaper.
- AZAlex Zhang
Yeah.
- SPSpeaker
And then we've seen other competitions where people are, like, setting records, and they're like, "We're doing some auto research loop," and these are people that don't have a background-
- AZAlex Zhang
Yep
- SPSpeaker
... in any kernel writing, right?
- AZAlex Zhang
Yeah. So even on the GPU Mode leaderboard, uh, if you look at, like, a lot of the recent problems, almost all the solutions are AI-generated.
- SPSpeaker
Mm.
- AZAlex Zhang
However, you'll notice on the leaderboard... So there's this guy named Gaunurst, who's, like, a very, very, like, regular member of GPU Mode. We've always known for a long time that he's, like, a super, super cracked, like, GPU kernel writer. One thing we discovered on this leaderboard is, like, almost ev- I mean, he also used AI to help him with these solutions, but for the most part, like, he helped prompt and move it in certain directions. Uh, we found that, like, his kernel was, like, basically the only one in, like, the top 10 that was actually stable in, like, actual, like, end-to-end systems. And it does bring into question, like, it's not... Like, I mean, GPU kernels have a verification problem. Like, we've kind of known this. It's been a problem since KernelBench was released. Like, there's a lot of reward hacking that goes on. But you al- yeah, you also notice, like, the, the lines of code is a lot, a lot smaller, but-
- SPSpeaker
Yeah, I was gonna ask, is that noticeable, or is it just-
- AZAlex Zhang
Yeah, no, it's-
- SPSpeaker
Okay
- 9:34 – 15:16
Human Expertise vs. Brute-Force AI Search
- AZAlex Zhang
Yeah, yeah.
- SPSpeaker
Yeah.
- AZAlex Zhang
So I think it's like, it's a mix of those things, of, like, the people who work... Like, the people who know how to look at these problems and how to solve them, like, also know how to use AI to do them because, like, you're acting as a very strong verifier. Like, if you are knowled- or if you know what to do and you're also... Like, I, I, I think the thing that we've kind of discovered with all these agent swarms and things like this is, like, when you throw enough compute at a problem, you, like, can sufficiently explore solutions to that problem. But oftentimes, like, you know, maybe you can burn, like, 100 billion or a trillion tokens on something, but if you bring in someone who knows something about the problem, um, they can uncover something for the model that would, like, erase that one trillion token spend. I mean, it's not, it's not super clear, like, what exactly the, the trends are here. But I think, like, there are so many problems in the wild still right now that we want to solve, and, like, we can't afford to just always, you know, throw as much compute as possible at it. Like, there is still an efficiency aspect of, of all of these things that is super, super important.
- SPSpeaker
Is there, like, a theoretical right answer that you can just calculate based on physics, and then you just get close to the physics limit?
- AZAlex Zhang
Yes. So for GPU kernels, you can compute... It, it's actually not that easy to compute sometimes, like, depending on how complex the problem is. Like, for, for matrix multiplication, it's very easy to compute, um, this, like, speed of light kind of, uh, estimate of what the fastest kernel can be. And, like, this is also assuming, like, you know, maybe all of your, all your data starts on the CPU, or maybe it starts in DRAM, on the GPU, et cetera. Like, this changes these, these numbers slightly, but-
- SPSpeaker
The transfers and all these things.
- AZAlex Zhang
Yeah.
- SPSpeaker
Yeah.
- AZAlex Zhang
But I, I, I will say, like, it's not clear, though, like, i- in a lot of cases, if it's even possible to hit this theoretical number, if that makes sense. Like, this is assuming, like, perfect overlapping and transfer of data and, and, like, there's maybe some bottleneck that you can't get around. But often, the kernels are not even close. Like, that we write are not nearly close enough to this number to be, like, meaningful at all.
- SPSpeaker
Yeah. And is it speed that matters? Do you also care about, uh, obviously memory, which-
- AZAlex Zhang
Mm
- SPSpeaker
... which feeds into speed? Do you care about power consumption? So one of my, uh, one of our top pods of the year was Jeff Dean, who was like, "Actually, I just tracked the microjoules or, like, the nanojoules, picojoules."
- AZAlex Zhang
Yeah, pico.
- SPSpeaker
Okay.
- AZAlex Zhang
It's often picojoules is what I think.
- SPSpeaker
Picojoules.
- AZAlex Zhang
Yeah.
- SPSpeaker
Uh, do you care about that?
- AZAlex Zhang
So I don't. [laughs]
- SPSpeaker
Yeah. [laughs]
- AZAlex Zhang
I guess maybe I'm not, I'm not as-
- SPSpeaker
But everything here is speed, right? Like-
- AZAlex Zhang
Everything here is speed
- SPSpeaker
... no- nobody's counting picojoules.
- AZAlex Zhang
Um, but I... There's a caveat here, which is I think, like- There is speed in the context of a single kernel, and there's speed in the context of a larger problem, like maybe the N10 model. Because, like, one thing to consider, and this is why it's important to talk about what, what speed of light is referring to, because in these cases for the kernels, like we always start with everything in, in, uh, like HBM for example, right? But you can imagine that like an end-to-end kern- like an end-to-end model, what you might wanna do between two layers is like you might sacrifice the speed of the first operation to keep things in the cache for the second operation. And like these are things that, like, you can't really get out of, in isolation with like these kinds of kernels. And, you know, people call this like the fusion pro- or like the-
- SWswyx
Mega kernel
- AZAlex Zhang
... kernel fusion problem, yeah, or like mega kernel stuff. And it generally only applies like when you are, like, memory bound in, in, in most cases. But this is something that, like, also there, there is this question of like, as these models get better, like should we just be generating me- like mega kernels? Is that like what we want? Um-
- SWswyx
What's, what's your take?
- AZAlex Zhang
I think that this is really difficult because you need the data to do this, and I think, like, I have yet to see an example in the wild of like we bootstrap the ability to solve a very difficult class of problems without any examples. Um, and I think, like the other reason why I think maybe this isn't that interesting is that at the level of an individual kernel, A, like they're not that... they're not as complex. But B, you're somewhat confident that there's not as much structure in a single kernel. But like in a mega kernel, like I would be more inclined to believe that like a compiler would be better here. Like some compiler over like higher level ops makes sense. Because in general, like actually I think mega kernels are very... Like the pieces are very composable of like the individual kernels. There's some areas where you might wanna do like weird fusions and everything, but in general I think these are cases that, like, a compiler can probably handle. And there is a company that's working on this from, from what I understand, that has given some talks on GPU Mode as well.
- SWswyx
Yeah, I wanna basically cluster all the GPU Mode discussions here-
- 15:16 – 21:24
Research Taste and Taking Big Bets
- SWswyx
involved in this-
- AZAlex Zhang
Yep
- SWswyx
... field.
- AZAlex Zhang
Yeah. I will give maybe the in- the background story of like I am actually involved in a lot of benchmarks, or I used to be, uh, maybe prior to my PhD. It started because I was at Princeton. Uh, I worked with the SWE-bench team there.
- SWswyx
John, Carlos-
- AZAlex Zhang
John, Carlos-
- SWswyx
Ophir
- AZAlex Zhang
... and Ophir. They're all great. Like I-
- SWswyx
Karthik
- AZAlex Zhang
... I love them.
- SWswyx
Yeah.
- AZAlex Zhang
Yeah. Uh-
- SWswyx
There, there's basically this, like... I think people don't understand how mi- I mean, how much benchmarks come from the same group-
- AZAlex Zhang
Yeah
- SWswyx
... at Princeton. [laughs]
- AZAlex Zhang
[laughs] Yeah. It is crazy.
- SWswyx
Do you know Shenyu?
- AZAlex Zhang
Yes. Yeah, yeah, yeah.
- SWswyx
Uh, we had him on the pod before. Now, now he's like running Tensor.
- AZAlex Zhang
Yeah, now he's like a-
- SWswyx
[laughs]
- AZAlex Zhang
... he's like a superstar. Uh, when I, when I, I met him... So he, uh, he was advising my friend Michael, Michael Tang, who is now at Anthropic, but they worked together a lot. We were like the two undergrads in Karthik's lab. I... And then some others joined later as well. But yeah, Shenyu's great. I, I did not know he was like such a superstar until like later on, like after I left, but-
- SWswyx
Yeah
- AZAlex Zhang
... uh-
- SWswyx
I mean, like, like, okay, so there are very few PhD students. Like yours is, yours is like the next one. Like once a year we fe- feature someone, like who is like basically entire PhD, uh, has been like on target.
- AZAlex Zhang
Mm-hmm.
- SWswyx
There's not that many of them. Shenyu was like clearly one of them. And uh, you know, Ja- Jack Morris is another one. And like, you know, we talked... Before the show, we talked about research taste.
- AZAlex Zhang
Mm-hmm.
- SWswyx
Right? Like somehow some grad students just have a very blessed career where like yep, mostly like yep, yep, yep, yep, yep, this is like going to stick around relevant, everyone should know this.
- AZAlex Zhang
Yeah.
- 21:24 – 30:59
GEV and Rethinking the Language Model
- AZAlex Zhang
project.
- SWswyx
It's-
- AZAlex Zhang
I want to bring this up because this also happened with RLMs and, uh, it's happened with many other works, like things get overhyped, right? To, to an extent, like something gets overhyped and then people are like, "Why is this overhyped?" Like, "This is trivial. This is stupid." And I saw the same thing with GEV because I think the release was like-- I mean, there was this whole thing about like academics or they're not an academic group, but like people have to do branding and they have to like kind of market their research, and so like I understand. But I think there was a lot of discourse about GEV just being like something we've known for years, and I think it's kind of missing the point of like why is such a system so interesting? It's why is it not just some stupid MLP classifier that like we've, we've been doing, you know, back in our intro ML classes or something. I think what's really interesting about GEV is that it kind of opens up this question of are language models correct? Like in the form that they're in, can we consider a different design space other than text-to-text? 'Cause what, what they're doing is they're basically saying like, "I will take advantage of this language model backbone. Like I know it captures a lot of information about language, but I'm going to change the output space of the model to give you a trade-off, which is I will do very, very fast inference over like if you have some prior about this problem." Like let's say I only need to make a binary classification. Am I gonna ask my language model to do this and pay like a 400x cost? Like no. It's, it, it's like silly, right? Um, and I think for the longest time because the labs are the only places that control, you're never gonna use something other than like GPT- f- uh, GPT-6 or Fable because they're the best models. But because of that, like people have gotten kind of accustomed to this idea that a language model is just a autoregressive decoder. Like we have accepted this. And I think when RLMs came out, it was the same thing. Like one of the comments I-- like a, a very frequent criticism I got was like, "This is not a language model," or like, "When I look at this, like I thought it was a new architecture, but it's actually not." And my response to that is like, well, a language model is just modeling language. It doesn't have to be this transformer decoder, you know? Uh, and GEV is really, really interesting in that like we now have a new thing to tune, which is like what is the output space and how does this affect inference latency? Uh, and I think we can actually start asking this about various parts of the language model itself. Um, we are seeing this too with like loop transformers. It's a similar idea of a lot of the attention around it was like this is a silly idea. Like why-- who cares about this? But it's like it is a simple idea, but it's actually, it opens up a whole new set of questions that I think like, especially if you're a PhD student, these are the things that you wanna answer. Because I, I think it's like we don't know- For GEV, for example, we don't know how far we can take this. Uh, for loop transformers, we also don't know how far we can take this. What if you loop only a subset of the model? Um, what if you route to like only, uh, like you have some router to different parts of the model? Like, can you mimic what you do in a harness inside of the model? What can you bridge between a harness choice and a model architecture choice? These are all questions I think that open up with works like this, and that's like really exciting. GEV in particular, when I saw it, I was like, "This is actually really useful for RLMs." Like, I think it's, it's-- it makes sense 'cause the, the biggest bottleneck in RLMs or swarms or, or systems like these is they're slow. When you do multiple language model calls all the time, you're not distributing your compute correctly because like maybe there's something trivial that you just want a simple model to do, but you can't do it because your language model is just this, this bulky thing, you know. So I'm very excited. I think, I think we will start to see new types of models emerge beyond just the bog standard frontier model, and that is like there's so many things that you can do with these like new trade-offs. Um-
- SWswyx
Also shout out Thinky with their interaction models.
- AZAlex Zhang
Yes. Yeah, yeah. Another great example.
- SWswyx
Yeah, so like basically try to break the paradigm from sequence to sequence decoder only and just l-literally do anything else.
- AZAlex Zhang
Yeah. [laughs]
- AFAlessio Fanelli
I mean, I think there's, there's a level of if you're trying to compete, you're not gonna compete with a frontier lab doing an autoregressive-
- AZAlex Zhang
No
- AFAlessio Fanelli
... decoder on like the amount of compute scaling resources they have even Thinky will not... You know, okay, there may be one of the handful that can, but you're not really gonna do much in that at least.
- AZAlex Zhang
I don't know too much about Thinky, or I don't wanna say anything either. But it's like if their strategy is just to replicate OpenAI or Anthropic, like that's a horrible strategy-
- AFAlessio Fanelli
[laughs]
- AZAlex Zhang
... because, well, because like I mean, th-they just don't have... Like you, you kinda just have to think of it in terms of like what advantage do you have? And if, if you're going to use the same setup, I, I'm sure they're not. But it's like if you're going to do the same setup, like you're basically competing on the things you can control, which is data and compute, and obviously they cannot compete with the frontier labs on that. So yeah, it makes sense that like if you are a Neo lab, like actually, I don't know if you would consider them to be a Neo lab, but I guess like-
- AFAlessio Fanelli
Yeah. That's why they're in there
- SWswyx
They're in there
- AZAlex Zhang
... I guess they're kind of a weird one. Yeah.
- SWswyx
They're in that list. They ship Inkling.
- AZAlex Zhang
Yeah.
- SWswyx
Like they, they count.
- AZAlex Zhang
Anything other than OpenAI or Anthropic, maybe like f- Meta and, and, and GDM, like you just... You gotta do something else, you know. Like it's just, it's the sad reality, but I think I, I actually think it's a good thing. I'm very happy that like scale works, and these companies will just keep doing it because it opens up like potentially new players, like if they uncover something like really, really interesting because I sort of have my doubts that this is like seriously going on at frontier labs 'cause it's like, like why would you do that? You know, like why would you take the risk of allocating a large amount of compute to new bets when the old bet already works? So-
- AFAlessio Fanelli
And I think that's what spins off a lot of Neo labs, right?
- AZAlex Zhang
Yeah.
- AFAlessio Fanelli
You have a side bet, and you don't get compute-
- AZAlex Zhang
Yeah
- AFAlessio Fanelli
... and you're like, "Okay, I'll go, I'll go do it."
- AZAlex Zhang
Exactly.
- AFAlessio Fanelli
And, you know, your example of the potential upside is something like GEV, which is x hundred times cheaper, you know, comes out, and maybe it is language model. This-
- AZAlex Zhang
Yeah
- AFAlessio Fanelli
... in this case, it's just different.
- AZAlex Zhang
Yeah, yeah, yeah.
- 30:59 – 32:57
Video Game Agents and the Harness Problem
- AZAlex Zhang
Oh, yeah, yeah.
- AFAlessio Fanelli
I mean, th- you know, you, you did do a benchmark on any-
- AZAlex Zhang
So, um-
- AFAlessio Fanelli
... any fun gameplay.
- AZAlex Zhang
Yeah, yeah, yeah. Um, I guess these numbers are very outdated.
- AFAlessio Fanelli
Yeah. [laughs]
- AZAlex Zhang
'Cause a lot of the models are, are, are very different. And I've seen actually people run... There, there are some folks out there that are running newer models on these games, which is really cool. I guess the general premise of this benchmark was we just want to see if vision language models are, are good enough at just, like plugging into games with the latency constraint included. 'Cause this, this actually, I came out with this right after Claude Plays Pokemon came out.
- AFAlessio Fanelli
Mm-hmm.
- AZAlex Zhang
Um, so this was like two years ago, which I guess is like ancient now. But I think what's really, really cool about this suite of tasks, it's very diverse in terms of what games they are. And also, I think most of the games are games that people know-
- AFAlessio Fanelli
Yeah
- AZAlex Zhang
... or like have seen before. I saw, yeah, GEV playing Doom. I will say I don't think-- I think they were just playing like really simple levels and stuff. But honestly, like most models still can't really do... Or I don't actually think any models can solve these games very meaningfully. Like, there are some games that they can. I think I, I've seen Astra be able to solve the Kirby game. And we also, for this benchmark, we intentionally designed a really minimal harness. And I, I will get to this point about harnesses because I think there's a whole conversation to be had about like what is the value of a harness? Like, what is even the purpose of a harness for a model? But in general, like I think it's, uh... Yeah, I, I hope to see very quickly or very soon like all of these games beaten by newer models.
- AFAlessio Fanelli
Yeah, it's interesting, like the old, old Claude Plays Pokemon, they like read state from-
- AZAlex Zhang
Mm-hmm
- AFAlessio Fanelli
... RAM and saw what tiles are walkable and whatnot. We did a podcast with them-
- AZAlex Zhang
Mm-hmm
- AFAlessio Fanelli
... a long, long time ago.
- AZAlex Zhang
Gotcha.
- AFAlessio Fanelli
Yeah.
- AZAlex Zhang
Gotcha.
- AFAlessio Fanelli
Just fun, just-
- SWswyx
Yeah, I mean, and, and s- and similar, right? Like GEV doesn't have vision-
- AFAlessio Fanelli
Yes
- SWswyx
... so you have to kind of feed in, uh-
- AFAlessio Fanelli
Yeah, yeah
- SWswyx
... these like, uh, game state and all these things. I mean, let's go right into the harness stuff-
- AZAlex Zhang
Awesome
- SWswyx
... because you me- you brought it up. Language model harnesses are compositional generalizers.
- AZAlex Zhang
Yes.
- AFAlessio Fanelli
Struggled
- 32:57 – 38:38
Why Claude Code, Codex, and Pi Are So Similar
- AFAlessio Fanelli
to read that one. [laughs]
- AZAlex Zhang
Explain-- Yes. Okay, okay. So I have been a little unsatisfied maybe with, with how people think about harnesses, because people compare like, "Oh, like I love Claude Code, I love Codex, like I love Pi." Like, "No, I love o1 my Pi. I love Prime Agent." To be honest, I think all of them are the same. Most of the design decisions or like the design choices around these harnesses are the same. Maybe Prime Agent is a little bit different 'cause it's like inherently an RLM. But in general, like I think we can be a lot more creative with harnesses. And what I mean by that is if we think about this from the perspective of what exactly is the harness doing for the model, well, basically, when you're trying to solve a problem and you want to use a language model to solve it, like a very difficult task, one thing that we have discovered is that next token prediction is a really, really awkward form to do a lot of these tasks. So for example, take SWE-bench. When you're navigating a code base, like are you going to be able to figure out how to do all of this with a single language model call? Like you just say, "Solve code," or like, "Solve my query over this code base"? No. So we rely on a harness to help you do these kinds of things, and I think what, what is interesting is like a harness is a very, very opinionated program over how you want a language model to be form fit over a problem. And I bring this up because when we think about the-- what a harness is doing, we should really think about like what choices in the harness let the language model solve this task, and can I actually just have a language model that just does this? Because a harness, if, if you think about it, now that looped transformers are a thing, I think what's really interesting about it is you can model a looped transformer in some ways like with a harness as well, right? You're just looping over the mo- Now, you can say like, "Oh, I'm not decoding," so it's like a little bit different. But in general, we, for whatever reason, have stuck with the same model architecture choice forever. And I-- there's many arguments for why. But clearly, like we are now training models around harness rollouts, and so there is this very awkward way of doing training over harnesses, which is that we train a language model to act within a harness, but like now it's like a really long, maybe like multiple agent rollout that we're doing, and there's like really awkward hacky ways of doing this. So what this blog talks about is like, well, one way you can think about what is going on here is if the harness is basically helping the model solve a particular task, can different harness design choices actually do something a little bit more meaningful beyond just here are some tool calls that will help you? Here is a way to grep through your code base. And so this actually-- the idea for this blog came with the RLM idea as well. We just didn't package it that way, and I, I think this is actually true. There are many other ideas around RLMs that like we will be coming out with, but were all there from the beginning. I mean, these are all design decisions around... Um, I think w-with what, what is-- what I like about the RLM is that there were many, many, many iterations and versions of different abstractions that I was interested in doing, and ultimately, the RLM made the most sense. But there's a lot of reasons that aren't public as to why that's the case. Um, I mean, you, you will see. So in this example, one of the things that we see with an RLM is that if you sufficiently offload context and write-- ask the model to write code over that context, you get this really, really weird but useful property, which is that when the model recognizes during training how to solve a task, it turns out that the solution to many tasks is very similar across like tasks where you, you don't even... Like, it's not even that clear to you that the solutions are similar. So in this example, we have like a retrieval task, and we have like a, an aggregation task, and they're very, very different query. Like, the domain is just completely different. And when you train a regular language model over these two tasks, the trajectories look very different. And so what you're relying when you-- like let's say you use Pi or Claude Code or something, which is not in this blog, but we do have these results. You'll find that, like, these harnesses distinguish too much between these problems, even though the solutions are the same. And so one thing that we find when training RLMs is that, like, when it sees these problems, it's the same. And the reason it sees these problems as the same is the sub-agent sees different problems, but the sub-agent is solving an easier sub-task, and so you're confident it's smart enough to do it. But for the base overall strategy, they end up looking the same. And so when you train on the left task, for example, it can immediately solve the right task. And so if you go down to like the plots that we have, one thing you'll kind of observe is that the-- as you just naively train your RLM on these tasks, they naturally learn to generalize, for example, to longer tasks because the strategy is basically the same. You're just modifying like a length variable. And this actually also holds for tasks that are different, and they're-- it's not even-- they're not different across length. They're completely different tasks, math tasks versus writing tasks. But the solution, the like meta high-level solution is the same. And so when you train the RLM on one of them, it generalizes this behavior to the second one. And there's no magic here. I, I guess maybe that's the thing that I wanna kind of stress. Like-
- AFAlessio Fanelli
How, how would you kind of verbalize what they're learning?
- 38:38 – 46:20
Harnesses as Compositional Generalizers
- AZAlex Zhang
Mm-hmm.
- AFAlessio Fanelli
So I think in here you say you train it on short tasks, they generalize to stuff eight to 30 times longer.
- AZAlex Zhang
Yeah.
- AFAlessio Fanelli
They're learning how to solve these type of problems, or what's the, what's the core thing they're actually learning?
- AZAlex Zhang
Yeah. They're learning how to solve these types of problems at a certain length, and it turns out that when you take the strategy that they learned, it is directly transferable to the longer length. Like they're-
- AFAlessio Fanelli
Yeah
- AZAlex Zhang
... effectively the same program. And this is something that is really exciting because what this kind of implies is that when you have a corpus of data or environments that you train your model on, the hope is that, like, A, you can train on less environments and generalize to more than what existing models can, can do through ni- or harnesses through existing kind of like naive training. But B, also you still wanna use all the data that you have. So when you train on these tasks, like, hopefully it generalizes to a wider class of, of problems. And why this is also even more exciting, at least in the context of RLMs or any recursively calling system, is this argument holds inductively. So, like, I'll give you an example, 'cause I said I claimed that competitive programming and GPU optimization use very similar skill sets. The model can-- the harness potentially, I mean, you might have to nudge it a certain way, but it can learn that, like, okay, how I'm gonna go about solving this GPU programming task is very similar to what I learned for competitive programming. So I'm gonna list out a set of solutions. I'll, I'll, like, spawn sub-agents to list out promising solutions, and then I'll, like, write this loop to go through and check these solutions, maybe evolve them, and, like, evolve them against a verifier. And between these two tasks, this looks the same. But what the sub-agents are doing are maybe, like, unique and something, something, like, different. But even what the sub-agents are solving might actually also be of the same form, right? 'Cause it's like a-- it's a recursive argument. And so what I'm trying to get at with this whole blog post is just that, like, we should rethink what the role of the harness is because harnesses can actually greatly increase the generalization capability of your model and the amount of data that it's given. And this is not exclusive to RLMs. I think there is a wide class of harnesses that are yet to be discovered that actually can yield similar properties. And to extend this argument a little bit further, I think what you can also extrapolate from this is, like, if I look at an RLM, what are the components of an RLM that are actually necessary? And can I actually just directly train a model to do this? Can I train a model to act as an RLM implicitly in its forward pass? It's a really weird thing to think about because, like, you know, you might say, like, "Ah, code is non-differentiable," blah, blah, blah, blah, blah. But there are many approximations of, of this behavior that we will start to uncover. And I think, like, we will see beyond just, like, I'm gonna design a new coding harness that uses a special form of compaction or something. I think we can be a lot more creative here. Like, there's, there's so much we can do with these language models that I think we are just not doing. And I'm, like, very excited about this 'cause I think, like, I think we can get serious gains from very, very opinionated and good harness design that lends itself better to scale. And what I mean by this is, like, you know, the RLM, for example, is a very primitive inductive bias. Like, there's nothing super special about the design other than the fact that it's very different than what we currently do. But this may potentially scale much better with, like, the data and the environments that we have available to us.
- AFAlessio Fanelli
I guess the, you know, opposite thing that people would probably ask is current harnesses-
- AZAlex Zhang
Mm-hmm
- AFAlessio Fanelli
... are very generalized towards coding, which people see works for a lot of domains. Claude Code is being used for design-
- AZAlex Zhang
Yep
- AFAlessio Fanelli
... presentations, everything. Um, Muse, Spark, GroqBot-
- AZAlex Zhang
Mm
- AFAlessio Fanelli
... these are very simple, non-opinionated harnesses that are good at code, and that is also scaling out. Um, what's the example of how we
- SPSpeaker
improve those, I guess.
- AZAlex Zhang
Hmm. Yeah. Um, let me, let me bring up another paper, uh, which came out very recently. It's like the harness tax paper. I think it's by Arena. I really like this paper because it puts forward a prior that I had, which is basically that like- [laughs] Most-
- SPSpeaker
As it confirms a prior. [laughs]
- AZAlex Zhang
Yeah, like most harness choices don't matter-
- SPSpeaker
Yeah
- AZAlex Zhang
... because all of these harnesses are the same. But I-I will say, like, GroqBot, for example, is actually quite different, I think, from my understanding, than, than how some of these harn- other harnesses have been designed, and I like that a lot. Um, and I think it-it's clear from here at least that, like, I'm pretty sure Anthropic or OpenAI are exclusively training on their harnesses. They're probably not training on their competitor's harn-- I mean, I'd assume not because I-I don't know why they would do that. But-
- SPSpeaker
But I mean, you know, th-this is a thing you see in open models, right? Like, Qwen is really good at using open code.
- AZAlex Zhang
Yes.
- SPSpeaker
They need to train in harnesses. Old Gemas were notoriously bad at this.
- AZAlex Zhang
Yeah.
- SPSpeaker
Uh, models are good, but you need to train in a harness.
- AZAlex Zhang
I-I think, though, as models get smarter or, like, as they get better, this distinction becomes, uh, not that important in the sense that, like, if you take Astra and you put it inside of open code, like, it's not gonna go crazy because I think it's, like, just sufficiently good. And so I say this because the only benefits between these different harnesses is just cost for the most part. And, and I think, like, what I'm getting at here is if you plug these models into RLMs, though, they're not that good still. They're okay. And I think it's mainly because the types-- like, the class of harness that we are training around is this class of harness, this, like, Pi loop, this, like... I, I like to call it trajectory as a prompt, which just means, like, you keep the whole trajectory of the rollout as the context that, that, that your main model is, is using. Even if you use sub-agents, it's still, like, kinda of this form. And I think we're going to-- if we want to explore new harnesses, there needs to be teams that are dedicated to actually running meaningful experiments over, like, scaling out new harnesses, like maybe post-train scaling out on different harness designs. Like, I, I think we can actually get very meaningful knowledge or gains from doing this kind of thing, uh, whether it's an RLM or whether it's something different, and that's exciting 'cause I think, for example, if you train a lot on... Fable for a long time was the best model for RLMs because they had dynamic workflows, and it was pretty obvious that, like, this was a capability that was somewhat trained in. Even if the model was still, like, a little dumb, like in the RLM harness, it still worked a lot better than, than other models did. Uh, Astra is now also, like, good enough at doing these things. But-
- SPSpeaker
Wait, is this, is this where we see that Fable is the best for RLMs, or is there, is there some other-
- AZAlex Zhang
Oh, no, these are all internal results-
- SPSpeaker
Nice. Ah
- AZAlex Zhang
... I guess. Yeah, I don't, I don't have them-
- 46:20 – 53:57
RLMs Explained
- AZAlex Zhang
This one. Okay.
- SPSpeaker
Okay.
- AZAlex Zhang
This is, like, the best-- I wish I had this in the original paper. An RLM is basically just a harness design where the only tool in the harness is code, uh, which is this programmatic subagent calling thing where it has the option to call itself as a tool, and it has other tools, but all of these things are functions in code, and the context that it's dealing with is always stored in some memory inside of this code environment. So this could be a file system. Like, this could be like... I'll give you an example, Prime Agent. The trajectory of Prime Agent, like the context, even the-- when you compact into all these things, is stored on disk. So the model can always reference its original context, even if it's compacted, and all of its tools are run inside of, let's say, like a Python REPL or a Bash REPL. And so this-- it's, like, this very primitive abstraction. And I would say, like, where most harnesses differ is, A, context offloading is not done that, like that. I mean, if you look at Prime Agent, by the way, Prime Agent does context offloading but not all the way. So, like, it still maintains the standard Claude code Codex loop of, like, trajectory as a prompt where you compact, but it has the additional kind of like the context is offloaded, and it only has... The unique point of, of Prime Agent is that the only tool is, is IPython. So this is, like, the very kind of generic abstraction around, like, RLMs. Yeah.
- SPSpeaker
Concretely, what's the core thing RLMs are trying to solve?
- AZAlex Zhang
Yeah.
- SPSpeaker
At one point, I think when it first came out, it was context-
- AZAlex Zhang
Mm-hmm
- SPSpeaker
... but help us with the question.
- AZAlex Zhang
Yeah. So the original problem was harnesses have a really bad time dealing with long context. They typically were used to only really do them for, like, specific things like code. Like, you know, they could deal with your code base 'cause it was trained on it. But now it's more around what this blog is talking about, which is compositionality and the fact that, like... I think harnesses, we want to have language model systems that have much more control over the actions they make at every step. And what I mean by this is tool calls are very limited because you have to invoke them every turn. Like, you have to invoke tool A, then tool B, then tool C, and there's no central context that you can kind of draw back from. And RLMs are specifically like designed around composition and having like a central context that you can always draw from. Um, and this context is like designed around the existing language models. Another like very similar example actually in design is like agent swarms, for example, the, the hugging face incident. Like these agent swarms have like a message board that they, that they learn to communicate over, and this message board in some sense is the shared context that they like act over. And RLMs basically say that like the best way to communicate through this is in code. Like you write the code to do this, and it's, it's because these models are so good at writing code. Um, like we wanna take advantage of, of that fact.
- SWswyx
And then for, for this compositional thing, up to and including generating your own harness, you know, specific for the task.
- AZAlex Zhang
Exactly. Exactly.
- SWswyx
Right?
- AZAlex Zhang
I think we will start to see that if you go up to, uh, this figure. Okay. So we, we talk about this idea of locally in-distribution tasks for a harness, and it is like a, an idea on top of like in-distribution tasks. When we think about language models, an in-distribution task is just a task where like the prompt is something that the model has either seen before or like has seen some version of it. Most harnesses work like the one on the right, which is they keep appending the trajectory as a prompt, and so eventually, unless you're Anthropic or OpenAI and you train on like kind of these like user trajectories, most of these things end up being out of distribution for the most part. But locally in distribution is basically the compositional argument of if an RLM breaks down its computation into like, um, kind of a meta harness of sorts or like a program that involves sub-agents that like look at a local problem, every individual language model call over the course of this task is in distribution even if the entire task itself is out of distribution, and this is a very desirable property, I think for, for obvious reasons. Like if every task is in distribution for each individual language model call, you will probably get to the right answer. Um, so...
- SWswyx
[laughs] I mean, the, the logical limit of RLMs is RLLMs where like you're not just-- you don't just write the harness, you also train a custom model.
- AZAlex Zhang
Exactly.
- SWswyx
You collect data, e-everything.
- AZAlex Zhang
Yeah.
- SWswyx
Like it's a fully automated AI researcher inside of your harness.
- AZAlex Zhang
Yeah. We will see where the training of RLMs goes. I will say as an academic, I am not working on this at MIT, or at least in the scaled sense because I [laughs] I can't afford to. Uh, but there are companies out there that are working on this. I think Prime and Select is very clearly working on this, and it's very cool. Like I'm, I'm very excited to see maybe we'll observe, I don't know, but maybe we'll observe better like post-training scaling laws with when you train around a smart harness. Maybe we'll even see smarter harnesses that come out and like they, they work better around these kinds of principles.
- SWswyx
What is a smarter harness? Like that doesn't mean anything. You just said they're all the same. [laughs]
- AZAlex Zhang
No. What-- More of what I mean is like Claude Code, Codex, Pi, et cetera, are all the same in that like when you break down the, the logic of the harness, it's like virtually the same thing.
- SWswyx
Yeah. Two calls in the loop or-
- AZAlex Zhang
Yeah
- SWswyx
... whatever.
- AZAlex Zhang
But with RLMs and, and with other harness abstractions, it looks very different. And this is where I think you really distinguish-- I mean, it's, it's in the same way that like I think with language model architecture choices, a lot of architecture choices end up kind of looking the same when you like scale it out or like it... The, the differences end up being like somewhat minor in terms of-- I mean, for a lab it's not minor, but, you know, maybe one model converges better than the other one like slightly. But in general, like if you were-- So for example, pre-training scaling laws only hold because the architecture choices we have are somewhat stable right now. But if you were to completely change the architecture, pre-training scaling laws probably don't hold. Or like these kinds of-- this like power law is gonna look very different. And it's like the same thing with harnesses. Like I think all the harnesses we have right now for the most part roughly look the same, but there are some exceptions to this I think that are, that are coming out.
- SWswyx
I was gonna say like actually one of the things that I've been more interested by, you know, talking about PhD students who take big risks is that people have been-- people are also pursuing the other side, which is pre-training scaling laws don't hold if you change data. Uh, right now it's just raw unstructured text, uh, corpus of internet. What if you had a better data representation to train on? Then, then yes, your scaling law would change as well.
- AZAlex Zhang
Yeah.
- SWswyx
So there's architecture, there's data, and you know, whatever else, uh, you can think about. Well, so I just wanna get back to, to this. I-- it, it all makes sense. It's, it's very interesting how you sort of recurse up and down the stack. [laughs]
- AZAlex Zhang
[laughs]
- SWswyx
From like very conceptual to like, nope, like well this is, this is where we are today.
- 53:57 – 59:37
Prime Agent and Persistent Subagents
- SWswyx
and get your thoughts on each.
- AZAlex Zhang
Yeah. Yeah. Uh, I got involved with Prime, uh, after they, they released a blog post, by the way, not affiliated with me at all, about like how they believed RLMs were kind of the, the future. And I, I had a friend that was working there, GPU Mode, Matei. Like we got in touch and I think I agreed with a lot of the researchers there and like what they believed about harness design. Like I, I, I was, I was very impressed I think that like they understood the purpose of the RLM paper which is not necessarily just to say that like, you know, we're solving long context tasks but actually like we want more opinionated harness designs.
- SWswyx
Yeah. There, there's always like the result of the paper that you choose to highlight-
- AZAlex Zhang
Yes
- SWswyx
... versus the actual point.
- AZAlex Zhang
Yes. As-
- SWswyx
[laughs]
- AZAlex Zhang
And I, I, I would love to talk about like the incentives of academia and like things around like why it's kind of flawed and all the issues and we will get back to that.
- SWswyx
[laughs]
- AZAlex Zhang
Yeah. So, so anyways, um, you know, I, I, I, I love the guys at Prime so we kind of had been- After, after we decided to work together, we decided to look into training in RLM and also build this kind of RLM harness and kinda see where we can take it. That is how, like, Prime Agent came about, and I think the reception for Prime Agent has been pretty good. Like, the one thing I was worried about with Prime Agent is that none of them, at least at the time when we were building it, none of the models were that good at doing RLM stuff. So this was like pre-Fable, pre-Astra. Um-
- SWswyx
I guess, I think to take a step back, can you explain what Prime Agent is, how it's different than-
- AZAlex Zhang
Yeah
- SWswyx
... a traditional, you know, Claude Code, what people would expect harness?
- AZAlex Zhang
Yes. So Prime Agent, I think I, I mentioned this a little bit earlier-
- SWswyx
Yeah, there's the diagram. Yeah
- AZAlex Zhang
... is, is basically, uh, it, it is a, a harness on top of Pi, like Pi Mono, uh, which is Pi Mono for context is like the, like-
- SWswyx
Core agents
- AZAlex Zhang
... a minimalist-
- SWswyx
Yeah
- AZAlex Zhang
... yeah, like harness. I, I use Pi as the reference for everything because I think all other harnesses are basically just Pi. Um, but it is Pi except we explicitly restrict IPython to be the only tool that's available to it. Every other tool gets loaded in as like a Python module, um, or like a, like a bash kind of script that it can, it can run. So it uses the core RLM abstraction on top of Pi, and then it also has this continual harness thing, which is, uh, Seth, he's another PhD student. This is a thing that he used to get language model harnesses to play games. Like, so he worked a lot with Joel, uh, who is the, like, Gemini plays Pokemon guy. And continual harness is also, by the way, very simple. I, I quite like it. It, it basically is this design, uh, principle around, like, what parts of the harness can you let the harness itself modify? There are certain pieces that, like, you'll let it modify its own skills, the subagents available to it, uh, what the system prompt to the model is. And continual harness is available basically as a tool inside of the IPython kernel. And so that's what Prime Agent is, like, how, what it's designed around. Everything else in Prime Agent is like-
- SWswyx
Standard
- AZAlex Zhang
... standard.
- SWswyx
Standard.
- AZAlex Zhang
Right? Yeah. I think what is-- what I, I, I really liked about it, and we got kinda lucky, is that, like, a lot of the new frontier models actually work really well inside. And actually even a lot of the open source models work really well, at least some of the newer ones. And there is another thing in Prime Agent I should highlight, which is that, like, we have a very particular agent-to-agent communication system or like, uh, like framework, which is because RLMs tend to spawn many subagents, we want a way for subagents to communicate with maybe the root or with each other. And so there are some design decisions around, like, what each subagent is allowed to talk to, how it does it. Again, everything is in code, so it, it writes the code to do this kind of communication, which I think is really, really cool. And then there's, I guess, persistent subagents is another thing that was kind of added, which is subagents, they can last beyond, like, the standard runtime of, of the actual, like, original agent. And you can go into that subagent, you can prompt it more, um, like you have more visibility and flexibility into what is kind of going on.
- SWswyx
This is my number-one pain with Codex right now. They, their subagents are just very ephemeral-
- AZAlex Zhang
[chuckles]
- SWswyx
... and they, they actively discourage you from using it for long-running things.
- AZAlex Zhang
Yeah. Yeah, yeah, which, which I think it makes sense. Yeah, I-
- SWswyx
So I mean the, the trick is just externalize to a file system, right?
- AZAlex Zhang
Yes. Yeah.
- 59:37 – 1:02:26
RLMs in the Wild
- AZAlex Zhang
later.
- SWswyx
Yeah.
- AZAlex Zhang
Yeah.
- SWswyx
Okay, I wanna close out a little bit more of your research, and then we can, we can-
- AZAlex Zhang
Cool. Yep
- SWswyx
... uh, start pointing it out. Uh, since you released RLM, a lot of excitement about it. Any secondary third-party work that you wanna shout out as like that you guys should take a look at this?
- AZAlex Zhang
Oh, yeah, yeah, yeah. So, uh, Harvey, the legal AI company, released a blog post, not affiliated at all. But they post-trained an RLM on their, like, legal work, which often involves a lot of, like, sifting through documents and kind of looking through, like, a variety of, uh, specific information that maybe is not so easy to retrieve with like a pure retrieval system, and they show like really, really good results. It's very exciting. I mean, I, I, I was, uh, I was shocked that, that, that they worked on this. They did not tell me, so when this came out, I was like, "Oh, that's awesome." So there's this one I think is super, super cool, and, and what they're doing there, I think this is a collaboration with Base10-
- SWswyx
Yes
- AZAlex Zhang
... by the way, as well.
- SWswyx
This was Base10.
- AZAlex Zhang
Headlong, which is law-- the law institute's kind of, uh, it is their like persistently running harness. Um, it's very cool-
- SWswyx
Oh, they renamed it? They used to call it something else.
- AZAlex Zhang
It was like Auto-
- SWswyx
Terminus.
- AZAlex Zhang
Ter- yeah, I know, I know.
- SWswyx
Yeah.
- AZAlex Zhang
They, they, they've, they've gone through... Yeah.
- SWswyx
All right.
- AZAlex Zhang
Um, so this is Andy Konwinski's big project. Um, it's super, super cool. I love Andy. I don't want to downplay what they're doing because they're using the RLM abstraction, but they're doing something much cooler than the RLM, uh, which is like they, they have a system that kind of what they call like thinks persistently. So- Even when you don't query it, it has a way to, um, think through problems that it has in its context.
- SWswyx
Oh, so it's just like a always-on type thing.
- AZAlex Zhang
It's like an always-on thing-
- SWswyx
Yeah
- AZAlex Zhang
... but it's, like, not that expensive. Like, they, they, they control the, the, the token costs, um-
- SWswyx
Mm-hmm
- AZAlex Zhang
... to make sure it's not, like, burning through all your, all your credits.
- SWswyx
Okay.
- AZAlex Zhang
This is super cool. I'm trying to think. There, there are many... Actually, if you go to the RLM, uh, GitHub page, there's [chuckles] a bunch of things I've linked, uh, below. There's a ton of really, really cool kind of things that people have been doing. Ax is another really cool one that I think it's just by this one guy. It's like a harness around DS Pi and RLMs. DS Pi also has an RLM. Oh, the last thing I'll shout out is on ArcAGI 3, I believe, there were a lot of harnesses on their, like, Kaggle competition, like the official one, not, not the, like, public Prime Intellect one that... or, like, what people have evaluated on. They all, like, claim to use or they reference, like, like Tufa for example, some form or some inspired form of the RLM abstraction in their harness, which is really, really cool. I think it's, um... This is where-- I mean, this is exactly the setting where you would see a lot of benefits from composition and, and, and using code and combining, like, neurosymbolic systems with, with AI. And so-
- SWswyx
Yeah
- AZAlex Zhang
... yeah, very, very cool.
- SWswyx
We love a good neurosymbolic
- 1:02:26 – 1:09:22
OpenAI Swarms and the Future of Language Models
- SWswyx
reference.
- AZAlex Zhang
Yeah. [laughs]
- SWswyx
Uh, you're also, uh, mentioning ArcAGI 3, OpenAI comes out and says, "We're at 99.9% on this."
- AZAlex Zhang
Yep.
- SWswyx
Uh, they also say, "We, we solved Navier-Stokes. We just threw a model at it."
- AZAlex Zhang
Mm-hmm.
- SWswyx
Uh, there's some debate around whether or not it's just model.
- AZAlex Zhang
Mm-hmm.
- SWswyx
Are they using an RLM? Do you know?
- AZAlex Zhang
I, I mean, I, I, I would guess probably not. Unless you say, like... I'll be, I'll be careful here because, you know, people debate what is an RLM, what is not an RLM. It's somewhat clear that what they used is some kind of swarm of agents with a shared, some shared context, like some shared file system. And, like, this is very much in the spirit of RLM stuff, but I think there's a, there's a lot of, like, you know, more clever things that they did that's not maybe related to, to the RLM itself. I agree a little bit with the idea that, like, a harness is not that necessary for what they did. The way that I would put this is that I think a model like a GPC6 Astra-type thing is technically smart enough, conditioned on the right information, to come up with a proof for these very, very difficult problems. Now, how you get to that information is a giant question mark. And in, in their case, it probably came down to, like, a very long search over, like, many of these subage- or many of these, like, agents in the swarm and, and maybe also, like, researchers con- I'm, I'm actually not sure about this part, but putting in, like, their kind of intuition as to, like, what you should explore and things like this. And ultimately, like, this produced some information that some agent was able to take to finish the proof. And so in that sense, like, I think was the harness that important? No. And I, I think what this is pointing at is, like, the specific details of a harness do not really matter, and I think that's also what that, uh, what the harness tasks paper is, is pointing at, which is that, like, beyond the user's feeling of the harness, realistically all that matters is just, like, how are you composing these agents in a meaningful way to get to the final answer? And maybe that's what, like, swarms and all these things are, are, are really about. And so from my POV at least, if we, if we start to think about, like, for user use cases, what do we want out of harnesses and, and, and things like this? Like, we want to take the good parts out of these, like the Claude codes, the Codexes, like the stream that people like to see. But, like, under the hood, whatever is running can be some really weird, complicated swarm of agents that, like, ultimately come up with an answer. The user doesn't wanna see that, though, obviously, right? Like, it's, it's not legible information. And so I, you know, I, th-this was another kind of thing in the spirit of RLMs, like recursive language model. It sounds like it's a language model and, but it's not a language model architecture. But the reason for this is, like, I think we will start to see in the future probably one day, and I, I wrote a blog about this, what we think of as a language model, like the thing that we query might actually be like a swarm or like a scaffold or like some weird harness design that scales very well, but the user just doesn't see it. Ultimately, all the user sees is some front-end version of this harness. And yeah, I, I, I, I think it's a relatively safe bet at least to make that this is what we will see.
- SWswyx
Yeah.
- AZAlex Zhang
Um, and this maybe goes back to the limitations of the base transformer. Like, you know, obviously, if you just took a base transformer and you said, like, "Solve Navier-Stokes," or something-
- SWswyx
Mm-hmm
- AZAlex Zhang
... it's not gonna do it. Like yeah, we, we all know this is not what's gonna happen. But yeah, I think this is maybe the more interesting part. Uh, and may-maybe the claims around, like, you know, did the harness matter is, is more around this, of like just arbitrarily pointing models, like, or agents at a growing kind of context of information maybe is just enough to solve very difficult problems. Um, that I can buy.
- SWswyx
Uh, well, there's a lot in there. I do wanna say, uh, the, the, the amount that OpenAI spent is semi-public. It's, uh, 10,000 agents in 88 hours.
- AZAlex Zhang
Oh, yeah.
- SWswyx
Uh, 130 billion to- output tokens, which is estimated to be about $40 million in public pricing.
- AZAlex Zhang
Mm. Surprisingly, actually, like less than I thought.
- SWswyx
Yeah, not that much.
- AZAlex Zhang
Yeah. [laughs]
- SWswyx
Yeah.
- AFAlessio Fanelli
130 billion output for the final, but as you said, there was a lot of context being passed around.
- AZAlex Zhang
Mm-hmm.
- AFAlessio Fanelli
It's more than double that in just the total agent messages being sent.
- AZAlex Zhang
Yeah.
- SWswyx
Yeah, yeah.
- AZAlex Zhang
Yeah, yeah.
- SWswyx
I think, uh, one thing I was honored to bring up also was Cursor as far as, like, swarm stuff is concerned.
- AZAlex Zhang
Mm.
- SWswyx
Uh, this is slightly older, uh, meaning February, which is ancient.
- 1:09:22 – 1:17:48
Open-Endedness and Sakana AI
- AZAlex Zhang
cetera. So...
- SWswyx
Yeah. Uh, and then, uh, just a quick check in case you have opinions on this. Have you looked much into open-endedness as a general category of problems?
- AZAlex Zhang
Mm. Oh, oh-
- SWswyx
Meaning no, no prompts. Just go.
- AZAlex Zhang
A, a little bit. Um, so I, I was at Sakana for a summer, uh, right after or I guess right before my PhD, and, and that's something that they work on a lot there. And I think there's a lot of people even at like Recursive superintell- there's many of them now.
- SWswyx
Yes, we just had Richard Socher on.
- AZAlex Zhang
Oh, yes. Yeah. So, so like Richard's company and then, and then, and then also... Actually, wait, that might be the, it might be the same company. I, I, I don't remember. Is, is Tim Rocktäschel also-
- SWswyx
Yeah, yeah.
- AZAlex Zhang
Okay. Yes, that company.
- SWswyx
He's the main co-founder. He's the head of open-endedness for Google.
- AZAlex Zhang
Yeah. Yeah.
- SWswyx
Um, yeah.
- AZAlex Zhang
So I think with open-endedness problems, like I view them as somewhat similar to even like unsolved math problems. Maybe that's a weird way of, of, of framing it, but like I think a lot of the techniques in terms of like how people s- like approach them are kind of the same. Like evolutionary search is like very similar to just launching swarms of agents and hoping that like they come up with like an interesting s- And this is what, what like AlphaEvolve and some of these other works did, uh, like a year or two ago. But I think what maybe is, is not, uh... And, and I, I'm not sure if, if this is what you were alluding to, but I think what's not super clear in open-endedness style search problems is do we frame all of them as basically like an unsolved, very difficult problem where like, where the objective is clear? If that's not the case, I still don't know yet entirely what the value of it is. Uh, maybe you have other opinions. Like I, I, I don't have too many opinions on this. But at least from my time at Sakana, like I, I got the sense that like we ultimately still kind of wanna approach things the way that, like, say, OpenAI approached Navier-Stokes. We want... We- there's still a lot of nudging in certain directions that we want to have to, like, get to the point where we have something interesting.
- SWswyx
My version of it is like, um, maybe it's a split between basic science and applied science. Basic science, you're researching for researching's sake.
- AZAlex Zhang
Mm-hmm.
- SWswyx
You just wanna understand things better. Uh, I have no idea if, like, there'll be any application at all whatsoever, but, uh, I mean, that, um... And then applied, you have a goal.
- AZAlex Zhang
Mm-hmm.
- SWswyx
You're, you're trying to minimize loss in some way, you know? Uh, and so, uh, w- what I really, uh, think, you know, in terms of like the, the big bets that people have, uh, what if there was no prompt? Like [laughs]
- AZAlex Zhang
I see. I see.
- SWswyx
You know what I mean?
- AFAlessio Fanelli
You just pick domain and let it run.
- SWswyx
Like, you just, like you just spawned-
- AZAlex Zhang
You just let it run
- SWswyx
... in this like swarm of things, and you're like, "Hey, what's up, guys? Like, what are you g- what's everyone working on?"
- AZAlex Zhang
Yeah.
- SWswyx
And like you just decide-
- AZAlex Zhang
I see
- SWswyx
... like this is an interesting problem.
- AZAlex Zhang
The biggest issue that like... And may- maybe there's a clear path to this, but when I was there at least, the biggest issue that people had with open-endedness was like how do you ultimately choose at the end? How do you pick out the interesting stuff? Because when there is no... Like maybe the agent comes up with a goal, but in a lot of cases, like what they... And they have something called Fuu, I think, which is like a-
- SWswyx
Mm-hmm
- 1:17:48 – 1:22:02
Kimi vs. OpenAI Agent Swarms
- SWswyx
Yeah.
- AZAlex Zhang
Like-
- AFAlessio Fanelli
Any thoughts on Kimi agent swarms?
- AZAlex Zhang
Yes. Um, one thing I will say is whatever OpenAI is doing with their agent swarm is, like, clearly the right thing to do. You have to kinda think about it this way. Like, nothing, especially without, like, a very, very smart harness design, which I, I don't, I don't think anyone really has so far, nothing comes so easily for free. For example, like, the agent swarm design is not something you can just take for granted. Like, it's not like GPT-6 Astra is just super smart, and then it just got agent swarms running well. They clearly trained-- I mean, the, the hugging face incident was them training a system to be like a swarm. Uh, and I, I think, like, clearly they've done something really well to the point where you can throw forty million dollars and solve an unsolved problem. And I, I think with the Kimi agent swarm thing, like, at least from when I read it, it just came off as, like, this is interesting, but I don't actually know whether or not this can solve anything novel.
- SWswyx
Yeah, they just-- they were like, "It makes spreadsheets for you." [laughing]
- AZAlex Zhang
They kinda were just like, "Yeah," like, you know, "here is a, here is a swarm that, like, kinda does stuff," and it's cool. Um, but, uh- And I, I will say same thing with dynamic workflows. I actually kind of think the dynamic workflows release was sort of a flop. I, I don't know how, how you guys feel about it, but I think it's like, uh, my understanding is like it's not used that often or, or like-
- SWswyx
It's just very expensive.
- AZAlex Zhang
It's too expensive and like-
- SWswyx
It's ultra code. It's basically like, like, you know, take over my bank.
- AZAlex Zhang
It, it doesn't a- I've tried it and like it doesn't act in the way that like... Again, I, I, I'm not, I'm not the biggest OpenAI like, you know, stan or something, but I think whatever they did was very impressive. Like, they somehow managed to get a way for this, this swarm to actually act-
- SWswyx
I see, I see, I see
- AZAlex Zhang
... towards, towards a goal and yeah.
- SWswyx
I see.
- AZAlex Zhang
It's very difficult.
- SWswyx
I see. So, so like efficiency of s- of the multi-agent swarm is the objective function here.
- AZAlex Zhang
Yeah.
- SWswyx
Right?
- AZAlex Zhang
Yeah. Yeah.
- SWswyx
Like how much of this is slop?
- AZAlex Zhang
We-
- SWswyx
Like this is a lot of slop.
- AZAlex Zhang
Yeah. I think so.
- SWswyx
OpenAI is less slop.
- AZAlex Zhang
I, I, we, we take for granted what it means for a swarm to converge to an answer.
- SWswyx
Yeah.
- AZAlex Zhang
Um, it's just like, it's not, not something we can take for granted.
- SWswyx
Yeah.
- AZAlex Zhang
Yeah.
- SWswyx
Uh, we've, we've done one pod with Noam Brown and like his-
- AZAlex Zhang
Oh, yes. I remember his.
- 1:22:02 – 1:30:15
Capability Overhang and Speculative Tool Calling
- AZAlex Zhang
Genius for a little bit.
- SWswyx
Yeah.
- AZAlex Zhang
I... Uh, the only comment I'll say on speculative PTC is that it's a really simple idea. It's almost like obvious that this should be done and like there's not much more to talk about it. Like, I think it's just like you should just use it for, for like coding. Like anything with programmatic agent calling, like RLMs or Code Act, like yeah. It's like the, it's like a no-brainer.
- AFAlessio Fanelli
What's the... You know, for, for people that haven't read it-
- AZAlex Zhang
Yeah
- AFAlessio Fanelli
... what's the one-liner?
- AZAlex Zhang
The, the-
- AFAlessio Fanelli
I like the-
- AZAlex Zhang
... simple thing is when the model is, uh, writing its code or like even as, like after it finishes writing the code, a lot of tools tend to be like sequential or like you have to wait on them, so you should just launch them in advance. Like if, if you're able to JIT compile this code, you can probably figure out like even though it's like kind of variables and stuff, like you, you can figure out like how to fix it up.
- SWswyx
Yeah. Statically analyze.
- AZAlex Zhang
Yeah. Yeah. So-
- AFAlessio Fanelli
Speculation.
- AZAlex Zhang
Yeah. There is, there is this, uh, someone pointed to me some actually like academics have, especially PL, like programming languages people, have some like very kind of cool ways of doing this and so like, you know, at some point, maybe I'll, I'll, I'll like work on this.
- AFAlessio Fanelli
I guess I-
- SWswyx
Mos-mostly you have to change language because if you are in JavaScript Python, you can't do this.
- AZAlex Zhang
Yes. Yeah, yeah, yeah.
- SWswyx
So like, uh, Haskell, yes. Um, um, what's, what's the, ugh, what's the normal one that's, that's not Haskell?
- AZAlex Zhang
Lisp.
- SWswyx
Uh, Lisp, uh, or Caml.
- AZAlex Zhang
Oh, Caml. Oh, yeah, yeah, yeah.
- SWswyx
Any, any functional language-
- AZAlex Zhang
Yeah, yeah, yeah
- SWswyx
... you can actually like pipeline this.
- AZAlex Zhang
Yeah.
- SWswyx
Um, so Effect TS if you wanna do TypeScript.
- AZAlex Zhang
Yeah.
- SWswyx
Okay. We can switch over to-
- AZAlex Zhang
Yeah, yeah
- SWswyx
... uh-
- AFAlessio Fanelli
I like this diagram
- 1:30:15 – 1:43:24
Neuralese, Future Research, and AI for Science
- AZAlex Zhang
Um-
- SWswyx
PTX?
- AZAlex Zhang
Let's say a, a mix of English and Python, and I only say this because the capability of a model is somewhat a reflection of what we train them on. So we still want like, like... Yeah, I don't really buy the, the binary argument.
- SWswyx
[laughs]
- AZAlex Zhang
Um, I guess I, like, I, I understand, but it's like, uh, like-
- SWswyx
Yeah. You, you wanna-
- AZAlex Zhang
I don't know
- SWswyx
... you wanna model the world in some way. I think the, you know-
- AZAlex Zhang
Yeah
- SWswyx
... uh, one thing I'll, I'll bring up is always, which I always do in this kind of conversation is Sapir-Whorf-
- AZAlex Zhang
Mm-hmm
- SWswyx
... uh, which is you, if you choose English, you will have locked into however long English has been around, which is, let's say, five hundred years, uh, which is not that long. Like, actually-
- AZAlex Zhang
Mm-hmm
- SWswyx
... uh, you know, like what you-- well, the language that you speak constrains how you think.
- AZAlex Zhang
Mm-hmm.
- SWswyx
And if you learn a different language, uh, for example, uh, someone, uh, in Chinese, we don't have tenses. Um, I don't know if you-
- AZAlex Zhang
I, I-
- SWswyx
I, I actually didn't know that.
- AZAlex Zhang
Oh, oh.
- SWswyx
And I speak Chinese.
- AZAlex Zhang
Oh, oh. I, I, I did know that, but my, my Chinese is not great.
- SWswyx
Okay.
- AZAlex Zhang
But yeah.
- SWswyx
Yeah. Or like, uh, in, let's say in Japanese or, uh, no, uh, I forget what l-language it is. Like, in, in Korean, everyone you speak to, yeah, you have to, like, acknowledge social status.
- AZAlex Zhang
Yeah, yeah, yeah, yeah.
- SWswyx
That is a different dimension than gender. Right? Like, it just like, it just influences everything you do. Um, yeah, I when, when I take Ling 101, um, apparently there's a, there's a, there's a language in Africa where, like, there's a vegetable gender. Uh, yeah, right? Like-
- AZAlex Zhang
Hmm
- SWswyx
... it's just like n- you have... Or like Eskimos know no word for snow or whatever. Like, anyway, so like the, the language that you adopt affects your thinking and if you-
- AZAlex Zhang
Hmm
- SWswyx
... choose to output your train of thought in English, you are biasing towards whatever English solves.
Episode duration: 1:43:24
In the uListen app, the transcript lights up word by word as you listen. Get uListen
Transcript of episode kog7mwsDqnk
