Building the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil

Jerry Tworek 0:00 If I play football, for example, it looks very, very closely to reinforcement learning.

Pat Grady 0:04 Mm-hmm.

Jerry Tworek 0:04 I kick a ball a lot of times, and every time I adjust it a little bit and I see if it roughly matches what I, what I wanted, and either there, there is some self-reinforcement happening. When I learn mathematics, it's very different type of thinking.

Pat Grady 0:17 Mm-hmm.

Jerry Tworek 0:17 It's, it's like reading ver- about hard concepts and thinking about them very deeply inside my head until things click, and I-

Pat Grady 0:24 Mm-hmm

Jerry Tworek 0:24 ... until I have them connected. And both of those, in some way, are learning from experience, they are just very different. We probably are spending the most compute than ever on learning from experience, but lear- reinforcement learning is, is not the end of learning from experience, and there will be better approaches that researchers will be coming up in the coming years on how to, how to use that data.

Sonya Huang 0:43 [upbeat music] Jerry, Rohan, thank you so much for joining us today. The two of you are the founders of Core Automation, one of the hottest neo labs in San Francisco right now. And before starting Core Automation, you led some of the most important research projects of the AI era. Uh, Jerry, you were VP at OpenAI, where you worked among- amongst other things, on running the Strawberry and Reasonings teams. Uh, and Rohan, you were one of two of the four pre-training leads at Gemini, and before that, led a lot of the fundamental AI research at Google Brain, and were the fix-it guy across Google, um, and then at Anthropic. And so between the two of you, you've seen more than a fair, your fair share of, uh, what the world looks like in terms of doing frontier research. And so I'm very, very excited to dig in. Um, let's start

Sonya Huang 1:46 with you, Jerry. You tweeted a very spicy take recently, "The first step to replacing Transformers is appreciating deeply how far they were able to carry us." Is that a eulogy for the Transformer? What does that mean?

Jerry Tworek 1:57 Thank you very much for inviting us here, Sonya. I feel like a lot of my interviews these days is explaining my tweets and what did I-

Sonya Huang 2:06 [laughs]

Jerry Tworek 2:06 ... what did I mean. But appreciating Transformer means, like, understanding what it does well.

Pat Grady 2:12 Yes.

Jerry Tworek 2:12 When you are not solving the problems that it is solving well, you have to focus on its, on its weaknesses.

Pat Grady 2:17 Yes.

Jerry Tworek 2:17 You have to, you have to understand good parts and bad parts. And it's very easy in a lot of the work what people are doing in architectures, is trying to make Transformers cheaper-

Pat Grady 2:28 Yeah

Jerry Tworek 2:28 ... and trying to make Transformer more efficient. I, I very rarely see people thinking about how do we make Transformers more powerful-

Pat Grady 2:35 Yeah

Jerry Tworek 2:35 ... trying to do more expressive. But, but, like, seeing someone weak parts and seeing someone strong parts are almost, almost, almost the same thing. It's just, just understanding the shape of Transformer a little

Jerry Tworek 2:44 bit more. What I think right now we are in this stage, we got really, really good at training really, really big models. We mastered two algorithms. We mastered pre-training at a large scale, and we mastered reinforcement learning at a large scale. And I'm asking myself a lot, what is next in machine learning? And I think at this moment, what the bottleneck is to better models and to smarter systems is the architecture itself. It is this moment to revisit, uh, the train we've been riding for the last six years of trying to add more and more parameters to essentially two of the same operations, which is MoE and, and attention. And when I am, when I'm thinking about it, like, what's, what, like, where we are today and what we are doing, I am thinking a lot about, about, like, what Codex and what Cloud Code are doing for us. And I'm really, really appreciative of those systems and of the coding and of the workflow automation and of the systems of the products that we have today, that we essentially have built over those, those six years of scaling. And I think, I think this is, this is the first step of thinking, like, what is the... If we wanna work on the replacement, we need to, like, see where we are, what problems we have solved, to, like, start seeing what the next stage is, what problems we haven't solved yet. What, what, what kind of are we missing? And this is, this is kind of whenever, whenever I use Codex and I am successful at a task, I also start thinking, "Why didn't I, why didn't I try to push that thing harder?" Whenever, whenever I come to work, there are a lot of things I do with Codex, but I still come to work. I still ask it to do certain things for me, and I'm always asking myself, "Why, why am I even needed there? Why is, uh, Core Automation is name and its concept is we wanna, we wanna be automating tasks, and why, why, why are those things not yet automated? Why, why, why is not Codex doing everything for me?" And this is, this is the question of, like, the research, where we want to go. And with that research, I'm trying to think what kind of models, what kind of, what kind of systems do we need? What kind of qualities do we need? What that, that, that we don't, we don't have today, and that's, that's what I'm thinking a lot

Jerry Tworek 4:54 these days.

Pat Grady 4:54 And you have this starting premise of the architecture is the issue, which I think is a contrarian point of view.

Jerry Tworek 5:00 It is.

Pat Grady 5:00 So what, what led you to that point of view? What did you see that made you think the architecture was the issue?

Jerry Tworek 5:05 It's fundamentally what is the issue. It's, it, uh, it, like, comes back from the, from, from the previous implication. What, what I think is the issue is that the models are being trained in the lab and are being deployed in the, in the real world. That is the, that, that is the fundamental, fundamental tension that is, that is there. And, mm, like- A bit of, a bit of my disappointment can, comes from my, my, my personal

Jerry Tworek 5:32 story. Whenever we were starting, uh, the research and progress on scaling up reinforcement learning at OpenAI, I basically believed that scaling up reinforcement learning is a necessary stepping stone on a path to AGI. Uh, since, since, since I started like, uh, working, working at OpenAI, and I was always reinforcement learning maximalist. I was always believe this is what we need to focus on, this is what we need to do. I've seen LLMs being scaled up to, for, to, to higher and higher levels, from GPT-3 to GPT-4, and we're still doing very little RL.

Sonya Huang 6:05 Mm-hmm.

Jerry Tworek 6:05 And I had this, this internal belief that the moment we start scaling up RL, ev- we'll, we'll, we'll solve everything. We'll, we'll, we'll be able to solve, solve all the problems. And we eventually started solve, uh, scaling up RL. I was, I was just in there. I was, I was in the center of it. I was thinking, "Here we are. If you ask Jerry in 2024, when do we get AGI? I would say 2025 will be, will be that year. This, uh, this, this is where we solve everything." And I saw us training model after model. This model was getting better and better. All the benchmark scores were going up. And w- did we also solve all the real world tasks at that moment? Unfortunately, unfortunately not. And we, we, we still have work. And I realized there was this bit of, mm, distinction as all the benchmarks that we are evaluating our models, they were essentially the same thing as we were training the models on.

Sonya Huang 7:01 Mm.

Jerry Tworek 7:01 Like, all the evals and training si- tasks are the same sides of the coin.

Sonya Huang 7:05 Mm-hmm.

Jerry Tworek 7:05 But the real world distribution and real world task is much messier, much murkier, much more, much more different. Our, our training data didn't really replicate the real world use cases, and despite us b- basically maximizing all the tasks, if you see, ask anyone training models, "Hey, what is one of your main issues? I don't have hard enough tasks. I don't, I don't have what to-

Sonya Huang 7:26 Mm

Jerry Tworek 7:26 ... what to train our model on," yet we are still not covering the, the entirety of the, of the real world distribution.

Jerry Tworek 7:32 From that, my conclusion is we need to have models that learn at test time. We need to have models that learn with users on their data, on their real world task, on the real world distribution. And there, when you ask why, why, why, why, why, why don't we have that today? Why, why, why are transformers not, not learning any- anywhere? And there are essentially two types of learning that we could be doing at test time. We could be doing in-context learning-

Sonya Huang 7:59 Mm-hmm

Jerry Tworek 7:59 ... essentially of transformers, which is... It, it, it doesn't have fundamental problems of catastrophic forgetting. It doesn't have that issue, and it's pretty data efficient, so that, that is great. But it's not very scalable. We only can have so much of it. It is limited, and it has some more, even more of mechanical limitations of, uh, what actually are you doing when you, when you build context. But maybe, maybe we can, we can come back to it later. But we ha- we have in-context learning, which is very limited and very, very small amount of data. Whenever I'm using Codex, roughly around 20 minutes of usage, I need to, I need to compact it and move, and move it afterwards, which is not that much, not that much data. If we have all, all we can learn is for, for 20 minutes, it's not that much. And the second thing is fine-tuning. We could try to continuously fine-tune our models, but then those have the issues of catastrophic forgetting. We have issues of very low data efficiency, and neither those are very solvable.

Sonya Huang 8:58 Mm.

Jerry Tworek 8:58 Neither of those are very easy to, uh, to find ways people have been trying. If those were easy to solve, someone would already solve it. So my personal belief is we need to find an algorithm that we can, we can meta-learn, that we can express on the architectural layer that can represent how does, how does learning look like.

Sonya Huang 9:21 Mm.

Jerry Tworek 9:21 How does learning look like that can work on much, much longer horizons?

Sonya Huang 9:24 Do you expect the architecture will look transformer-like? 'Cause my, my, my understanding from the, from the chief seats is that, you know, OpenAI have been trying to scale up reinforcement learning for a long time, and it wasn't until the transformer came about that it seemed like there was an even kind of scalable prior on the world upon which to even scale RL. And so how do you even go about trying to think about scaling up this, this new regime?

Jerry Tworek 9:50 Yeah. It's a, it's a great question.

Jerry Tworek 9:52 I think those two things happened at the same time. But if anything that, that happened there was, was mostly about, about economics because technically you can scale up LSTMs.

Sonya Huang 10:03 Mm.

Jerry Tworek 10:04 Just no one, no one, no one really, really dared to go in that direction, and they did scale much, much more poorly. They're, they're, they're, they're scaling... In a scaling loss paper, there is, uh, presented a comparison of LSTMs and transformers.

Sonya Huang 10:18 Yeah.

Jerry Tworek 10:18 And fundamentally, the scaling of, of transformers was better. There is a world where we never invented transformers, and we would be scaling LSTMs, and we, we would be having some models. But because they would be much more expensive to train and much less impressive as a product, we would have, we'd have just as worse experience, and maybe no one would be able to convince people to spend as many dollars training th- that, those gigantic LSTMs because we wouldn't get a market return. The, the, the majestic thing about transformer, which goes back to, like, why, why, why do we have to appreciate transformers so deeply, is that transformers are economically valuable. They're... Training them, the cost of training them is lower than the revenue that they, that they generate, which is just magic of machine learning. It's not, not guaranteed by itself. But for, for LSTMs, it probably wouldn't be that way, which, which, which made it happen. But you can, in many ways you can scale most of the architectures. Uh, I th- I think a lot of reasons why people didn't scale things before was because researchers before OpenAI had a lot of reluctance to scaling. It was o- often seen as unscientific, and, uh, research in algorithm was, was providing how do we become more and more efficient, how do we, for the same compute budget, get better and better result. And it was, it was a bit of a contrarian bet by OpenAI at that moment to try to say, "Hey, we don't care about be- better and better algorithms. We care about more and more scalable algorithms."

Rohan Anil 11:40 Mm.

Jerry Tworek 11:41 And how do we, how do we pour more and more compute and get better, uh, get better results, which OpenAI was criticized repeatedly by many people in the community for a, for a long time. But thanks to that we have, we have the models that we, that we have today. Uh, and I think there are tons of architectures that can be scaled up, and I am part of the Core Automations mission, and our, our belief is that a lot of architectural research happened at too small scale for too long time.

Rohan Anil 12:08 Mm.

Jerry Tworek 12:09 A lot of people are trying to say, "Hey, let's, let's try to first try our arc- architecture on a, on a, on a, on a small dataset, on a sm- in a small compute regime, and then, and then, and then see where, where, where, where it scales only after, after it prove itself." But for example, when you do work on reinforcement learning, you know that to get to any interesting results, you only n- you need certain level of compute to even see the capabilities in the model. Reinforcement learning needs a baseline of, of, of ability to only, only start working. So, so where I am coming from, probably there are many architectures that need a baseline of compute to even start doing anything, anything interesting and anything useful.

Rohan Anil 12:47 Can I ask you then maybe a touchy question?

Jerry Tworek 12:49 Please do.

Rohan Anil 12:49 If you need a baseline of compute, that sounds like a job that would be well-served inside of a big research lab. Why start a company to go do this? [laughs]

Jerry Tworek 12:58 It's a, it's a, it's a great question, and it's, I think in ma- in many ways it's likely a timing thing, timing issue. Market is right now in a, in a very specific place where the biggest and the most successful labs by coincidence or by fate are probably in the most competitive market fight ever right now-

Rohan Anil 13:22 Mm

Jerry Tworek 13:22 ... which makes them not very keen on trying different paths-

Rohan Anil 13:27 Mm

Jerry Tworek 13:27 ... trying alternatives. If transformer is profitable, and if you can spend more efforts and more resources scaling transformer to win in the next quarter, it's very hard to put at least a lot of attention and a lot of energy to work on something that will, that will maybe better or maybe, or maybe will, will redefine the field in a year or two. So, so I f- I think the big- biggest labs, and I talked to basically all of them, don't, don't have that much interest in-

Rohan Anil 13:55 Mm

Jerry Tworek 13:55 ... trying the alternatives to, to transformer. And the labs that are not the biggest are doing whatever they can to do what the, what the, what the, the most successful labs are doing, and everyone is trying to train the same, the same coding agent. If you look at-

Rohan Anil 14:07 Mm

Jerry Tworek 14:07 ... the, uh, last week's, uh, releases, everyone, everyone is, is, is trying to release a coding agent right now. And, and I think we need different paths and different, and different approaches here. So, so that's what, the, the niche in the ecosystem we are trying to, trying to fill in.

Sonya Huang 14:24 Hmm. And Rohan, you were at G- you were at Brain when the transformer was invented.

Rohan Anil 14:27 Yep.

Sonya Huang 14:27 Do, do you agree with Jerry's eulogy for the transformer?

Rohan Anil 14:30 Yes. Um, in some sense, like, um, once the... First when the transformers, Ashish, Noam, and others came up with it, I, I had like work- I worked on my work on online distillation around the same time we presented it at the same internal research conference. It wasn't a big deal internally. There was only a few people who actually got it.

Jerry Tworek 14:49 Hmm.

Rohan Anil 14:49 A lot of peo- people were like, "Oh, it's, it's like, yeah, it's another work." And people were finding ways to... And it was also very focused on, at least the original work was very focused on a real problem, which is translation. So they solved, like they beat LSTM on translation, and it took OpenAI... I mean, internally at Google, there was definitely, like Noam and a few others were definitely interested in scaling language models. I think it is until GPT-2 and GPT-3 that we saw the benefit of transformers working quite well. At least the way I think about architecture is how do we spend computation, and transformer is one way, very efficient way to spend computation. But now that I look at the industry, it's, it's a lot of our computation is inference time and spending it on tokens. Let me ask this question, like if I want to optimize for a better architecture, I want to look at both pre-training and RL together, and I would like to find architectures that spend computation much better than current chain of thought token generation. Uh, in, at a, like to give a much better overview, it's I think of like pre-training as build the transformer with certain context length, and RL comes in and is like, "Well, that's not sufficient. I need more computation. Let me do it via adding one token at a time." This is quite inefficient from, uh, like inference perspective. You're doing one token at a time, so most of the solutions have been finding to do better ways of speculative decoding, so it's like a band-aid to a problem that we've picked something that's can only generate one token at a time, so autoregressive decoding. Uh, there is problems with the transformer in terms of, uh, how do we spend the computation. Uh, for the longest time, I think, uh, most of the world was training very large dense models, and it took, uh, like the industry like roughly two to three years to get to refine the architecture to what we now take for granted was not obvious to a lot of people, sparsity and mixtures of experts, and getting good training efficiencies with them, right? So then you can ask like what's wrong with the transformer? Well, it's, if the computational depth is poor, you, how do we increase computational depth? And just posing that question opens up like 20 new directions on how we can modify the mechanism to incorporate it. So, um- I, I see like to do work like this, it takes time, and usually like fundamental research in the past have taken like five, six years to land into industry, and it's largely from organizational knowing that it is important. This is the bet. Like just like, uh, Jerry had the inner belief that RL is needed. Absolutely did not have that belief at Google. I was a pre-training maximalist, pre-train a bigger model-

Jerry Tworek 17:31 That's why you guys are a good fit.

Rohan Anil 17:32 Right? Like-

Pat Grady 17:32 Things are, uh, quite well.

Rohan Anil 17:33 Right? And then so that inner belief. And second is you need your architecture to run efficiently on hardware.

Pat Grady 17:39 Mm-hmm.

Rohan Anil 17:40 A theoretically optimal architecture is not useful to anyone. It is something when it comes into practice. So you need, uh, the research inception to getting it productionized and getting kernels and everything written, the end-to-end loop. And there's only a few places right now which have integrated teams doing that, and I think we have built the team in a way that puts every-- like the experts together-

Jerry Tworek 18:04 Mm.

Rohan Anil 18:04 ... not in different silos that like we are accelerating on having everybody look at the problem holistically from end to end. So that's like where I'm quite bullish. That's why I'm here. Um, the c-current mechanisms are quite poor, and it will t- if, if you leave it to the world, I am afraid that it'll take us like a much longer time horizon before we replace the transformer, and I think a lot of folks are already complaining a lot on token costs, and, uh, that seems like as someone-

Jerry Tworek 18:33 We're not complaining.

Rohan Anil 18:34 Uh, [laughs] I mean, in terms of like... Yeah, exactly. I come from the Google mindset where we had to like serve billions of people. So like finding more efficient architectures that fit, have a, like a deadline on latency and the number of tokens that you can serve. So like when I look at that, like the amount of like the world that can use like frontier tech is very little, and s- we-- someone or some group has to, uh, like accelerate and make this better, and we are taking that shot at doing that. The current technology just scaled up is still only relevant to, uh, like a subset of humans.

Jerry Tworek 19:09 Yeah.

Rohan Anil 19:09 And this is

Rohan Anil 19:11 the bet we're making.

Pat Grady 19:11 So one of the things I heard you say was the problem with transformers is the computational depth is poor.

Rohan Anil 19:17 Yeah.

Pat Grady 19:17 If that's the crux of the issue, tell us, what does that mean? Why is that the case? How do you fix it?

Rohan Anil 19:24 I can, uh, give you like, uh, one insight, like most transformers that we train are quite shallow. It's at most like 100 layers deep. Uh, depth is like, it's called deep learning 'cause you wanted deeper representations. There has been experiments on going into depth, but no one has actually shown, uh, us learning extremely deep representations. Chain-of-thought reasoning and RL to do chain of thought on by model itself is one way to increase computational depth, because every token you add, you add like one more pathway. And there has been, uh, so then you can get out of like this bottleneck that the pre-trained architecture has set you up on. You can only do, uh, layer, number of layers times, uh, sequence length. Now you can increase the sequence length, and you get much stronger, uh, results. You can do inference time scaling. Now, uh, the issue with inference time scaling is that models now have to produce more tokens to get, uh-

Jerry Tworek 20:19 Mm-hmm

Rohan Anil 20:19 ... better results, and that's very one token at a time. And from this, you can see like you can directly address many of these things, and this is like a subset of work that we are looking at, right?

Pat Grady 20:30 Mm.

Rohan Anil 20:30 Making this much more efficient.

Pat Grady 20:32 What's your forecast for the transformer-based architecture? If it's not the end state, how far can it get us? When do we start to see it topping out?

Jerry Tworek 20:41 I, I, I think it all, it all comes back to what we are, what we are training transformers for and what we can, what we can do with them. We're doing pre-training, which is very good at distilling all the knowledge from the internet-

Pat Grady 20:54 Yeah

Jerry Tworek 20:54 ... into transformers, and then we can RL them through, which is we basically can bake all the workflows that we want into a transformer. So what, what, what where transformer is like capped out is we have all the knowledge of, of, of humanity in the model together with the relationships and how do they, how do they work together, how, how they can be combined. And basically, any task that we have training data for, we can, we can put into the model, and we-- this can be gigantic model training with a lot of compute on all the, on all the, all the data in the world. And then if we ever stop training that model, what would, what would happen? Like the, the question worth asking often and thinking about transformer, what would happen if OpenAI and Anthropic stopped training new models, and we got, we got the transformer we have today and say, "This is, this is it. This is, this is, this is the best, the best model we have." Uh, months pass, uh, year, year, years pass, and the model is getting less and less useful. Maybe the lab really like recorded all of every human on Earth, what they were doing and what their, what their tasks were and their environments and put them, put them in the model, put them in a, in a learning environment. But then the, the, then, then what happens if the, if anything of that changes? If there are new events in the world, if those new events have new relationships between them, if there are, there are new types of tasks, if there are new code bases, new tools to use, transformers are getting a lot of their usefulness and all value through the things that are, that are valuable have to be present in, in, in, in, in training. And when, when, when, when they are not, they, they, they, they suffer. There, there's some ability to adapt, but it's not very, not very big and not very, not very flexible. So in my mind, this is, this is kind of the level where the, where the transformers, uh, top, which in many ways what I think is a, is a tool to use for us. If we-

Pat Grady 22:45 Mm-hmm

Jerry Tworek 22:45 ... if we kind of know if there, there's a human who knows the limitations of a transformer, they can, they can schedule that model. They can, they can write a prompt of what is, what is the task that you want.

Pat Grady 22:55 Mm-hmm.

Jerry Tworek 22:55 And by doing the training we are doing, you can, you can like get very successful in that. In any task, the model fails, you get added to your training data-

Pat Grady 23:03 Mm-hmm

Jerry Tworek 23:03 ... and the model, and the model can, can succeed. But that, that loop is, has to go through the lab training, training the model for you. And if the model that, that fundamentally needs to be trained in the lab, like how much do you think of it that this is, this is the goal or, or, or you would want to be able to update the model somehow-

Pat Grady 23:20 Mm

Jerry Tworek 23:20 ... not having to, to go back there?

Sonya Huang 23:22 Have you read the, uh, Rich Sutton and David Silver have this paper, the, The Age of Experience. Have, have you read it? I'm curious how much you agree or if you have any different opinions. Where do your opinions diverge?

Jerry Tworek 23:32 Reinforcement learning is not a particularly new approach, particularly, particularly new thing to do. So, uh, in some way, in some way age of experience, I think, I think always has been there, and people have been criticizing a bit pre-training, because pre-training very clearly is, is, is the other way of, of looking at the models, which is like we have static data. That data is mostly generated by others. Although I, I per- have this personal view that pre-training today is largely distilling other models, older models into the new model because most of the tokens in the internet are, are coming from, fro- from AI. But there, there's clearly pre-training, which is, which is behavioral cloning, which is mimicry, which is, which is compression of, of, of, of, of internet data. But reinforcement learning is not something that, that people haven't been thinking and people haven't been, haven't been doing. Reinforcement learning was used to solve backgammon back in the day. Then it used to solve Go, StarCraft, Dota, um, to solving programming right now. And every, and, and every time it comes down to model writing its own experience and learning, learning from that experience. But it is very clear, and what I think is interesting and what I think is still, still perplexing to people, that reinforcement learning is not really the only way to learn from experience. And there, there will be more and there will be, there will be a little bit more of, I think you, you can call it algorithmic, but essentially innovation of how we learn from experience just because, just because reinforcement learning is only one way to do it, it's a mathematical formulation. And, uh, es- especially right now, how we are using it, it really likes those parallel rollouts for variance reduction and for, uh, for comparing how the model does in, in, in parallel versions of the world, which is not how, not how we do it, not how we learn from experience. We learn from our experience much more efficiently and much more... Um, we use, we use those in many, in many, many ways. At some moment, I've been trying to explain to people that what, what brain does, how, how we learn. It's not that there's one learning algorithm in a brain. I think, I think there are, there are multiple actually, and they-

Sonya Huang 25:41 Mm

Jerry Tworek 25:41 ... and they, they work together. But if I play football, for example, it looks very, very closely to reinforcement learning. I kick a ball a lot of times, and every time I adjust it a little bit and I see if it roughly matches what I, what I wanted, and either there, there's some self-reinforcement happening. When I, when I, when I learn mathematics, it's very different type of thinking. It's, it's like reading ver- about hard concepts and thinking about them very deeply inside my head until, until things click and until-

Sonya Huang 26:10 Mm

Jerry Tworek 26:10 ... until I have them connected. And both of those in some way are learning from experience. They are just very different. So, so summarizing my thinking of the learning from experience is that we've been doing it for a while. We probably are spending the most compute than ever on learning from experience, but lear- reinforcement learning is, is not the end of learning from experience. And there will be, there will, there will be better approaches that will... Researchers will be coming up in the coming years on how to, how to use that data in a, in a richer and richer settings.

Sonya Huang 26:39 Interesting.

Sonya Huang 26:41 Um, Rohan, I'm curious, uh, since a lot of your work has been around optimization and efficiency, how do we get to a orders of magnitude more efficient, I guess more compute efficient and more data efficient learning algorithm?

Rohan Anil 26:55 It had to start with measurement. I think pre-training, as we define it right now, is about compression. We look at perplexity and then measure, uh, how do we decrease the perplexity, and then we find that scaling and increasing parameter count and putting more compute, uh, is the way, and every time we increase, uh, compute in log scale, we get this epsilon more improvement in these metrics. Uh, this is, I think, uh, this is fine for building the prior, but I think this is the wrong way to look at, uh, the problem. We should be looking at the end-to-end. What are we training these models for? Look at the outcome. Like for example, I train this model and give it to Jerry. Jerry will do RL and destroy all the perplexity metrics that I have created, right?

Sonya Huang 27:38 [laughs]

Rohan Anil 27:39 Like, so then it's sort of like it was the best way we had, uh, so far to attempt to solve the problem, and I think the labs and everyone else have done a great job in producing intelligence that's super valuable and makes my work so much fun. But it was the bootstrap process to get there. We have to combine pre-training and RL together, and that's like where one order of magnitude improvement would come from.

Sonya Huang 28:05 Hmm.

Rohan Anil 28:06 And that's like a training procedure. You can say it's a learning algorithm. Um, in terms of optimization, um, my story is I started optimization at Google for logistic regression back in 2016, got nerd sniped by it, worked on some solvers for what we used to call Sybil, which was the large scale linear solver that was used at Google before neural network took off and then, uh, replaced this. Then I, I, I asked myself like, "What do I want to work on with neural network?" And it was quite clear, like I want to understand the training algorithm-

Sonya Huang 28:37 Hmm

Rohan Anil 28:38 ... and make it better. And then someone, Vineeth Gupta, just showed up one day at my desk. It's like, "I heard you're really good at writing optimization methods. We have this idea that, you know," uh, like we worked out on a whiteboard, uh, and what turned out to be the Shampoo algorithm, "Can you help us implement scale, make sure it works at large scale for neur- neural net for training?" So when I was working on this, I, I thank my manager, Yonghui Wu, who's like supported it throughout then to like till my end of my tenure 2024. But largely, like the community and most of the other people were not as excited by this idea.

Sonya Huang 29:12 Hmm.

Rohan Anil 29:12 And for me, this was the most exciting thing because I was like, "I'm putting in computation and making training better. This is the thing I have to figure out. I will spend as much time I would take to do it." And then people were, um, making this assumption, "Oh, why? Like, what's the upper bond? You could still use Adam, that's fine. Like, why are we... You could spend all the time on everything else, not optimization." But in some sense, optimization is like- Like you have a model, you're optimizing it, you want to optimize it better. Now connected to some of the stuff that we talked about architecture, what has happened is that a lot of the work that we've done in architecture is to make these tr- networks train, and in some sense it's like two sides of this, uh, coin, and optimization and architecture go together. You could have a stronger optimizer train a much more harder, harder to optimize model and get better performance, or you can use a weaker optimizer on easier to optimize models and get decent performance.

Sonya Huang 30:11 Mm.

Rohan Anil 30:11 So there are th- these trade-offs that appear all over, and for me it's, I spend a lot of time working on it. I think we used it for Gemini 1.5 Flash, and then like the community started getting like more interested in it. There was the soap paper published. We have like an entire literature of like shampoo, soap, all like bath time, the things that you would use. And then it was quite clear, like so that was like maybe a 2X improvement over what was happening. But even then, if you look at shampoo, it's quite weak in what it's doing. It's not using all the information that's available to you when you train. And a- as you use more and more information a- as part of training, you can get better improvement.

Sonya Huang 30:50 Mm.

Rohan Anil 30:50 And in some sense, like your optimization algorithm defines what architectures you will discover.

Sonya Huang 30:56 Mm.

Rohan Anil 30:56 Like I have colleagues, it's not very popular in the literature, it's only like maybe se- like four people in the world care about it, kind of ideas that are extremely interesting, like residual connections have been extremely useful for training neural networks. There are folks who've now like gotten rid of them and learned deeper representations, but they needed a better optimization method. So like for me, optimization methods and the question you asked, which is like how do we get there, it's, it's combined with architecture and thinking about the problem end-to-end. It is where a lot of the computational efficiency is in.

Sonya Huang 31:29 Yeah.

Rohan Anil 31:30 And I also see RL as spending a lot of compute not as efficiently. And so like if you could spend it, because you don't get much feedback and you're spending a lot more compute because you have to decode all this long-

Sonya Huang 31:42 Yeah

Rohan Anil 31:42 ... chain of thought to get this one bit of information into the network, seems quite inefficient and an easy target to get orders of magnitude on top of. I can go on talking about optimization all day, like-

Sonya Huang 31:54 I love it. How-

Jerry Tworek 31:55 Rohan already has-

Sonya Huang 31:56 Do you think we'll ever approach or surpass, uh, biological learning efficiency?

Rohan Anil 32:01 I do not think so because I think, um, uh, we would need to change, uh, maybe like that was a strong statement. At least with the hardware we have, it seems pretty unlikely.

Sonya Huang 32:11 Mm.

Rohan Anil 32:11 Um, it, our bi- uh, biological, like we have something as, uh, Jeff Hennessy's model computation. So, uh, we build our own circuit as we grow up, and we build our own learning algorithm with the hardware, and then we die, and then we're gone. Uh, neural networks are quite different. Our hard- the hardware stays, the neural network stays, but it's learning very inefficiently and you need a lot more of them and a lot of parallelism to get small amounts of information through.

Sonya Huang 32:36 Mm.

Rohan Anil 32:37 So until I think we design hardware to be much more like how humans operate, maybe more analog-

Jerry Tworek 32:43 Mm

Rohan Anil 32:43 ... uh, figure out how to deal with analog circuits and figure out how to do it at a correction, figure out how to get information through it, it'll be much harder. Uh, we, uh, I think we are safe.

Jerry Tworek 32:54 Mm.

Sonya Huang 32:55 Safe. That's an interesting way to put it. Um, the idea that pre-training and RL should be optimized end-to-end seems like such a, you know, clear, maybe obvious statement. Do, do you think, do you think the labs realize this? And, and is it just hard for them to, you know, get rid of org charts and process to be able to make that come together? Or what stops the labs from, from being able to unify the, the two?

Rohan Anil 33:20 Uh, I don't think it's as obvious-

Sonya Huang 33:22 Okay

Rohan Anil 33:23 ... 'cause, uh, it's a completely, again, different optimization problem. You have a prior, you're doing rollouts, you have higher variance, you're g- uh, and then, uh, pre-training is much large batch and like more parallelism or compute for the unit of time that you can spend. So the, it is not an obvious thing for folks to combine these two training procedures until, uh, you think a bit more like why is it that the naive combination doesn't work? So that's one. Um.

Jerry Tworek 33:51 Mm.

Rohan Anil 33:52 Uh, the second one is if I poll like some of the best researchers in these labs, they would say, "Oh, this makes sense. We should probably explore it," but it would be probably not in the top bucket because they have to train a model for the next cycle before... It's like as, as, uh, Jerry said, like there are companies now competing for release cycles because tokens are not sticky, so it's much harder to do long slight, like even long-term research of six months in many of these labs-

Sonya Huang 34:21 Mm

Rohan Anil 34:21 ... in the environment they are in.

Sonya Huang 34:23 Mm.

Sonya Huang 34:24 So it seems like one of the core premises for Core Automation is, you know, you're starting, you're starting up the lab at a time when I know Sam's been talking about the AI scientists, I think Dario's been talking about the AI scientists. It seems like your job as researchers has actually fundamentally changed. Um, and you're get- you get to start the company native to that era, and as a result, you're maybe gonna be able to run a lot more experiments, uh, than otherwise might be possible. How automatable do you think the research job is? Um, and how are you guys approaching building your lab to be as, I believe your mission, one of your missions is to be the most autonomous lab there is?

Jerry Tworek 34:59 Mo- mo- most automated lab-

Sonya Huang 35:00 Automated

Jerry Tworek 35:00 ... AI lab in the, in the world, and- To, to start with, I th- I think the automation, the version of automation, uh, by Core Automation is about giving each human maximum level of agency in some way. It is-- We are not trying to really get humans out of the loop, which is like one version to automate, but it is about give humans ability to do the most with, with their, their amount of time.

Sonya Huang 35:25 Mm-hmm.

Jerry Tworek 35:25 Whenever, whenever you are walking, you can get some distance. Whenever you get a bike, you can go, go walk a larger distance. Whenever you are a car, you can, you can go much, much, much, much larger. When, whenever, whenever, uh, humans started farming, they had to farm by hand and w- work on a small plot of land. When you have a machine, you work on a much, much larger plot of land. Uh, personally, I am, I'm both really great fan of the, of the current coding agents and, and very happy it's in some way it is what I've been working for many years, both doing coding research and working on various versions of, of AI scientists inside, inside of OpenAI. And, uh, in the end, I realized starting a company to realize that vision is, is, is one of the best, one of the best ways to realize it because the way you can do research today is very, very different because-

Sonya Huang 36:15 Mm-hmm

Jerry Tworek 36:15 ... like single researcher can do much more. In the end, the speed of iteration, the speed of research, the speed of how quickly you can, you can move through ideas and how quickly you can get data on your ideas is something, something very, very different. And-

Sonya Huang 36:29 Hmm

Jerry Tworek 36:29 ... you can try to move the old structures around it and the, the teams, workflows, how, how, how, how, how data is gathered, or you can try to build like, like, like you said, you can, you can try to build, build natively for it for processes that, that like maximally empower each researcher and allow them to just, to just iterate on their, on their idea much, much quicker. We are, we are here and we are trying to, to, to rebuild the full deep learning stack and try to think how we can do almo- almost each operation differently. What are, what are various options? And if we can execute at least even one of those experiments a day, that's already a pretty good it- iteration speed versus anything that was, that was done before, and there isn't really like any, any fundamental like laws of physics reason why not. And maybe, maybe one day we get to, to 10 of those a day. Maybe one day we get, we get to 100 those a day. And fundamentally for that like search process, optimization process, we should be able to just, just find things that work in a better deep learning setting. And, and like what we, what we, what we are trying to do is like within like almo- almost all of us, we are, we are, we're a team that is very agent-built and automation-built. We are, we're, we're, we're trying to do an experiment like how far, how far we can, we can push those things and how far, how, how much an or- organization that tries to do as much as we can with a small team, how far, how far we can get with that.

Sonya Huang 37:53 Hmm.

Rohan Anil 37:54 When will we know that we've reached AGI?

Jerry Tworek 37:57 At some moment I used to say it's, it's very much in everyone's heart whatever, whatever they consider AGI. OpenAI says the system that can, uh, outperform all humans in economically valuable work, but it goes to m- my, my, my previous statement, what if, what if OpenAI stops training models?

Rohan Anil 38:11 Yeah.

Jerry Tworek 38:12 Would that, would that, would that, would that still keep working when they still... Would, would that still keep the automation level the same, the same, or would that, would that drift?

Sonya Huang 38:19 Hmm.

Jerry Tworek 38:20 Um, for me, AGI is a model that can improve itself without human, uh, in the loop-

Sonya Huang 38:28 Hmm

Jerry Tworek 38:28 ... in any, in any, in any way. That's, that, that, that's I think, I think the moment where we can meaningfully talk about, about AGI because that is in some way it is a sub definition of the previous one because improving AI models is actually a job that humans can do. It is economically valuable work, and it definitely, definitely is the case. But removing humans from, from, from loops with models has been actually notoriously, notoriously difficult so far. We haven't come anywhere close to it. It's very, very hard for me to find any task where LLMs were able to like get, get humans out in the loop. We are, we are the, the, the human-LLM hybrid is, is really, really successful right now, but LLMs without humans, not so much, not at all. And what I have seen there in 2024 and in early 2025 is that the current like path doesn't, doesn't get us there, and I think we need, we need some pretty, pretty serious research on why this company or some other to, to try to unlock how do we, how do we make our models learn and, and adapt at, at test time on a deeper level than they've been so far.

Sonya Huang 39:32 Hmm.

Rohan Anil 39:32 Hmm.

Sonya Huang 39:32 I feel like we've alluded to this throughout the conversation, but it's kind of been one of these, you know, five blinded, blindfolded people trying to find the elephant things. Uh, what is the grand master plan for, for Core Auto that you're willing to share?

Rohan Anil 39:45 We can share our six-month roadmap. Um, in some sense, like building architectures, as I said, like would... It's not just how good the architecture is. Does it run well, and can we get users, including ourselves as part of the lab, to use it, right? So then, um, that's directly like we can do many things now, but the thing that is gonna be difficult and the thing that we want to automate away is kernel generation. So we would, uh, we have a set of hardware, GPUs, Blackwells that we have to train and run inference on, where we will build the best model that we can to, um, basically reduce the time from having a very cool idea that can make, um, these bottlenecks go away in the architecture to having them run at the highest TFLOPS on GPUs.

Sonya Huang 40:32 Hmm.

Rohan Anil 40:32 And in some sense, like current coding agents plus humans can go a long way. But, uh, like, uh, an example of this is our QR kernel competition that we hosted with, uh, GPU mode. It's for, uh, running this very old linear algebra operation QR.

Sonya Huang 40:50 Hmm.

Rohan Anil 40:50 Um, it's used for optimization, like Shampoo line of work uses it, many other places uses it, and you want to run this efficiently on a B200 node. Uh, if you use CuSOLVER for the shapes that we care about, you get some efficiency, and then a human plus, uh, some search loop can get you like something like 7X. But, uh, it requires the high-taste human, like there's maybe three people in the world to... and spend about $100,000 on these coding agents over a span of four weeks to get to a solution that's 60X faster.

Sonya Huang 41:25 Hmm.

Rohan Anil 41:25 So these models today are nowhere close to getting that 60X faster kernel. And does requi-- there is a real bottleneck. Now, uh, that was a single pro-problem. It has, like, perhaps three different operators. Uh, work on this panel, do this matrix multiply, fold it back in, and do this repeatedly. That's what the secure factorization of a matrix would look like.

Jerry Tworek 41:45 Mm.

Rohan Anil 41:46 And if you give this problem to entropic models, OpenAI models, Gemini, it just wouldn't solve it. It just... It's not-- Our models are not even close to solving this problem. So for us, it's like something that we've talked about, something that we are getting close to as sort of like, mm, getting to that point because that's our inner loop to having more efficient architectures.

Jerry Tworek 42:06 Mm-hmm.

Sonya Huang 42:07 Why kernels? Is it just 'cause, like, maximize intelligence per flop of compute, you need to generate kernels?

Rohan Anil 42:12 Yeah, i-in some sense, um, I've had, like, three projects. Two of them have kind of landed in the industry. So first is second-order methods. Kernels was a bottleneck 'cause you have to run it well. If you were at a place like Google, you cannot spend ten X, uh, amount of compute and get a two X win. So I could only spend maybe a budget of twenty percent and get the two X win. Everyone's very happy, but on, like, great. So, like, I think the-- that's the market, right?

Sonya Huang 42:39 Mm.

Rohan Anil 42:39 You spend, uh, less than-

Sonya Huang 42:41 Yeah

Rohan Anil 42:41 ... you, uh, get. So kernels ended up being a bottleneck there 'cause most of the operations were novel that we haven't gotten a lot of people to look at it. There's only, like, two humans at Google who could write it, Rasmus and Peter Hawkins, 'cause it was deep XLA LLO code that you have to write to make this work, and then that took them two years to do. The other i-idea I had with one of my coworkers at that time was, like, replacing some of the parameters in a transformer with extra memory, and we called it engrammer, engram memory.

Sonya Huang 43:15 Mm.

Rohan Anil 43:15 We worked on it in twenty twenty. It's, uh-- We had versions of it internally deployed, not the big version, but the smaller version. But there, I needed, like, something that can accelerate sparse, uh, gathers and scatters as part of training.

Sonya Huang 43:29 Yeah.

Rohan Anil 43:30 Uh, it required a hardware change and a hardware making use of the hardware. It never arrived. I had c- uh, conferences set up with the TPU team, us, and a bunch of others. We were talking about it during COVID, like, "Oh, we're gonna have it happen," and it never arrived. I also was using TPUs at Anthropic while I was leaving, just barely started the surface of being able to, um, do it. But at the same time, six months before that, DeepSeek wrote, uh, their engram, which is a improved version of adding more memory, showed scaling laws that, yeah, you don't need MOEs. You could ac-actually replaced it with these-

Sonya Huang 44:06 Mm.

Rohan Anil 44:06 ... engram embeddings. For me-

Sonya Huang 44:07 Interesting

Rohan Anil 44:08 ... that was like, ah.

Sonya Huang 44:09 Yeah.

Rohan Anil 44:10 Oh.

Sonya Huang 44:10 Very interesting.

Rohan Anil 44:11 It was like a five-year thing, and I was very happy for them.

Sonya Huang 44:14 So much of the, so much of the state space to explore isn't even possible if you're not writing kernels.

Rohan Anil 44:18 Kernels.

Sonya Huang 44:18 Very nice.

Rohan Anil 44:18 And you need to be assisted in writing kernels or solve that kernel-

Sonya Huang 44:22 Interesting

Rohan Anil 44:23 ... uh, to have the highest performance.

Jerry Tworek 44:24 Yeah.

Rohan Anil 44:24 Like, and the, uh, the roof line is pretty high, so it's like the QR. If I use Q solver's QR, I get some performance. If I-- we use our, the competition winner's QR, you get sixty X faster.

Sonya Huang 44:37 Yeah.

Jerry Tworek 44:37 Mm.

Rohan Anil 44:37 And that is a completely different playing field. Now it opens up an entire new set of algorithms you can apply.

Jerry Tworek 44:42 Mm.

Rohan Anil 44:43 And in terms of training transformers, training optimizers, QR is so fundamental in analyzing the eigen decomposition-

Sonya Huang 44:49 Yeah

Rohan Anil 44:49 ... for eigen decomposition and many other things. So it is a thing that I think also, if you think about it, only few people have the skill set to, and they're very much not at the same place. It's like one person here, one person there.

Sonya Huang 45:04 Yeah.

Rohan Anil 45:04 And it, it, it would be ideal if models had those abilities.

Sonya Huang 45:08 Yeah.

Jerry Tworek 45:08 Yeah. Maybe I'll, I'll summarize a little bit and talk from the high level of what we want. Core Automation is a lab created to, uh, build models that continuously learn and learn, learn from deployment. We believe, as, as, as, as, as I mentioned, that transformers are incapable of continual learning. There, there's no way how to put continual learning on transformers, so we know we have to find a different, different architecture. Some of-- In, in a way, our quest is to find that, that new architecture, find that transformer replacement, and we wanna build the most automated lab to do it. We wanna be able to build experiments at scale the quickest we can, iterate on them, try a lot of new architectural ideas, have strong priors of what we want to do to search the space of architectures efficiently to find, to go to that, go to that place fastest than, than anyone else. That's what we, what we want to do and all the work we are doing on kernels, on large scale training, on trying new architectural ideas is, is, uh, exploring that space. So you're gonna experiment your way into finding a superior architecture. How will you know when you've found it? What are you looking for to say, "Aha, this is the one"?

Rohan Anil 46:22 Hmm.

Jerry Tworek 46:23 That's, that's a great question. There are, there are always two angles. W-In, in, in my mind, in my experience, every successful research had a plot that shows something that-

Rohan Anil 46:33 Yeah. [laughs]

Jerry Tworek 46:33 ... that other, other plots don't show. There, there is one line that is a little bit bending in a, in a, in a different, different way, and you're saying this, this, this is what you want. But at least, at least my experience with research always has been that plot is already quite late in a journey where most of the time you already know what you want and already know what you are up to. I, I, I, I am a bit, a bit joking, but it's actually true that all the best plots in my life, I had them in a dream before, before actually they were-

Rohan Anil 47:00 Hmm

Jerry Tworek 47:00 ... they were, they were real. I kind of knew what I was looking for. Just, just the question is, it's like when it, when it actually clicks-

Rohan Anil 47:06 Hmm

Jerry Tworek 47:06 ... if you, if you know what I mean. Because most of the time you kind of know what you are looking for, but you are, you are not finding it. You, you try one thing and, ah, it doesn't work. Try second thing. Oh, it doesn't, it doesn't work. But eventually the, all the, all the right pieces fall into it, and most of the deep learning systems are very intricate. So usually you have to get five things right in a row for the-

Rohan Anil 47:25 Mm

Jerry Tworek 47:25 ... for the thing to start working, and then eventually you get the plot that looks like, like, like, like you want, and then, and then you know. So-

Rohan Anil 47:31 Mm-hmm

Jerry Tworek 47:32 ... so I, I think, like, what we are, what we are looking for is systems that learn at test time, and if we see meaningful long-term adaptability of our, of our systems, and like we are, we are-- We're joking, but it's, it's, it's, uh, it's real. We wanna be evaluating our systems of our, of our everyday work. They get better at s- at, at doing the work of core automation scientists each day.

Rohan Anil 47:57 Like, yeah, we, like, go on a vacation as a team and see if the lab produces something better for the week. Give the-

Jerry Tworek 48:04 And then what do you do when you get back?

Rohan Anil 48:05 Um, we'll see where-

Jerry Tworek 48:07 Extend, extend the vacation two times instead of four times until we are on permanent vacation. [laughs]

Sonya Huang 48:13 [laughs] Uh, that is a beautiful note to end on. Um, Rohan, Jerry, thank you so much for joining us. You've both worked on, like, really, really transformative work for where we are today, and I'm so excited to see you starting a lab on this, this new journey, uh, and very excited to see what you're able to come up with.

Jerry Tworek 48:35 Thank you so much.

Sonya Huang 48:36 Thanks for joining us.

Rohan Anil 48:36 Thank you. It's been, it's been great to be here and to chat with you.

Jerry Tworek 48:39 Thank you. [outro jingle]