By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Sequoia CapitalBuilding the Automated AGI Lab: Core Automation's Jerry Tworek and Rohan Anil
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
Core Automation’s bet: replace transformers via continual, test-time learning
- Jerry Tworek said appreciating transformers means understanding what they do well and why they became economically viable, while also recognizing their limitations for the next phase of progress.
- Jerry argued that as RL scaling improved benchmark results, real-world tasks still lagged because the deployment distribution is messier and changes, pushing him toward “learn-at-test-time” systems rather than lab-only training loops.
- Rohan Anil framed the transformer’s weakness as poor computational depth and inefficient inference-time scaling via one-token-at-a-time chain-of-thought, motivating architectures that spend computation better across pre-training and RL.
- Both suggested organizational and market pressures make incumbent labs less likely to pursue risky architectural alternatives, which is part of why they started Core Automation.
- They described Core Automation as building a highly automated, agent-augmented research lab, with an initial focus on automating kernel generation to shorten the loop from architectural idea to high-performance GPU implementation.
IDEAS WORTH REMEMBERING
5 ideasThey argue architecture—not just scaling—is becoming the main bottleneck.
He said the field has gotten very good at scaling pre-training and reinforcement learning, but that progress is now bottlenecked by the underlying architecture rather than just more parameters or efficiency tweaks.
Benchmark gains don’t translate cleanly to real-world performance.
He described models improving on benchmarks as RL scaled, yet still failing at messy, real-world tasks because training/eval tasks mirror each other while real deployment is different and changes over time.
Current “test-time learning” options are either too limited or too brittle.
He contrasted in-context learning (data-efficient and avoids catastrophic forgetting, but limited by context/horizon) with continual fine-tuning (limited by data efficiency and catastrophic forgetting), arguing the missing piece is a new approach that supports longer-horizon learning.
Transformers won partly because they are economically scalable, not only technically capable.
He said transformers are economically valuable because training cost is lower than the revenue they generate, and that this economic advantage helped make large-scale training happen compared to alternatives like scaling LSTMs.
Competitive market dynamics push big labs toward short-term transformer iteration.
He argued frontier labs are incentivized to keep scaling profitable transformer-based approaches on short release cycles, making it hard to invest attention and resources into uncertain architectural alternatives.
WORDS WORTH SAVING
5 quotesWhat I think right now we are in this stage, we got really, really good at training really, really big models. We mastered two algorithms. We mastered pre-training at a large scale, and we mastered reinforcement learning at a large scale.
— Jerry Tworek
Our training data didn't really replicate the real world use cases, and despite us b- basically maximizing all the tasks, if you see, ask anyone training models, "Hey, what is one of your main issues? I don't have hard enough tasks. I don't, I don't have what to- what to train our model on," yet we are still not covering the, the entirety of the, of the real world distribution.
— Jerry Tworek
From that, my conclusion is we need to have models that learn at test time. We need to have models that learn with users on their data, on their real world task, on the real world distribution.
— Jerry Tworek
A theoretically optimal architecture is not useful to anyone. It is something when it comes into practice.
— Rohan Anil
For me, AGI is a model that can improve itself without human, uh, in the loop- in any, in any, in any way.
— Jerry Tworek
SPEAKERS
Jerry Tworek
guestAI researcher and co-founder of Core Automation, previously at OpenAI.
Pat Grady
hostPartner at Sequoia Capital and co-host/interviewer on the episode.
Sonya Huang
hostPartner at Sequoia Capital and primary host/interviewer of the episode.
Rohan Anil
guestAI researcher and co-founder of Core Automation, formerly at Google Brain with a focus on optimization and training efficiency.
High quality AI-generated summary created from speaker-labeled transcript.
