By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Latent SpaceRecursive Language Models — Alex Zhang, MIT PhD
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
CHAPTERS
- 0:00 – 1:09
Hook: agent swarms, tool sameness, and the $40M question
Alex Zhang says many popular coding/agent tools feel essentially the same, and the real lever is “trying harder” with better harnessing. The conversation immediately frames a future where you can spend massive compute—tens of millions of dollars—on swarms to solve hard problems, even if it’s inefficient.
- •Alex says Claude Code, Codex, Pi, and Prime Agent feel “the same” at a high level
- •The host frames the idea that model underperformance can be a “skill issue” in how people use them
- •They discuss the notion of spending around $40 million in public pricing to get a result with large agent runs
- •Alex emphasizes it’s exciting that throwing huge compute at a problem is even possible
- 1:09 – 2:26
Show opening and Alex’s affiliations (MIT, GPU Mode)
The host welcomes Alex and establishes his background, including GPU Mode and his connections to the community. Alex describes GPU Mode as a collaborative network that extends beyond the Discord itself.
- •The host introduces Alex as known for recursive language models (RLMs) and mentions GPU Mode
- •Alex describes being close with GPU Mode members and collaborating beyond the server
- •GPU Mode is framed as a tight technical community with strong signals for talent and hiring
- 2:26 – 7:08
GPU Mode origins: from CUDA lectures to KernelBench and automated kernels
Alex recounts how GPU Mode started as a Discord focused on learning GPU kernel writing through lectures and community. He explains how interest in specialized kernels and the “competitive programming” flavor of optimization led to ideas like Popcorn/leaderboards and KernelBench for AI-written kernels.
- •GPU Mode began around 2023 with lectures aimed at learning GPU kernels
- •Alex joined while exploring specialized kernels (inspired by work like Infinite Attention)
- •The community explored the idea that kernel optimization has a small, learnable optimization space
- •KernelBench emerged from the goal of getting language models to automate GPU kernel code
- •Alex notes kernels are crucial for making research ideas usable end-to-end
- 7:08 – 15:17
AI-written kernels vs human expertise: stability, verification, and system-level speed
The discussion turns to how AI-generated leaderboard solutions often work but may be unstable in real systems. Alex explains kernel “verification” and reward hacking issues, the relevance of theoretical performance limits, and why isolated kernel speed differs from end-to-end model speed.
- •Alex says many recent top leaderboard solutions are AI-generated, but stability is a differentiator
- •He highlights a strong community kernel writer whose approach (with AI assistance) produced uniquely stable results
- •Alex describes verification and reward hacking as known issues since KernelBench
- •He explains “speed of light” estimates for kernels and why reaching them can be hard in practice
- •He distinguishes optimizing a single kernel from optimizing an end-to-end model, including cache/fusion considerations
- •Alex argues mega-kernel generation is hard without data and may be better handled by compilers in many cases
- 15:17 – 21:17
Benchmarks, research taste, and why big bets matter in academia
Alex shares his benchmark background (including ties to Princeton work) and broadens into how research taste develops through opportunity and advisor alignment. He argues the most impactful academic work often targets problems that industry isn’t already optimizing, and that PhD students should take big, risky bets.
- •Alex describes prior benchmark involvement and relationships with benchmark teams at Princeton
- •The host and Alex discuss “research taste” and how some researchers repeatedly land on enduring problems
- •Alex critiques academia chasing industry-shaped goals (like narrow harnesses/benchmarks) instead of exploring new directions
- •He uses examples of initially-dismissed ideas later becoming central as evidence for taking contrarian bets
- •Alex argues academia’s advantage is freedom from bureaucracy, enabling riskier exploration
- 21:17 – 30:13
GEV as a provocation: changing output spaces and rethinking what a “language model” is
Alex explains why he finds GEV interesting beyond hype: it challenges the assumption that language models must be standard text-to-text autoregressive decoders. He connects the idea to latency and cost tradeoffs, suggests it opens a broader design space, and discusses why smaller labs need differentiated bets.
- •Alex argues backlash calling GEV “obvious” misses the point: it reframes model design around output space and latency
- •He questions why binary classification should require expensive generative decoding
- •Alex pushes back on narrow definitions of “language model,” connecting this to criticism he received about RLMs
- •He speculates that training/optimization details—more than architecture labels—can be the real moat
- •A host brings up calibration as an under-discussed axis and how it might relate to fast classification models
- •Alex and Alessio discuss why non-frontier labs need strategies other than copying frontier autoregressive scaling
- 30:13 – 32:51
Video game agents and the latency-harness constraint
Alex describes a benchmark he built to test whether vision-language models can play games under real latency constraints. He argues many models still struggle to meaningfully solve games, and he uses games to motivate why harness design matters.
- •Alex describes building a benchmark to test game-playing with latency constraints
- •He notes the benchmark is diverse and uses games many people recognize
- •He says models generally still can’t solve the games meaningfully, though some progress exists
- •Alex emphasizes they intentionally designed a minimal harness, setting up a broader discussion of harness value
- 32:51 – 45:59
Harnesses as compositional generalizers: why common agent products feel identical
Alex argues most agent “harnesses” (Claude Code, Codex, Pi, etc.) share the same underlying loop and design philosophy. He proposes that harnesses should be treated as powerful, opinionated programs that can induce better generalization by structuring tasks into locally solvable subcalls.
- •Alex says many popular harnesses are effectively the same set of design choices
- •He frames harnesses as opinionated programs that shape how a model fits a problem
- •He critiques next-token prediction as awkward for complex tasks like navigating codebases
- •Alex suggests harness design can create compositional structure so different tasks share the same high-level solution strategy
- •He argues harnesses can boost generalization by breaking problems into easier subcalls that stay locally in-distribution
- 45:59 – 53:40
From “trajectory as prompt” to locally-in-distribution calls (and why training matters)
The group digs into why current harnesses often push models out of distribution by appending long rollouts into context. Alex references work discussing the “harness tax,” argues many harness choices don’t matter because they’re too similar, and claims progress requires real experimentation on alternative harness designs and training around them.
- •Alex references a paper arguing many harness choices don’t matter much because harnesses are broadly similar
- •He argues models are increasingly robust in standard coding harnesses, but still weak in RLM-style harnesses
- •Alex critiques “trajectory as a prompt” as a dominant pattern that becomes out-of-distribution over long tasks
- •He calls for teams to run serious scaling experiments on new harness designs (including post-training around harness rollouts)
- •He shares an internal observation that certain models worked better for RLM workflows due to trained-in workflow behavior
- 53:40 – 59:38
RLMs explained crisply: code as the only tool, context offloading, and shared memory
Alex gives a clean definition of recursive language models as a harness where the model’s primary tool is code that can call the model again as a subroutine. The key is that context lives in an external memory (like files on disk), and tools are invoked via a code environment such as Python or Bash, enabling composition and persistent reference to original information.
- •Alex defines an RLM as a harness where the tool interface is code, including self-calls as a tool
- •He explains context is stored in external memory (e.g., filesystem) rather than only in the prompt
- •Alex uses Prime Agent as an example where compaction and original context can coexist via disk storage
- •He contrasts RLMs with tool-call harnesses that require turn-by-turn invocation without a durable central context
- •Alex argues RLMs aim to make each local call in-distribution even if the overall task is novel
- 59:38 – 1:02:27
Prime Agent: IPython-only tooling, continual harnessing, and persistent subagents
Alex explains how Prime Agent adapts a minimalist harness approach while enforcing a strict “IPython is the tool” interface. He describes continual harness ideas (what the system can modify about itself), how subagents communicate through code, and why persistent subagents improve visibility and long-running work.
- •Alex says he joined Prime after they publicly aligned with the RLM vision and understood the paper’s intent
- •Prime Agent is described as a harness built on a minimalist loop, but restricted so IPython is the only direct tool
- •Other tools are loaded as Python modules or scripts, keeping actions programmatic
- •Alex describes a “continual harness” idea: allowing certain harness components (skills/subagents/system prompt) to be modified
- •He highlights a communication framework for many subagents and the addition of persistent subagents that outlive the main run
- •The host notes ephemeral subagents in other tools as a practical pain point; Alex points to filesystem externalization as a key trick
- 1:02:27 – 1:09:22
RLMs in the wild: legal workflows, always-on systems, and competition harnesses
Alex highlights third-party adoption and adjacent systems inspired by the RLM abstraction. He cites a legal AI company’s post-trained RLM, an always-on “persistent thinking” harness, and references to RLM-like ideas in competition setups focused on compositional reasoning.
- •Alex cites a legal AI company post-training an RLM for document-heavy legal work and getting strong results
- •He describes an always-on harness system that keeps token costs controlled while thinking persistently
- •Alex points listeners to multiple community projects linked from the RLM repository
- •He notes competition harnesses referencing RLM-inspired abstractions, where composition and code-centric scaffolds help
- 1:09:22 – 1:17:48
OpenAI-scale swarms, when harness details matter less, and the $40M compute reality
The conversation shifts to high-scale agent swarms solving very hard problems via long search and shared context. Alex suggests that at sufficient scale, the specific harness may be less important than how agents are composed and how information accumulates, while user-facing products should hide the messy swarm internals.
- •The host quotes a large-scale agent run in terms of agents, runtime, and output token volume and frames the public-pricing cost
- •Alex argues the core may be long search over many agents plus human intuition guiding exploration, rather than a special harness trick
- •He suggests the “specific details” of harnesses may matter less than composing agents to reach an answer
- •Alex predicts what users call a “language model” may become an opaque scaffold/swarm behind a simple interface
- •They discuss the tradeoff between centralized coordination (gather/merge bottlenecks) and freer swarm behavior
- 1:17:48 – 1:22:00
Open-endedness and Sakana: “no prompt” exploration and the selection problem
The host asks about open-endedness—letting systems run without a fixed prompt or objective—and Alex relates it to evolutionary search and swarm-style exploration. Alex emphasizes the hardest part is choosing what’s meaningful from a large amount of generated “slop,” and reflects on Sakana as a research-oriented lab that embraces weird ideas.
- •Alex connects open-endedness methods to evolutionary search and swarm exploration strategies
- •He questions whether open-endedness always reduces to a hard problem with a clear objective, and what the value is otherwise
- •Alex highlights the selection/filtering challenge: finding “hidden gems” in huge volumes of generated content
- •He mentions exposure to open-endedness work during a summer at Sakana and notes ongoing nudging is still often needed
- •Alex describes Sakana as operating more like a research lab than a frontier product lab and appreciates their willingness to do unusual work
- 1:22:00 – 1:29:48
Kimi vs OpenAI swarms, skepticism on dynamic workflows, and thoughts on Gemini/GDM
Alex compares swarm efforts and argues that getting swarms to reliably converge is non-trivial and likely requires dedicated training, not just a smart base model. He contrasts his impression of OpenAI’s swarm approach with Kimi’s, critiques “dynamic workflows” as too expensive/underwhelming, and shares a mixed view of Google DeepMind’s impressive harness work alongside organizational friction.
- •Alex says OpenAI’s swarm results imply serious training work to make swarms act toward goals, not something to take for granted
- •He characterizes Kimi’s swarm work as interesting but unclear in terms of solving novel problems
- •Alex calls “dynamic workflows” expensive and says it didn’t behave as compellingly as the promise suggested
- •He emphasizes the key metric is how much swarm exploration is wasted versus converging usefully
- •Alex praises Gemini-era harness cleverness for strong results when base models were weaker, but criticizes GDM bureaucracy and product adoption challenges
- 1:29:48 – 1:43:24
Capability overhang, speculative tool calling, Neuralese tangents, and Alex’s research posture
Alex argues today’s frontier models still can’t reliably do long-running work over weeks, calling it a solvable “skill issue” with better harness design and continual-learning-like behavior. He briefly explains speculative tool calling as launching tool execution early to reduce latency, then the group explores whether models might benefit from alternative “languages” beyond English. Alex closes by describing how he picks projects, what kinds of outreach he values, and how he thinks about AI-for-science timelines and feedback loops.
- •Alex claims current frontier models aren’t yet reliable at long-running simple work, and he believes better harnessing could close the gap
- •He frames a mismatch between what an equally “smart” human could do versus what a model can do under current harnesses
- •Alex defines speculative tool calling as starting tool execution early (via analysis/static structure) to reduce sequential waits
- •The host raises the idea of “Neuralese” and alternative output modalities; Alex prefers a mix of English and Python based on training signals
- •Alex says harness/agent research feels fuzzier than systems work but may be where major innovation still remains
- •He invites collaboration from people with strong, reasoned opinions and specific ideas rather than vague “pick your brain” outreach
- •Alex notes science applications have long feedback loops and project choice must account for rapid shifts in model capability
