Recursive Language Models — Alex Zhang, MIT PhD

ℹ️ EPISODE INFO

Released

October 2, 2026

Duration

1h 43m

Channel

Latent Space

Watch on YouTube

▶ Open ↗

📝 EPISODE DESCRIPTION

From GPU kernels and KernelBench to Recursive Language Models, agent harnesses, and massive multi-agent swarms, Alex Zhang is exploring how much capability we’re leaving on the table by wrapping increasingly powerful models in primitive systems. In this episode, the MIT researcher behind RLMs joins swyx to unpack why Claude Code, Codex, and most coding agents are more similar than they look, why better harness design could unlock capabilities already latent in today’s models, and what happens when you can throw 10,000 agents and tens of millions of dollars at a single problem. We go deep on GPU Mode and AI-written kernels, research taste and why academics should take bets industry labs won’t, GEV and alternatives to the standard autoregressive language model, and the idea of harnesses as compositional generalizers. Alex explains RLMs, context offloading, programmatic subagent calling, Prime Agent, persistent subagents, and why the “language model” of the future may actually be an invisible swarm of agents underneath a simple interface. We also discuss OpenAI’s massive agent experiments, Kimi swarms, open-ended research at Sakana AI, speculative programmatic tool calling, capability overhang, Neuralese, and where Alex thinks the next big research opportunities may lie. We discuss:

• Why AI-generated GPU kernels still leave substantial room for human expertise
• How one expert insight can potentially replace enormous amounts of brute-force token search
• Why PhD students should take research bets that initially look trivial, weird, or pointless
• What SWE-bench, RLMs, ReAct, and Quiet-STaR reveal about research taste
• GEV and why a language model does not have to mean an autoregressive text-to-text decoder
• Why Claude Code, Codex, Pi, and many modern agent harnesses are structurally very similar
• How harness design can improve compositional generalization across tasks and domains
• RLMs: context offloading, code execution, recursive subagents, and shared memory
• Prime Agent, continual harnesses, and persistent agent-to-agent communication
• Why the model you query in the future may secretly be an entire swarm or scaffold
• OpenAI’s 10,000-agent experiment, 130B output tokens, and ~$40M-equivalent problem solving
• Why much of an agent swarm may be wasted search — and why convergence is still hard
• Kimi versus OpenAI’s approach to multi-agent systems
• Open-endedness, Sakana AI, and the problem of finding hidden gems in enormous amounts of generated work
• Why current frontier models may already have a large capability overhang
• Speculative programmatic tool calling and overlapping tool execution with generation
• Whether English, code, or an entirely new “Neuralese” constrains how models reason
• AI for science, fast-moving benchmarks, and how Alex chooses what research problems to bet on

Sponsorship and business inquiries: business@latent.space — Alex Zhang

• Website: alexzhang13.github.io
• X: @a1zhang

Timestamps 00:00:00 Hook 00:01:56 Introduction 00:02:45 GPU Mode, KernelBench, and AI-Written Kernels 00:09:34 Human Expertise vs. Brute-Force AI Search 00:15:16 Research Taste and Taking Big Bets 00:21:24 GEV and Rethinking the Language Model 00:30:59 Video Game Agents and the Harness Problem 00:32:57 Why Claude Code, Codex, and Pi Are So Similar 00:38:38 Harnesses as Compositional Generalizers 00:46:20 RLMs Explained 00:53:57 Prime Agent and Persistent Subagents 00:59:37 RLMs in the Wild 01:02:26 OpenAI Swarms and the Future of Language Models 01:09:22 Open-Endedness and Sakana AI 01:17:48 Kimi vs. OpenAI Agent Swarms 01:22:02 Capability Overhang and Speculative Tool Calling 01:30:15 Neuralese, Future Research, and AI for Science

🎤 SPEAKERS

• Alex Zhang

  guest

  MIT PhD student researching recursive language models (RLMs) and related evaluation/agent tooling.

• swyx

  host

  Co-host of Latent Space; AI engineer/writer who interviews researchers and practitioners in ML/AI.

• Alessio Fanelli

  host

  Co-host of Latent Space; ML engineer/research-focused interviewer covering agents, evals, and frontier-model work.

🧾 EPISODE SUMMARY

In this episode of Latent Space, featuring Alex Zhang and swyx, Recursive Language Models — Alex Zhang, MIT PhD explores alex Zhang on RLMs, harness design, and agent swarms Alex Zhang explains how GPU Mode and KernelBench grew from the idea that GPU kernel optimization has a learnable, competition-friendly structure and could be partially automated with LLMs.

🔗 RELATED EPISODES

Robots That Predict the Future: Dyna-2 & Physical AI — Jason Ma, Dyna Robotics

Supabase duplicate

Why AI Compute Comes in Rectangles — Erik Bernhardsson, Modal

How Anthropic ships the #1 LLM in the World — with Katelyn Lesse and Angela Jiang

Why AI Labs With Unlimited GPUs Still Fail — Anjney Midha, AMP

🔬 The Limits of AI in Science - Why We Need Self-Driving Labs — Joseph Krause, Radical AI