Skip to content

By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

Latent SpaceLatent Space

Recursive Language Models — Alex Zhang, MIT PhD

From GPU kernels and KernelBench to Recursive Language Models, agent harnesses, and massive multi-agent swarms, Alex Zhang is exploring how much capability we’re leaving on the table by wrapping increasingly powerful models in primitive systems. In this episode, the MIT researcher behind RLMs joins swyx to unpack why Claude Code, Codex, and most coding agents are more similar than they look, why better harness design could unlock capabilities already latent in today’s models, and what happens when you can throw 10,000 agents and tens of millions of dollars at a single problem. We go deep on GPU Mode and AI-written kernels, research taste and why academics should take bets industry labs won’t, GEV and alternatives to the standard autoregressive language model, and the idea of harnesses as compositional generalizers. Alex explains RLMs, context offloading, programmatic subagent calling, Prime Agent, persistent subagents, and why the “language model” of the future may actually be an invisible swarm of agents underneath a simple interface. We also discuss OpenAI’s massive agent experiments, Kimi swarms, open-ended research at Sakana AI, speculative programmatic tool calling, capability overhang, Neuralese, and where Alex thinks the next big research opportunities may lie. We discuss: • Why AI-generated GPU kernels still leave substantial room for human expertise • How one expert insight can potentially replace enormous amounts of brute-force token search • Why PhD students should take research bets that initially look trivial, weird, or pointless • What SWE-bench, RLMs, ReAct, and Quiet-STaR reveal about research taste • GEV and why a language model does not have to mean an autoregressive text-to-text decoder • Why Claude Code, Codex, Pi, and many modern agent harnesses are structurally very similar • How harness design can improve compositional generalization across tasks and domains • RLMs: context offloading, code execution, recursive subagents, and shared memory • Prime Agent, continual harnesses, and persistent agent-to-agent communication • Why the model you query in the future may secretly be an entire swarm or scaffold • OpenAI’s 10,000-agent experiment, 130B output tokens, and ~$40M-equivalent problem solving • Why much of an agent swarm may be wasted search — and why convergence is still hard • Kimi versus OpenAI’s approach to multi-agent systems • Open-endedness, Sakana AI, and the problem of finding hidden gems in enormous amounts of generated work • Why current frontier models may already have a large capability overhang • Speculative programmatic tool calling and overlapping tool execution with generation • Whether English, code, or an entirely new “Neuralese” constrains how models reason • AI for science, fast-moving benchmarks, and how Alex chooses what research problems to bet on Sponsorship and business inquiries: business@latent.space — Alex Zhang • Website: alexzhang13.github.io • X: @a1zhang Timestamps 00:00:00 Hook 00:01:56 Introduction 00:02:45 GPU Mode, KernelBench, and AI-Written Kernels 00:09:34 Human Expertise vs. Brute-Force AI Search 00:15:16 Research Taste and Taking Big Bets 00:21:24 GEV and Rethinking the Language Model 00:30:59 Video Game Agents and the Harness Problem 00:32:57 Why Claude Code, Codex, and Pi Are So Similar 00:38:38 Harnesses as Compositional Generalizers 00:46:20 RLMs Explained 00:53:57 Prime Agent and Persistent Subagents 00:59:37 RLMs in the Wild 01:02:26 OpenAI Swarms and the Future of Language Models 01:09:22 Open-Endedness and Sakana AI 01:17:48 Kimi vs. OpenAI Agent Swarms 01:22:02 Capability Overhang and Speculative Tool Calling 01:30:15 Neuralese, Future Research, and AI for Science

Alex ZhangguestswyxhostAlessio Fanellihost
Oct 2, 20261h 43mWatch on YouTube ↗

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

At a glance

WHAT IT’S REALLY ABOUT

Alex Zhang on RLMs, harness design, and agent swarms

  1. Alex Zhang explains how GPU Mode and KernelBench grew from the idea that GPU kernel optimization has a learnable, competition-friendly structure and could be partially automated with LLMs.
  2. He argues that while AI-generated kernels dominate many leaderboards, real-world reliability exposes a verification and reward-hacking problem where expert human oversight still provides major leverage.
  3. The conversation reframes research strategy: impactful academic work often comes from taking big bets on ideas that initially seem trivial or “pointless,” as with SWE-bench and related harness-oriented papers.
  4. Zhang critiques today’s agent/coding harnesses as mostly similar and proposes more opinionated, compositional harness designs—especially Recursive Language Models (RLMs)—that offload context to code environments and enable recursive subagents.
  5. They discuss multi-agent swarms and open-ended search, noting both the promise (solving hard problems with enough compute) and the difficulty of efficiency and selecting “interesting” outputs from large volumes of generated work.

IDEAS WORTH REMEMBERING

5 ideas

Many popular agent/coding harnesses are effectively the same, leaving room for new harness design space.

He argues most mainstream coding/agent harnesses differ mostly cosmetically and follow similar loop patterns; the bigger opportunity is being “more creative with harnesses” by making them more opinionated and structured for how a model should solve a task.

GPU kernel work can be benchmarked and scaled similarly to competitive programming, enabling automation.

He describes KernelBench/GPU Mode as motivated by the idea that GPU kernel optimization has recurring optimization patterns and a relatively bounded space of problems, making it amenable to scaling data/competition and automation with LLMs.

AI can generate high-performing kernels, but verification and end-to-end stability remain a key bottleneck.

He notes that many leaderboard solutions are AI-generated but highlights a stability/verification gap: strong human verifiers and domain intuition still matter because kernels can “reward hack,” pass narrow checks, yet fail in real systems.

Human expertise can erase enormous brute-force search cost by acting as a strong verifier/steerer.

He emphasizes that throwing huge compute at a problem can explore solution space, but knowledgeable humans can steer search and verification so effectively that it can outperform massive token spend in efficiency terms.

High-impact academic work often comes from taking big bets that look trivial or pointless at first.

He frames academia’s advantage as the ability to take “big bets” on ideas industry may dismiss as trivial or purposeless, citing examples where initially-ignored ideas later became central once the field’s context changed.

WORDS WORTH SAVING

5 quotes

It- it- to be honest, I think all of them are the same

— Alex Zhang

I think it's like you just have to take big bets and like a lot of them will fail, you know? Like that's just, it's, it's natural.

— Alex Zhang

A language model is just modeling language. It doesn't have to be this transformer decoder, you know?

— Alex Zhang

It is-- it's very exciting that we even have the option to point $40 million at a problem and solve it.

— Alex Zhang

We take for granted what it means for a swarm to converge to an answer.

— Alex Zhang

GPU Mode and kernel-writing communityKernelBench and AI-generated kernelsVerification, reward hacking, and stability in benchmarksHuman expertise vs brute-force token/compute spendResearch taste, academia incentives, and “big bets”GEV and alternative language-model output spacesHarnesses as compositional generalizers and RLM definition

High quality AI-generated summary created from speaker-labeled transcript.

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.

iOSAndroidClaudeChrome