a16zCan Open Models Keep Up With OpenAI and Anthropic? | Nathan Lambert
CHAPTERS
- 0:00 – 2:56
Compute, data, and talent: the three capitals behind model progress
Nathan frames AI progress as a function of three forms of capital—compute, data, and talent—and argues that the frontier is largely determined by how well labs can assemble and deploy them. He also highlights that organizational execution (“org charts”) is an underappreciated differentiator in who advances fastest.
- •Model progress is best explained by compute, data, and talent rather than a single breakthrough
- •Frontier labs typically lead because they have more of all three capitals
- •Organizational structure and execution materially affect research velocity
- •Competition (not just research) is what continually drives the frontier forward
- 2:56 – 3:34
Why frontier labs lead: culture, bold bets, and paradigm-setting moments
The conversation contrasts OpenAI and Anthropic’s cultures and strategic strengths: OpenAI’s knack for reframing the field with scaling/inference plots, and Anthropic’s long focus on practical capabilities like coding. Nathan argues these labs win by making expensive, early bets that others consider impossible.
- •OpenAI and Anthropic are described as the clearest “frontier” employers from the outside
- •OpenAI’s inference-scaling framing is cited as a paradigm-shifting moment
- •Anthropic’s sustained investment in coding is linked to its later product success
- •Frontier leadership comes from willingness to spend heavily and take early risks
- 3:34 – 8:51
Why Chinese labs release open models: distribution strategy under compute constraints
Nathan explains that many Chinese AI companies use open releases as a practical go-to-market strategy to gain visibility and adoption outside China. He emphasizes that they are talent-rich but compute-constrained, which shapes their incentives and the “fast follower” dynamic.
- •Chinese labs are technically strong and highly motivated to matter globally
- •Open model releases can shortcut distribution and brand recognition
- •Their dynamic is often catching up rather than creating new paradigms first
- •Compute scarcity is positioned as their biggest structural disadvantage
- 8:51 – 14:59
Compute-starved and the distillation debate: what distillation really explains
Distillation is discussed as both an economic substitute for compute and an often-misunderstood explanation for fast catch-up. Nathan argues classical distillation (SFT on strong model outputs) doesn’t map cleanly onto modern reasoning/post-training, and the evidence suggests large uncertainty about how much it helps today.
- •Compute constraints make distillation attractive as an indirect way to “buy” compute via outputs
- •Older post-training relied heavily on supervised fine-tuning on teacher completions
- •Modern reasoning training is more RL-heavy and harder to replicate via simple distillation
- •Open literature suggests it’s difficult to get large gains just from reasoning traces
- 14:59 – 19:29
Is distillation just competition? APIs, security tradeoffs, and enforcement limits
Using Jensen Huang’s quote, the episode reframes distillation as an inevitable competitive force rather than straightforward “robbery.” They discuss why offering powerful APIs makes extraction hard to prevent, why product-only access changes the calculus, and why policy enforcement is murky when violations look like ToS breaches.
- •Distillation is likened to ordinary competitive reverse-engineering
- •Reasoning tokens and API exposure can enable large-scale trace extraction
- •Moving access from APIs to products/enterprise agreements can raise the barrier
- •Government intervention is complicated: ToS violations aren’t always enforceable like contracts
- 19:29 – 25:02
Open vs closed on different timescales: integrated flywheels vs slower ecosystems
Nathan argues closed labs compound faster because they integrate model building, inference efficiency, distribution, and revenue capture. Open models, by contrast, rely on a fragmented ecosystem (builders, inference hosts, post-trainers), which slows feedback loops—even if ownership and customization benefits are real.
- •Closed labs capture faster feedback due to integrated product + inference + revenue
- •Open ecosystems have slower information flow from customers back to base model builders
- •Economies of scale in serving and margins advantage integrated frontier labs
- •Open models win where private data, on-prem needs, and ownership outweigh raw frontier performance
- 25:02 – 28:14
Frontier progress is really about data: RL environments, buying signal, and ROI
A central claim is that non-verifiable domains (healthcare, legal, bio, knowledge work) improve mainly through expensive, targeted data acquisition and RL environments—not magic generalization. They discuss the booming environment/data supply chain and why it’s both effective and messy.
- •Big gains in messy real-world domains often come from buying/building high-quality data
- •RL environments are a major spending category because labs see measurable training signal
- •Domain progress is differentiated from verifiable domains (math/code) where scoring is crisp
- •Synthetic data is pervasive, often human-guided, and moving faster than hand-built worlds
- 28:14 – 31:33
Data supply chain problems: reward hacking, low quality, and operational security gaps
They dig into why the environment boom creates perverse incentives: vendors may optimize for shipping worlds rather than behavioral rigor, leading to reward hacking and flawed evaluations. Nathan notes significant operational security weaknesses in data pipelines and expects multi-year maturation toward higher quality and ROI.
- •Buying data at scale involves lots of misses; quality is uneven and often flawed
- •Poorly specified tasks and buggy environments can induce reward hacking
- •Vendors may prioritize revenue/world-building over robust behavioral measurement
- •OpSec gaps (contractors, leakage, careless handling) are common in today’s data industry
- 31:33 – 35:05
What Trillium will build: open-source, high-leverage environments and infrastructure
Nathan outlines Trillium’s mission as building and releasing open environments and tooling that are expensive enough to be neglected by academia but foundational for progress and safety research. He cites terminal-task environment work (prompted generation + bug filtering) as an example of scalable, synthetic environment creation that can be openly shared.
- •Trillium aims to produce open RL environments and other high-leverage research infrastructure
- •Environment creation can be systematic: generate tasks, diversify prompts, filter bugs via rollouts
- •Open releases are rare because profit incentives encourage selling data instead
- •Shared environments/trajectories become “units of study” for the broader research community
- 35:05 – 39:45
Open collaboration realities: contributions, noise, and the ‘agent spam’ problem
They discuss the promise and challenges of open collaboration—high-quality distributed contributions exist, but AI agents also generate low-quality issues and “fake results” that burden maintainers. Nathan argues the bigger barrier is the long ramp-up time required to gain context and contribute meaningfully.
- •Open projects can attract strong contributors globally, including independent researchers
- •AI agents can also flood repos with low-signal issues and misunderstood ‘findings’
- •Becoming useful in a new research area still requires weeks/months of real context-building
- •Maintainers increasingly need filtering and process to manage contribution quality
- 39:45 – 44:25
Demystifying RSI: making recursive self-improvement legible with open stacks and logs
Nathan argues RSI has undue mystique and proposes that open labs can reduce fear by documenting concrete workflows: tasks chosen, tools used, trajectories collected, and failure modes encountered. The goal is to show policymakers and the public what automation actually looks like in practice and how it changes over time.
- •RSI is often discussed abstractly; open documentation can ground the debate
- •Trillium aims to share tools/configs/trajectories and report what models get stuck on
- •Transparency can help policymakers understand real bottlenecks and failure modes
- •Multiple groups (e.g., OpenRSI) pursuing openness is framed as healthy pluralism
- 44:25 – 49:47
Can open source pre-train? Feasibility, ladders, and the harder mid/post-training frontier
Nathan argues pre-training is more accessible than many assume—good base models don’t require $10B—though frontier scale helps. The harder, less-understood battleground is mid-training and post-training (reasoning behaviors, RL cycles), where iteration speed and expertise matter and open recipes lag.
- •Pre-training tools and methods are widely accessible; ‘pre-training ladders’ help test scaling
- •Internet-scale data is still the first approximation; architecture/GPUs drive many gains
- •New entrants can still train solid base models; the challenge is landing a usable product
- •Mid-training/post-training (reasoning + RL) is the Wild West and often harder than pre-training
- 49:47 – 54:29
Harnesses and agent swarms: why open can catch up, and what stays defensible for closed labs
They discuss whether proprietary harnesses and agent-swarm coordination systems create durable moats. Nathan predicts frontier labs will pioneer these approaches, but the knowledge will diffuse; defensibility will come more from inference efficiency, infrastructure, and serving cost than from secrets that remain hidden forever.
- •Models are being pressured to work across many harnesses, reducing closed-only advantages
- •Agent swarms/coordination will emerge first in frontier labs due to compute needs
- •Breakthrough ideas tend to leak socially; durable advantage is more about execution + infrastructure
- •Frontier labs likely win on cheapest intelligence per watt/score, while others fill specialized niches
- 54:29 – 58:48
Managing risk without fast-takeoff panic: capability vs preparedness gaps
Nathan argues that while fast takeoff via RSI isn’t the base-case given exponential resource demands, real-world risks are already here and diffusing from closed to open over time. He’s more concerned about inadequate societal and cybersecurity preparation than about a sudden discontinuity—and believes more open science increases trust and readiness.
- •Scaling laws imply exponential compute for linear gains, creating practical limits and human bottlenecks
- •Even without fast takeoff, risks diffuse as capabilities move from closed to open ecosystems
- •Preparedness (cybersecurity, infrastructure, governance) lags capability improvements
- •More transparency and scientific scrutiny can help society calibrate and respond effectively
