Lex Fridman PodcastRaschka & Lambert on Lex Fridman: Why Post-Training Won 2025
Rlvr and inference time scaling, not architecture, drove 2025 AI gains. Deepseek open-weight releases showed frontier performance need not be closed-source.
CHAPTERS
- 0:00 – 1:31
State of AI framing: the “DeepSeek moment” and what 2026 feels like
Lex frames the episode around the rapid acceleration in AI research and products since DeepSeek-R1’s surprise open-weight release. Sebastian Raschka and Nathan Lambert are introduced as builders/educators with deep experience in LLM training and post-training.
- •DeepSeek-R1 as a narrative inflection point for 2025→2026
- •Acceleration across research + product deployment
- •Episode intent: technical depth without dumbing down
- •Guest backgrounds: Raschka (from-scratch books), Lambert (post-training/RLHF)
- 1:31 – 4:00
China vs US: who’s “winning” the AI race, and what “winning” means
The conversation distinguishes technological leadership from distribution, budgets, and hardware constraints. They argue there’s unlikely to be a permanent winner because ideas diffuse quickly via researcher movement, but compute and organizational culture create real advantages.
- •“Winning” depends on timescale and metric (tech, product, influence)
- •No lab has exclusive access to ideas for long; people rotate between labs
- •Budgets/hardware and execution culture matter more than secrecy
- •China’s open-weight strategy builds global influence while US dominates paid usage
- 4:00 – 21:38
Which frontier model is ahead: Claude vs Gemini vs ChatGPT vs Grok (and why hype misleads)
Nathan describes waves of hype (Gemini 3, then Claude Opus 4.5) and how brand, interface, and “muscle memory” drive adoption as much as raw model quality. They discuss speed vs intelligence trade-offs, routing, long-context performance jumps, and why users often “stick until it breaks.”
- •Claude’s coding-centric culture vs Google’s scale vs OpenAI’s product agility
- •Speed/intelligence toggles and routers change cost + UX dramatically
- •Interface, memory, and habit (not just benchmarks) shape user choice
- •Long-context “needle in haystack” as a differentiator and moving target
- •Why Chinese models are less used directly: platform + serving trade-offs
- 21:38 – 28:29
Best AI for coding in 2026: Cursor, Claude Code, Codex, and “programming in English”
They compare IDE copilots and agentic coding tools, emphasizing the difference between micro-managing diffs and guiding systems at a higher design level. Claude Code is highlighted as unusually strong even when using the same underlying model, suggesting tooling/UX and orchestration matter as much as model weights.
- •Codex VS Code plugin as a controlled, “sweet spot” assistant
- •Claude Code as more agentic and often better at end-to-end coding workflows
- •Cursor vs Claude Code: diff-centric iteration vs macro-level guidance
- •“Programming with English” as a skill and mindset shift
- •Grok 4 Heavy and Gemini as complementary tools for debugging/quick tasks
- 28:29 – 37:14
Open-weight landscape: who’s shipping, what stands out, and why openness is exploding
They inventory the fast-growing set of open-weight models across China, the US, and Europe, and explain why open weights are strategically valuable. Licensing terms, security concerns, and the ability for others to host/serve models (and use others’ GPUs) drive adoption.
- •Notable open models/orgs: DeepSeek, Qwen, Kimi, MiniMax, Z.AI/GLM, Mistral, Gemma, Nemotron, OLMo, SmolLM, Marin, LM360
- •Open weights enable customization, fine-tuning, and enterprise adoption without sending data to a vendor
- •China’s permissive licensing can be friendlier than US big-tech licenses
- •Distribution strategy: open models get global usage even when APIs won’t be trusted
- •OpenAI’s incentive: offload inference demand to users’ compute via open releases
- 37:14 – 48:06
Transformer evolution since 2019: MoE, attention variants, KV cache pressure, and “still GPT-2-ish”
Sebastian argues modern frontier LLMs remain close relatives of GPT-2: mostly the same transformer decoder with a growing set of pragmatic tweaks. The big architectural story is efficiency—especially attention/KV-cache optimizations and Mixture-of-Experts (sparsity) to scale capacity without proportional compute per token.
- •Core lineage: Attention Is All You Need → decoder-only GPT family
- •MoE explained: sparse experts + router to increase capacity efficiently
- •Attention tweaks: multi-head latent attention, grouped-query attention, sliding window, sparse attention
- •KV cache as a key inference bottleneck that motivates many design changes
- •Emerging “linear-ish” attention/state-space hybrids (e.g., gated delta net inspirations)
- 48:06 – 1:04:13
Scaling laws in 2026: pre-training, RL scaling, and inference-time compute (what still scales)
Nathan defines “scaling laws” precisely (power law vs held-out token prediction) and then expands the discussion to modern scaling axes: RL training and inference-time “thinking.” They argue scaling continues to work, but economics, serving costs, and “low-hanging fruit” shift where labs invest.
- •Classic scaling law: predictable power-law between compute/data and held-out loss
- •New scaling axes: RL compute scaling and inference-time scaling (longer hidden reasoning)
- •Serving cost dominates training cost at massive user scale; incentives push smaller/efficient models
- •Gigawatt-scale clusters (e.g., Blackwell-era) likely drive the next wave of scaling
- •Why “pre-training is dead” is more vibe than practice—labs still pre-train, but optimize trade-offs
- 1:04:13 – 1:37:16
How LLMs are trained now: pre-training → mid-training → post-training (and why data quality dominates)
They lay out the modern training pipeline, how synthetic/processed data enters pre-training, why mid-training exists (e.g., long-context specialization), and how post-training unlocks skills rather than raw knowledge. The discussion emphasizes data pipelines, OCR/PDF extraction, evaluation-driven data mixing, and legal/licensing pressures.
- •Pre-training: next-token prediction over trillions of tokens; quality + filtering matter
- •Synthetic data includes reformatting/rephrasing (Q&A, summaries), not only “made-up” text
- •Mid-training: specialization phases (e.g., long context) while managing catastrophic forgetting
- •Data work is a major moat: new sources (PDFs/OCR), better filtering, regression-driven mixing
- •Legal landscape: licensing vs scraping; court cases and secrecy around datasets
- 1:37:16 – 1:58:11
Post-training breakthrough: RLVR, inference-time reasoning, and what “thinking models” changed
They explain reinforcement learning with verifiable rewards (RLVR) and why it scaled better than traditional preference-based RLHF. The chapter covers domains where rewards are verifiable (math/code), the shift to longer internal reasoning traces, and research directions like process reward models and value functions.
- •RLVR mechanics: generate → grade with verifiable signal → update policy (e.g., PPO/GRPO)
- •Why RLVR scales: harder problems + more compute keep producing gains; RLHF saturates sooner
- •Inference-time scaling: longer hidden thought/tool loops can unlock step-function capability gains
- •Debates: “aha moments,” contamination, benchmark brittleness, and what RL is actually learning
- •Next directions: process reward models vs value functions; harder domains beyond math/code
- 1:58:11 – 2:21:15
How to get into AI: from-scratch building, reading codebases, and finding narrow research wedges
Sebastian advocates implementing small LLMs locally to understand internals, then reverse-engineering production models via configs and reference outputs. Nathan discusses how newcomers can contribute despite compute limits by going narrow (evaluation, tooling, niche questions) and leveraging proximity to the research community.
- •Start by building a small transformer/LLM on a single GPU to learn fundamentals
- •Use configs + reference implementations (e.g., Hugging Face) as unit tests for your own builds
- •Why Transformers library is hard to learn from directly: scale + generality + production differences
- •Research strategy: pick narrow areas with few papers and unanswered questions
- •Low-compute impact path: evaluation that frontier labs adopt
- 2:21:15 – 2:28:46
Work culture, 996, and the Silicon Valley bubble: competition’s upside and human cost
They discuss the intense pace of frontier labs and startups—often approaching “996” work culture—and how competition drives rapid progress while risking burnout. The conversation broadens to Silicon Valley’s echo chamber dynamics and the importance of grounding in wider historical and societal context.
- •996 defined: 9am–9pm, 6 days/week; “72+ hour weeks” pressures
- •Why tight culture + leapfrogging competition increases output—and burnout risk
- •Academia vs frontier labs: credit, autonomy, mentorship vs pay and closed work
- •Silicon Valley as a productive but distorting bubble; importance of leaving the echo chamber
- •Bubbles: productive build-out vs speculative financial bubble risk
- 2:28:46 – 2:38:45
New research directions: text diffusion, tool use/agents, recursive calling, and open vs closed integration
They explore alternatives to autoregressive transformers—especially text diffusion—and where it might win (fast long outputs like code diffs). They also examine tool use as a route to reducing hallucinations and enabling agents, plus “recursive” decomposition approaches that improve long-horizon tasks without changing base weights.
- •Text diffusion for text: parallel generation/denoising vs autoregressive token-by-token
- •Potential diffusion sweet spot: fast long-form outputs (e.g., large code diffs)
- •Tool use: reduces hallucinations by outsourcing facts/math to search/interpreters (but adds trust/security issues)
- •Recursive language modeling: break tasks into subcalls and stitch results for better long-context performance
- •Open vs closed tooling gap: closed labs can deeply integrate specific tools; open weights must generalize across tools
- 2:38:45 – 2:50:22
Long context, memory, and continual learning: context stuffing vs weight updates vs agent-managed compression
They distinguish continual learning (updating weights) from in-context learning and discuss practical “memory” implementations today. The chapter covers cost constraints, LoRA adapters, attention/hybrid architectures for cheaper long context, and the idea that agents will learn to manage/compact context as an action.
- •Continual learning vs in-context learning: adapting weights vs reloading context each time
- •Why per-user weight updates are economically hard; global updates already happen (model versions)
- •Memory today is mostly retrieval/context injection; LoRA as lightweight personalization with trade-offs
- •Long-context scaling is compute + data constrained; hybrid attention/state-space mixes seek a Goldilocks zone
- •Agentic compaction as a learnable policy: keep performance while minimizing token budget
- 2:50:22 – 2:59:31
Robotics, world models, and safety: where LLM progress helps—and where embodiment is harder
They briefly pivot to robotics and world-model ideas, arguing LLM/tooling advances can accelerate robotics research and infrastructure but embodied deployment has harsher safety requirements and customization needs. They’re bullish on structured industrial automation and self-driving, and more bearish on consumer home robots learning on the fly.
- •World models as simulation/structured intermediate correctness (beyond final-answer rewards)
- •LLMs accelerate robotics engineering via better coding + transformer training infrastructure
- •Safety is far less forgiving in embodied systems than in chat interfaces
- •Industrial robotics and warehouse automation are more tractable than in-home general robots
- •Customization/on-the-job learning remains a key bottleneck for real-world variability
- 2:59:31 – 4:25:12
AGI timelines, jagged intelligence, and the real question: when does economic impact show up?
They debate definitions of AGI/ASI and critique single-milestone narratives like “superhuman coder,” emphasizing jagged capabilities and system integration as limiting factors. The discussion shifts to measurable impact: tool use/computer use reliability, product integration, and whether we might see plateau-like outcomes in broad consumer value despite ongoing technical gains.
- •AGI definitions vary; “remote worker” is a common but imperfect anchor
- •Jagged intelligence: superhuman in pockets, weak in others; humans fill gaps via workflow design
- •Computer use demos remain weak; API-based tool integration may scale better than screen control
- •Economic impact likely hinges on reliable agents + tool use at low error rates
- •Cynical scenario: costs + hard tool-use problems limit GDP-scale impact even as models improve
