a16zFrom Vibe Coding to Vibe Researching: OpenAI’s Mark Chen and Jakub Pachocki
CHAPTERS
- 0:00 – 2:28
GPT-5’s core goal: reasoning by default (merging “instant” and “think longer” modes)
Mark and Jakub frame GPT-5 as an effort to mainstream reasoning and reduce user confusion about which model “mode” to pick. They explain the shift from separate fast GPT models and long-thinking O-series models toward a unified experience with more agentic behavior by default.
- •GPT-5 targets “reasoning into the mainstream”
- •Earlier split: fast-response GPT-series vs long-thinking O-series
- •Research challenge: choosing the right amount of “thinking” per prompt automatically
- •Reasoning as a prerequisite for agents and autonomy
- •GPT-5 also improves broad capabilities beyond prior o3 baseline
- 2:28 – 5:08
Measuring progress when classic benchmarks saturate: towards discovery- and economy-based evals
The hosts press on evaluation methodology as older benchmarks inch toward saturation. Jakub and Mark argue for evals that track genuine discovery and real-world economic relevance, citing competitive programming and math competitions as meaningful markers on the path to research automation.
- •Many long-used evals are near-saturated; marginal gains (e.g., 96→98%) matter less
- •RL enables targeted excellence on specific domains, complicating “general” eval interpretation
- •OpenAI feels a shortage of truly great evals for the next phase
- •Competitions (IOI/AtCoder/IMO) are viewed as real-world proxies for research talent
- •Next milestones: models demonstrating discovery and economically relevant progress
- 5:08 – 7:10
What surprised them in GPT-5: breakthroughs in hard science usefulness
Mark describes ‘light bulb’ moments from physicists and mathematicians seeing GPT-5 handle nontrivial scientific reasoning that prior models couldn’t. Jakub notes o3 as the point reasoning models became daily-useful and trustworthy for derivations, with GPT-5 extending that trajectory.
- •Surprising jump in hard-science capability (physics/math)
- •Model can compress months of student work into much shorter cycles in some cases
- •o3 marked a daily-usefulness threshold for math derivations and trustworthiness
- •Contest-style progress is small compared to expected gains over the next year
- •Internal usefulness as a key release criterion, not just benchmark scores
- 7:10 – 8:31
Roadmap: the “automated researcher” and extending reasoning time horizons
Jakub outlines the central research objective: automating the discovery of new ideas, including but not limited to ML research. They discuss time-horizon as a practical measure—moving from hour-scale competence toward longer planning, memory, and autonomy spanning much longer intervals.
- •Primary target: automated researcher (new idea discovery)
- •Avoiding self-referential focus: apply to other sciences too
- •Progress metric: how long models can reason and make coherent progress
- •Current horizon framing: ~1–5 hours of reasoning for near-mastery in some domains
- •Key enablers: long-horizon planning and stronger memory/consistency
- 8:31 – 10:18
Agency vs reliability: planning depth, consistency, and why reasoning matters
Sarah probes the trade-offs users see when adding tools and multi-step planning. Jakub and Mark argue “depth” and “stability” are fundamentally linked, and that reasoning is what allows agents to recover from mistakes, incorporate feedback, and stay on track over long horizons.
- •Observed trade-off: more tools/steps can regress quality today
- •Consistency over long horizons is the core challenge behind ‘depth’
- •Reasoning models have already extended reliable multi-step work
- •Agents require error-detection, iteration, and response to hard feedback
- •Long-horizon robustness remains a major focus area
- 10:18 – 12:11
Beyond verifiable tasks: extending progress into open-ended research and creativity
The conversation shifts from well-posed problems to open-ended domains where right/wrong is unclear. Jakub argues that as time horizons lengthen, even “verifiable” problems become open-ended via strategy selection and program design; Mark adds that OpenAI also explores extremes like creative writing.
- •Long-horizon problem solving becomes open-ended (field selection, strategy, program design)
- •Even optimization-focused research involves open-ended question selection
- •Research evaluation becomes harder as tasks become less clearly verifiable
- •Exploring ‘limits’ of open-ended generation (e.g., more creative writing)
- •Bridging verifiable reasoning with broader research-style exploration
- 12:11 – 14:44
Why reinforcement learning keeps delivering: anchoring RL in language + rich pretraining
Anjney challenges why RL hasn’t plateaued despite skepticism. Jakub explains RL’s versatility and OpenAI’s long arc: early struggles defining environments, then the language modeling breakthrough enabling a rich ‘world’ for RL to optimize within natural language, unlocking many fruitful directions.
- •RL is versatile; once the system works, many objectives become feasible
- •Early RL challenge: defining an environment anchored to reality
- •Pretraining on language created a rich, nuanced environment for RL
- •Combining pretraining + RL enabled sustained progress in reasoning
- •OpenAI sees this as among the most exciting recent research periods
- 14:44 – 15:54
Reward modeling and the enterprise mindset: expect rapid simplification and evolution
Asked how non-RL practitioners should approach reward modeling, Jakub predicts the tooling and best practices will evolve quickly and become simpler. He advises a mindset that avoids locking into today’s workflows (e.g., static fine-tuning paradigms) as learning becomes more human-like over time.
- •Reward modeling is hard today, but expected to get simpler
- •Analogy: past focus on crafting fine-tuning datasets; not the end state
- •Trajectory toward more human-like learning approaches
- •Best practice mindset: don’t assume current methods are permanent
- •Practical implication for enterprises: plan for fast-moving technique shifts
- 15:54 – 17:41
GPT-5 Codex: making reasoning intelligence usable in messy real-world coding
Mark explains Codex as translating reasoning-model ‘raw intelligence’ into practical coding help across real environments. He highlights behavioral specs (proactivity vs laziness, style) and tuning latency presets so easy tasks are fast while hard tasks get more deliberate compute.
- •Goal: operationalize reasoning intelligence for real-world coding workflows
- •Real coding is messy: environment handling and practical constraints matter
- •Soft behavior tuning: style, proactivity, avoiding ‘laziness’
- •Latency presets: low latency for easy tasks, higher for hard tasks
- •Prior generation misallocated time: too slow on easy tasks, too fast on hard ones
- 17:41 – 21:41
From “AI coding better than me” to ‘vibe coding’—and the push past the uncanny valley
Jakub and Mark reflect on models surpassing their competitive-programming abilities, invoking the AlphaGo/Lee Sedol milestone as psychologically and strategically significant. They describe modern workflows like large refactors done quickly, while acknowledging an ‘uncanny valley’ where tools help a lot but still aren’t fully coworker-grade—yet.
- •Competitions as an encapsulated test of idea generation under constraints
- •Models nearing the frontier on hardest contest problems; gap likely short-lived
- •Practical impact: large multi-file refactors become fast and accurate
- •Current limitation: ‘uncanny valley’—useful but not fully coworker-like
- •Cultural shift: younger developers see ‘vibe coding’ as the default
- 21:41 – 23:52
What makes a great researcher: persistence, hypothesis honesty, and emotional management
Prompted by ‘vibe researching,’ Jakub and Mark describe traits that matter when outcomes are unknown and failure is frequent. They emphasize persistence, honest hypothesis testing, problem selection with meaning, and the emotional discipline to persevere or pivot appropriately.
- •Persistence as a defining research trait amid repeated failure
- •Maintain clear hypotheses and be honest about progress
- •Avoid self-deception: distinguish conviction from motivated “proof”
- •Experience teaches right problem horizons and when to pivot
- •Research requires managing emotions over long time periods
- 23:52 – 27:17
Problem selection and real-world research obstacles: bugs, wrong assumptions, and rethinking
The discussion turns concrete: what derails research progress in practice. Jakub notes ‘bugs’ in both software and thinking—months-long hidden issues or skewed assumptions that invalidate experiments—where breakthroughs often come from identifying and correcting these fundamentals.
- •Conviction and truth-seeking need not conflict if you track progress honestly
- •Choose hard, important problems to sustain multi-year motivation
- •Common breakthrough driver: finding hidden software bugs
- •Equally important: ‘conceptual bugs’—wrong assumptions and skewed framing
- •Progress often requires rethinking from scratch when fundamentals are off
- 27:17 – 31:43
Hiring and sustaining a resilient research culture: mission focus, ‘cave dwellers,’ and diverse researcher shapes
Mark and Jakub describe cultural pillars: fundamental research protection, a clear frontier mission, and building a deep bench rather than relying on a few stars. They discuss hiring signals like having solved hard problems in any domain and valuing less-visible ‘cave dwellers,’ while recognizing researchers vary from idea generators to rigorous experiment drivers.
- •Retention driver: frontier mission and doing fundamental (not copycat) research
- •Organizational resilience via training pipelines and a deep bench
- •Hiring: evidence of solving hard problems across fields (physics, theory, finance, etc.)
- •Don’t over-index on visibility/social media; seek ‘cave dwellers’ too
- •Researchers differ: high-alpha ideators vs rigorous experimentalists; diversity is intentional
- 31:43 – 38:36
Balancing product and fundamental research: mandates, shared vision, and coherent long-term objectives
They explain how OpenAI avoids being whipsawed by competitor releases while still shipping successful products. The core alignment is a long-term objective—automated researcher—paired with clear mandates separating product-accountable researchers from those protected to pursue algorithmic advances, with flexibility for bottom-up exploration across modalities.
- •Protecting fundamental research space amid intense product/competitive pressure
- •Balance via delineated mandates and reward structures
- •Leadership alignment: product org buys into research vision; co-planning the future
- •Coherent program anchored on the automated-researcher goal
- •Be prescriptive at a coarse level while allowing bottom-up idea generation
- 38:36 – 53:03
Prioritization, compute as the binding constraint, academia interplay, and trust at scale
The closing stretch covers portfolio management: compute allocation across long-term algorithmic work and product needs, with a frank stance that compute remains the limiting factor. They discuss how academia and labs can cross-pollinate via hands-on implementation and persistence, how external reception influences product iteration more than long-term research, how OpenAI avoids a learning plateau, and the trust between Mark and Jakub built through early reasoning/RL work.
- •Resourcing framework: dynamic portfolio management; avoid being ‘second place at everything’
- •Compute is still the dominant constraint; skepticism about becoming ‘data constrained’ soon
- •Academia vs frontier labs: accelerated learning via implementation, intuition-building, and persistence
- •External reception: shapes product iteration quickly, but long-term research stays conviction-driven
- •Maintaining speed: constant paradigm shifts prevent a learning plateau; collaboration/trust grew from early reasoning+RL team-building
