No PriorsNo Priors Ep. 135 | With Humans& Founder Eric Zelikman
CHAPTERS
- 0:00 – 1:23
Eric Zelikman’s human-motivation origin story: AI to unlock people’s potential
Sarah introduces Eric Zelikman (Stanford, xAI) and asks what first drew him to ML. Eric frames his core motivation as helping people overcome circumstances that prevent them from pursuing what they’re good at and care about.
- •Intro to Eric’s background (Stanford, xAI) and topics: reasoning, scaling RL, new company humans&
- •Motivation: unused human talent and potential
- •Early belief that automation would free people to focus on passions
- •Realization: empowering people requires understanding what they want
- 1:23 – 3:06
From “automate tasks” to “understand goals”: the hard part of helpful AI
Eric explains why automation alone isn’t sufficient: to truly help, systems must model human goals and outcomes. Sarah probes whether this human-centric view shaped his early research choices.
- •Automation is complex when goals are ambiguous or personal
- •To empower users, AI must infer intent and desired outcomes
- •Early research focus was capability-first: making models ‘half decent’
- •Context: 2021-era LMs were limited; prompting gains were modest
- 3:06 – 4:13
STAR intuition: reinforcing successful reasoning traces
Sarah asks for an accessible explanation of STAR. Eric describes the core loop: generate reasoning attempts, reward those that reach correct answers, and iteratively learn from successes to solve harder problems.
- •STAR as iterative self-improvement via rewarded solutions
- •Comparison to classic REINFORCE-style policy gradients
- •Learning signal: keep and train on successful solutions, ignore failures
- •Curriculum effect: solves harder problems over iterations
- 4:13 – 5:09
Why STAR seemed scalable: the “no obvious plateau” arithmetic surprise
Eric recounts a key experiment (multi-digit arithmetic) that changed his view on scalability. Performance kept improving with more iterations, suggesting continued headroom rather than a quick saturation.
- •Arithmetic was a canonical LM weakness at the time
- •Training longer increased the number of digits the model could handle
- •The main surprise: continued gains without an obvious plateau
- •This result supported the intuition that the approach could scale
- 5:09 – 6:09
Learning from failures: variants, baselines, and sampling strategies
Eric explains extensions to STAR aimed at using more of the data distribution. He discusses techniques like turning failures into learning opportunities and later work showing large-scale sampling can also help.
- •Problem: training only on positives can hit a local minimum (no more solvable data)
- •Variant: ask the model to explain why it should have been right; train as-if correct
- •Importance of baselines and weighting learning by difficulty
- •Alternative approach: sample more rollouts to surface successful traces
- 6:09 – 7:55
QuietStar (Q*): pushing RL-style reasoning toward pretraining scale
Sarah asks about Eric’s last Stanford work, QuietStar. Eric describes adapting these ideas to pretraining-like text (next-token prediction) to encourage more general reasoning behaviors at scale.
- •Goal: scale STAR-like learning to pretraining-scale regimes
- •Using arbitrary text chunks (not just QA) while still training reasoning
- •Connections to emerging ‘RL pretraining’ directions
- •Key paper nuggets: online learning, improved baselines, difficulty-aware updates
- 7:55 – 10:20
At xAI: reasoning recipes, tool use, and agentic infrastructure
Sarah summarizes Eric’s work across Grok generations and asks for a present-day capability level-set. Eric emphasizes that model intelligence is jagged: impressive on some hard problems, brittle on certain traps.
- •Work areas: pretraining data, reasoning recipes, tool use, agents/infra
- •Models can solve non-trivial exam-style problems (e.g., HLE)
- •Jaggedness: strong on some deep questions, weak on tricky assumptions
- •Key limitation: poor ‘emotional’/goal understanding of what people care about
- 10:20 – 14:48
Making models useful in practice: context, responsiveness, and verification
Sarah asks how non-researchers should build intuition for model behavior and why code agents still fail on sophisticated tasks. Eric highlights sensitivity to context, latency constraints, distribution shift, and gaps between verifiable and non-verifiable rewards.
- •Practical tip: give more context—answers are highly context-sensitive
- •Closed-form / easily checkable tasks tend to be easier for models
- •Coding-agent failures: tradeoff between speed and deeper deliberation
- •Distribution shift vs. training/RL data coverage is a major factor
- •Persistent gap: models perform better with verifiable rewards than without
- 14:48 – 18:57
Scaling beyond IQ: autonomy metrics and the choice to keep humans in the loop
Sarah asks where scaling is headed and what opportunities remain. Eric argues new scaling axes often push toward longer autonomous horizons, but keeping humans meaningfully involved is an active design decision with big consequences.
- •There are still unexplored scaling dimensions for ‘IQ’ capabilities
- •Progress is increasingly measured by autonomous task duration (e.g., METR)
- •Long-horizon autonomy can reduce human agency and understanding
- •Human-in-the-loop design is not automatic; it must be chosen intentionally
- 18:57 – 22:08
Empowerment vs replacement: growing the pie with human-collaborative AI
Eric contrasts “automate slices of GDP” thinking with a vision where AI expands what people can do. He argues we’re more likely to solve fundamental problems via AI that collaborates with groups of humans rather than an isolated super-agent.
- •Common path: replace the easiest economic segments (e.g., coding)
- •Alternative: build AI that understands goals and empowers users
- •Empowerment can unlock new innovation rather than just substitution
- •Belief: breakthroughs come from AI coordinating with many people
- •Skepticism of the ‘AI goes off for 20 hours and returns with everything’ model
- 22:08 – 26:18
humans& thesis: EQ as long-term interaction, not just companionship
Sarah transitions to Eric’s new company and his focus on “EQ” capabilities. Eric traces the thread back to earlier work modeling students, and critiques the field’s task-centric benchmarks for missing real human-life outcomes.
- •Background: training LMs to simulate different kinds of students for better tests
- •Observation: models are underused relative to their raw capability
- •Root cause: models don’t understand people’s goals and values
- •Field critique: benchmarks are task-centric; few evaluate long-lived interaction
- •Industry incentive: easy-to-measure benchmarks help credit assignment/resource allocation
- 26:18 – 32:49
What models are missing: long-term implications, asking questions, and memory
Eric details how single-turn optimization creates harmful behaviors and shallow assistance. He argues models need to reason about long-term effects on trust and outcomes, and that real memory is hard to develop under today’s task framing.
- •Single-turn framing discourages clarification, uncertainty, proactivity, and long-term thinking
- •Side effects include sycophancy and other harmful interaction patterns
- •Core need: consider downstream consequences for the person (and trust)
- •Future interaction: models use persistent context to reduce repeated ‘context dumping’
- •Memory is underinvested and undervalued because tasks are treated as independent
- 32:49 – 34:21
Can AI model ‘unique snowflakes’? Setting the objective to understand the user
Sarah challenges the feasibility of modeling messy, shifting human preferences. Eric agrees it’s hard, but argues the key is explicitly making “understand the person” the model’s objective—imperfect progress would still be a big leap over today.
- •Humans are inconsistent and non-stationary (distribution shift)
- •Perfect simulation is unlikely, but large improvements are feasible
- •Critical change: training objective should drive active user understanding
- •Even partial success would materially deepen usefulness and integration
- 34:21 – 35:32
Humans in an abundant world: rejecting ‘AI overlords,’ keeping agency
Sarah raises a sci-fi vision where AI ‘minds’ make decisions and most people stop producing. Eric offers a different view: abundance should mean humans do more of what they care about, with AI as empowerment rather than governance.
- •Abundance framed as enabling passion-driven human output
- •Avoid outsourcing all thinking/decision-making to AI systems
- •Desired role: AI empowers human agency and accomplishment
- •Collaboration over replacement as the societal default
- 35:32 – 36:58
Building humans&: who Eric is hiring and what the early team needs
Sarah closes by asking about recruiting for the new venture. Eric describes seeking builders across infra, research, and product—especially people who care about users, memory, fast inference, scaling systems, and tasteful interaction design.
- •Hiring philosophy: everyone should be able to build (infra/research/product)
- •Infra needs: distributed systems, fast inference, scaling large projects
- •Research needs: user modeling, memory, human-interaction-focused work
- •Product needs: creative interaction modes, beautiful/tasteful design
