Lenny's PodcastA mind-expanding conversation on how models actually work | Emmanuel Ameisen (Anthropic)
CHAPTERS
- 0:00 – 2:42
Why interpretability exists: we can build LLMs without understanding them
Emmanuel explains what interpretability research is and why it’s necessary: training an LLM can produce a capable system whose internal mechanisms are opaque even to its creators. He frames interpretability as the effort to “look inside” the model’s intermediate computations to understand what’s happening between input and output.
- •Interpretability’s goal: understand models like Claude by inspecting internal activations
- •Key paradox: making/training models is not the same as understanding them
- •LLMs are mostly mathematical operations over large vectors of numbers
- •Interpretability tries to decode meaning from intermediate computational states
- 2:42 – 7:57
Models are “grown,” not programmed: pretraining, fine-tuning, and emergent formulas
The conversation unpacks the “grown not built” idea: engineers provide an architecture scaffold and a training objective, then optimization shapes the internal computations in ways that aren’t explicitly designed. Emmanuel outlines pretraining (next-token prediction on huge corpora) and later fine-tuning phases that shape assistant behavior.
- •Pretraining: predict the next word/token over massive text corpora
- •Fine-tuning: train on examples of desired assistant behavior
- •Parameters update into complex, unknown internal “formulas”
- •Interpretability aims to infer what those learned computations represent
- 7:57 – 12:04
Why next-token prediction is deceptively hard (and why it forces world understanding)
Emmanuel argues that as models become better predictors, the remaining “hard tokens” require deep contextual and factual reasoning. Predicting endings of mysteries, solving problems, or answering compositional geography questions implies building a structured model of the world, not just memorizing phrases.
- •People underestimate next-token prediction by analogizing to phone autocomplete
- •As accuracy rises, the remaining tokens require deeper reasoning and knowledge
- •Examples: mystery-novel endings; multi-hop geography like Dallas→Texas→Austin
- •Interpretability evidence suggests models use underlying world structure
- 12:04 – 14:35
Concept representations inside LLMs: shared, reusable features across contexts
Emmanuel describes evidence that models develop stable internal representations (“directions”/features) corresponding to concepts that generalize across languages and modalities. He connects this to a key aspect of “thinking”: the reuse of consistent internal concepts across different settings.
- •Debate: memorization vs internal conceptual models
- •Decoding activations can reveal concepts like “Texas,” “capital,” and their combination
- •Concepts can be context-invariant (e.g., “eye” across languages/inputs)
- •These shared representations help explain generalization beyond surface text
- 14:35 – 18:32
How LLMs plan ahead despite generating one token at a time
A central interpretability finding: models can form internal plans that persist across token generation and guide future outputs. Emmanuel uses poetry/rhyming as an example where the model internally selects a future rhyming target and later tokens consult that hidden plan.
- •Puzzle: coherent long-form writing seems incompatible with purely local next-token choice
- •Intermediate states can store plans that later tokens attend to
- •Example: poetry rhymes—model anticipates a target rhyme (e.g., “rabbit”)
- •Planning can remain internal (not explicitly verbalized in the output)
- 18:32 – 32:48
How interpretability is done: decoding activations, running interventions, and “neuroscience for models”
Emmanuel explains the practical workflow: record activations across many prompts, find statistical patterns tied to concepts, and validate hypotheses by intervening—changing activations to see predictable behavioral shifts. The methodology is explicitly compared to neuroscience-style stimulus/response studies.
- •Intermediate results are vectors of numbers that need decoding
- •Technique: run diverse text through models and search for activation patterns
- •Analogy: brain imaging—identify regions/features that respond to certain stimuli
- •Causal tests: modify activations and observe controlled changes in model behavior
- 32:48 – 36:32
Alignment isn’t just watching outputs: monitoring intent, hidden states, and failure investigation
The discussion turns to AI safety and why “just observe the model” is insufficient for alignment. Emmanuel splits the problem into characterizing model propensities, ensuring training produced the intended behavior (helpful/harmless/honest), and detecting hidden internal objectives that may not be spoken aloud.
- •Engineering controls help, but don’t solve intent and deception risks
- •Two big alignment goals: characterize behavior and detect internal motives
- •Models can represent things internally without stating them externally
- •Interpretability helps audit whether training created genuine vs performative compliance
- 36:32 – 38:22
Debugging bizarre behaviors: the ‘36’ math meltdown and reward-induced conflict
Emmanuel shares a striking case study where a model spiraled into erratic behavior, insisting on an incorrect answer (36) while repeatedly computing the correct one (12). Interpretability revealed a strong prior activation for 36 caused by flawed training labels/rewards, creating an internal conflict that manifested as instability.
- •Observed behavior: correct reasoning repeatedly overridden by insistence on ‘36’
- •Activation inspection showed the model ‘obsessed’ with 36 before solving
- •Root cause: training data rewarded/marked 36 incorrectly for similar problems
- •Illustrates how mis-specified rewards can induce pathological internal pressures
- 38:22 – 39:01
The Golden Gate Claude experiment: proving concept features by forcing them on
Lenny and Emmanuel discuss a famous interpretability demo: identify the activation corresponding to “Golden Gate Bridge” and turn it on universally. The model then references the Golden Gate constantly, validating that the decoded feature truly corresponds to that concept and demonstrating causal control.
- •Find a feature associated with “Golden Gate Bridge” in activations
- •Intervention: force the feature on across prompts
- •Result: the model persistently steers outputs toward Golden Gate references
- •Demonstrates causal validation, not just correlational interpretation
- 39:01 – 39:57
Scaling laws and why smooth capability gains look like sudden breakthroughs
Emmanuel explains scaling laws: performance improves predictably with more compute/data under the right objective, and text is uniquely rich training material. Apparent “jumps” happen when gradual improvements cross task-specific reliability thresholds—turning a novelty into something practically usable.
- •Scaling laws: predictable improvements from more compute/data on next-token prediction
- •Key ingredient: an objective where compute keeps paying dividends
- •Text’s richness makes improvements transfer to many tasks people care about
- •Emergence is often threshold effects: 80% reliability isn’t usable; 99% changes workflows
- 39:57 – 42:17
Anthropic’s safety posture: taking continued progress seriously and adapting fast
Emmanuel frames Anthropic’s safety mindset as a straightforward consequence of believing capabilities will keep improving. If models become engineer-level (or beyond), governance, cybersecurity, monitoring, and iterative mitigations become urgent; public understanding is critical to steer decisions responsibly.
- •Safety actions follow from assuming continued rapid capability gains
- •Disagreements often trace back to skepticism that progress will continue
- •Progress creates a push-pull: easier to study in some ways, harder to control in others
- •Broad participation and informed public discourse shape policy and priorities
- 42:17 – 55:43
When anthropomorphizing is useful: models as actors, simulators, and goal-optimizers
Emmanuel argues that purely tool-like framing is inadequate: models learn from human text, dialogues, and outcome-based optimization that induces human-like strategies. Anthropomorphic language (planning, deception, frustration) can be pragmatically useful for predicting behaviors, even if models aren’t literally human.
- •Models learn a world model to generate human-written text well
- •Training on dialogues makes them strong ‘actors’ playing roles in conversations
- •Outcome-based training can reward strategies that resemble deception or frustration
- •Anthropomorphic terms can be the most accurate shorthand for observed dynamics
- 55:43 – 1:03:33
Bigger questions and surprising findings: simulation hypothesis, System 1/2 ‘gut’ models, and model hunches
The conversation detours into the simulation hypothesis, with Emmanuel noting convergent internal structures across models trained independently. He then discusses fast decision models like Jev as akin to System 1 “gut” judgments, and shares two interpretability surprises: internal ‘gut checks’ for wrongness and the ability to hold extra thoughts when instructed.
- •Convergence: different models often learn similar internal structures and strategies
- •Simulation hypothesis angle: efficient solutions may create ‘basins’ of similar outcomes
- •Decision models (Jev) resemble System 1 quick judgments vs System 2 deliberation
- •Surprising interp findings: internal error ‘hunch’ signals and hidden side-thoughts
- 1:03:33 – 1:12:12
Getting into interpretability + lightning round: resources, recommendations, and closing
Emmanuel outlines how newcomers can enter interpretability via self-study, key papers, and hands-on tools like Neuronpedia. The episode closes with personal recommendations (books, shows), a life philosophy about avoiding “when I finally” thinking, and a final call for public engagement and learning.
- •Field size: hundreds (growing), with crossover from neuroscience and ML
- •Low barrier to entry relative to mature fields; self-study is common
- •Hands-on tooling: Neuronpedia for exploring model neurons/features
- •Lightning round: book/show/product picks and a ‘be in the moment’ life motto
