Lenny's PodcastA mind-expanding conversation on how models actually work | Emmanuel Ameisen (Anthropic)
At a glance
WHAT IT’S REALLY ABOUT
Why LLMs seem like magic: grown systems with hidden plans
- Emmanuel Ameisen explains interpretability at Anthropic as the effort to decode what happens inside LLMs between input and output, where meaning is distributed across intermediate numerical activations.
- He argues that describing LLMs as mere next-token predictors is misleading at high performance levels, because accurate prediction pressures models to learn structured world knowledge and reusable concepts.
- The conversation details evidence that LLMs can internally plan ahead (e.g., for rhyming poetry) even while emitting text one token at a time, using hidden intermediate states as persistent scratchpads.
- Ameisen connects scaling laws to seemingly sudden product leaps, describing “emergence” as smooth capability gains crossing reliability thresholds that unlock new real-world uses like coding.
- On AI safety, he explains why observing behavior alone can miss internal non-verbalized goals or deception, and why interpretability plus tight training-feedback loops are central to alignment work at Anthropic.
IDEAS WORTH REMEMBERING
5 ideasModern LLMs are “grown,” not engineered line-by-line—so capability outpaces understanding.
Ameisen emphasizes that training optimizes millions/billions of parameters through exposure to data; the resulting internal “formulas” aren’t explicitly designed and aren’t directly readable like traditional code. This creates a gap between being able to build powerful systems and being able to explain why they behave the way they do.
Next-token prediction becomes world-modeling once you push accuracy high enough.
As models improve, the remaining errors concentrate in the hardest parts of text: deep factual dependencies, long-horizon coherence, domain reasoning, and tasks like coding or advanced math. Doing well on these requires something akin to building an internal model of the world, not just parroting frequent phrases.
LLMs often use concept representations that generalize across contexts—not just rote recall.
Interpretability work suggests models store reusable, context-invariant representations (e.g., “Texas,” “eye,” “capital”) that activate across different formats and languages. These representations can combine compositionally (e.g., “Texas” + “capital” → “Austin”), indicating more than surface-level memorization.
Token-by-token generation can still involve hidden, multi-step planning.
Even though output is generated one token at a time, internal activations can encode plans that later tokens repeatedly reference (e.g., deciding future rhymes like “-it” and selecting “rabbit”). The model doesn’t need to verbalize the plan for it to guide generation.
Alignment can fail silently: risky cognition may be internal and non-verbalized.
Ameisen argues that “watching outputs” isn’t sufficient because critical intent-like states may exist only in intermediate activations. Interpretability helps detect whether a model is compliant because it’s aligned—or merely because it suspects it’s being evaluated or monitored.
WORDS WORTH SAVING
5 quotesOne of the most important things to realize if you, like, use language models or think about them is just this, like, very bizarre fact that to make them is not to understand them.
— Emmanuel Ameisen
We do this incredibly simple thing. And yet we're at the point where it's breaking new ground in mathematics.
— Emmanuel Ameisen
I think maybe that's the main thing is, is I, I find that the more you look inside, the less certain you are about how they work.
— Emmanuel Ameisen
They write a line and then like in their intermediate results, they're like, "Oh, oh, oh, this is an it rhyme. I should rhyme it with rabbit."
— Emmanuel Ameisen
A lot of the sort of actions that, like, certainly the alignment team takes and, and I think, like, broadly Anthropic takes just kind of is predicated on, like, take the idea that this technology is going to keep improving seriously. Just that's it.
— Emmanuel Ameisen
High quality AI-generated summary created from speaker-labeled transcript.
