No PriorsWhy Diffusion Will Win AI Inference with Inception Co-Founder and CEO Stefano Ermon
At a glance
WHAT IT’S REALLY ABOUT
Why diffusion could outscale autoregressive LLMs on inference speed
- Stefano Ermon traces his path from early Stanford generative modeling work (VAEs/GANs) to score-based methods that became modern diffusion, and explains why he believes diffusion is the next major paradigm shift for language inference.
- Inception’s thesis is that inference—not training—will dominate AI economics, and diffusion’s parallel token generation maps better to GPU compute than sequential autoregressive decoding, enabling materially lower latency and better scaling of test-time compute.
- The company claims its diffusion LLMs (Mercury) match speed-optimized frontier models on quality benchmarks while being significantly faster, and it has built custom serving and post-training infrastructure to run these models in production.
- Early customer traction is strongest where latency is product-critical (e.g., voice agents), and diffusion speed gains can reduce reliance on specialized inference hardware by delivering high throughput on standard GPUs.
- Ermon suggests diffusion may provide additional benefits beyond speed—such as improved controllability and possible data efficiency—though he frames higher-level capability differences as emergent and still uncertain at scale.
IDEAS WORTH REMEMBERING
5 ideasDiffusion is positioned as the inference-time parallelism breakthrough for language, analogous to transformers replacing RNNs for training.
Ermon argues that while transformers made *training* parallel, autoregressive decoding is still inherently sequential, making inference memory-bound and poorly matched to GPU throughput. Diffusion generation can update many tokens at once, turning inference into a more parallel, compute-friendly workload.
Inception’s wedge is “same quality, lower latency,” aimed at markets where inference economics dominate.
Inception’s internal result (first shown academically at GPT-2 scale) is parity on perplexity with much faster generation, and the company claims production models (Mercury) comparable to “fast” frontier offerings (e.g., Haiku/Flash/Mini-class) while being significantly faster. The bet is that as inference and test-time compute dominate costs, speed becomes a durable wedge.
Cracking discrete diffusion is mainly an R&D and systems problem, not just reusing image-diffusion recipes.
Diffusion was natural for continuous data (images/video) due to interpolability; text/code requires new techniques because there’s no natural ‘in-between’ token. Ermon frames their progress as a research-to-production transition: the core model may still be transformer-based, but the generative process and serving stack differ substantially.
Inference speed compounds: it improves product latency *and* accelerates RL post-training via cheaper/faster rollouts.
Ermon highlights that modern AI progress increasingly depends on generating large volumes of rollouts (for RL post-training and reasoning-style test-time scaling). Faster inference directly increases experimentation throughput and reduces the cost of RLHF/RL pipelines.
The moat isn’t only model IP—it’s the end-to-end diffusion LLM stack (serving, kernels, post-training, evals).
Inception claims key production readiness work: a custom serving engine (since vLLM/SGLang aren’t drop-in for diffusion decoding), plus infrastructure for SFT/RLHF/RL. This systems/IP layer is presented as a moat against large labs simply copying the idea.
WORDS WORTH SAVING
5 quotesThe diffusion model was significantly faster because it's diffusion, because you're outputting many tokens at the same time. We were able to generate text like 10X faster compared to the autoregressive model.
— Stefano Ermon
That kind of workload does not map well to GPUs. That kind of workload is extremely memory-bound. You're spending most of your time moving around weights across the memory hierarchy, and you're doing very little arithmetic, and that's a fundamental problem of autoregressive models.
— Stefano Ermon
If you have a model that scales better at inference time, then automatically you're gonna get better scaling during RL post-training.
— Stefano Ermon
And so that's why we decided to bet on a diffusion-based LLM because it's inherently more parallel, and the bitter lesson is that the more parallel solution is the one that is eventually going to win.
— Stefano Ermon
Whenever you train these models, you're effectively trying to identify common structure by trying to find an efficient way of compressing the data.
— Stefano Ermon
High quality AI-generated summary created from speaker-labeled transcript.
