By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Latent SpaceNext 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
At a glance
WHAT IT’S REALLY ABOUT
How inference gets 10× faster: caches, speculators, systems, networking
- The discussion walks through what happens inside a modern inference platform when handling extremely long prompts, focusing on cache-aware routing, disaggregated prefill vs decode, and speculative decoding to reduce latency.
- They distinguish “basic support” for newly released open models from “production-ready” support, which adds quantization, speculator training, architecture-specific runtime work, and ongoing debugging under real-world traffic.
- They explore failure modes like repeated-token loops and nondeterminism, attributing many issues to inference engines, kernel race conditions, and hardware/interconnect differences rather than model weights alone.
- They detail where large speedups come from (quantization, speculation, disaggregation, better runtimes) and how quality is preserved, including a discussion of quantization calibration and measuring fidelity via logit-distribution divergence.
- The episode broadens from LLMs to multimodal inference—arguing video diffusion is dominated by quadratic attention costs and may need sparse attention, chunk-stitching, or a shift toward autoregressive approaches to scale to longer videos.
IDEAS WORTH REMEMBERING
5 ideasLong-context requests are handled with cache-aware routing plus disaggregated prefill/decode and speculation.
He describes an inference path that first tries cache-aware routing to reuse cached prompt/KV work, then (when needed) sends the request to dedicated “prefill” GPUs to build KV cache and get the first token, hands off to separate “decode” GPUs for iterative generation, and often uses a speculative draft model to accelerate decoding before streaming tokens back.
Supporting a new model in production means much more than loading weights and generating a token.
They contrast “it runs” (easy via open-source engines) with “production-ready API” (hard), which requires quantization work, calibration to avoid intelligence regressions, training a speculator using hidden states from the base model, and building/testing infrastructure end-to-end—especially when architectures introduce novel components.
Tool calling failures are largely formatting/training issues; structured-output constraints help at inference time.
He explains tool calling problems often arise from training sensitivity and format brittleness (e.g., malformed JSON leading the model to hallucinate results), while inference-time structured outputs can constrain formatting via a state-machine/grammar approach that guarantees valid structure even if the chosen tool is wrong.
Some “model issues” are actually nondeterministic inference and kernel/hardware bugs.
He attributes repeated-token loops and other odd behaviors to inference/software/hardware interactions—e.g., different inference engines, subtle kernel race conditions, and cluster interconnect differences exposing nondeterministic failures—and notes mitigation like cutting generation after repeated tokens.
Quantization quality depends heavily on which layers you quantize—and errors can sometimes cancel.
He argues most optimizations are lossless (KV caching, speculation), while quantization is the main lossy lever; Ali Taha adds they found layer-selection strategies where quantization errors can cancel, evaluating fidelity via KL divergence between logit distributions rather than only task benchmarks.
WORDS WORTH SAVING
5 quotesSome of the GPU kernels that run GLM-5.2 within our inference engine is written by GLM-5.2.
— Ali Taha
The thing I'm always like explaining to people is the LLM is actually not capable of doing anything. It's only capable of making suggestions of what to do.
— Philip Kiely
So what's cool about working in inference, the reason that I think inference is going to be a useful place to do engineering for a long time, is that if you look at highly optimized domains like, say, finance, if you're in finance, you measure how much better you got in basis points. It's like, "Oh, I got five basis points better, like one-twentieth of 1% better." That's huge news because everything is, is so optimized. When we publish optimizations, it's, it's 20%, it's 100%— it's 200%.
— Philip Kiely
Also with, with like mega kernels, like honestly, I'm, I'm, I'm very bearish on... I'll be honest.
— Ali Taha
If you think of a sort of golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as how close are we getting to that, you know, 100% fidelity of the model.
— Philip Kiely
SPEAKERS
Ali Taha
guestBaseten inference engineer focused on multimodal/video inference, networking bottlenecks, and performance optimization for LLM serving.
swyx
hostCo-host of Latent Space and writer/commentator on AI engineering and developer ecosystems.
Philip Kiely
guestBaseten inference engineering leader known for practical work on model bring-up, quantization, and production LLM serving.
Alessio Fanelli
hostCo-host of Latent Space with an engineering/product background, guiding technical interviews on AI infrastructure.
High quality AI-generated summary created from speaker-labeled transcript.
