By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten

From routing a 200,000-token prompt across GPUs to having GLM-5.2 profile, rewrite, and optimize the kernels serving itself, inference engineering is becoming one of the most important layers of AI.

Ali TahaguestswyxhostPhilip KielyguestAlessio Fanellihost

Aug 3, 20261h 42mWatch on YouTube ↗

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

CHAPTERS

Jump to publisher timestamps

Chapter summaries

Cold open: A model optimizing its own inference by rewriting GPU kernels
Ali describes an internal loop where GLM-5.2 profiles itself running inside their inference engine, identifies bottleneck kernels, and generates improved GPU kernels. The team then rebuilds and repeats the profiling cycle, resulting in parts of the runtime being written by the model it serves.
- •An internal GLM-5.2 endpoint runs a forward pass, collects a profile trace, and analyzes bottlenecks
- •The model generates new GPU kernels to address identified bottlenecks
- •The team iterates: re-profile, re-optimize, and ship updated images
- •Some kernels used to run GLM-5.2 in their inference engine were authored by GLM-5.2
Studio kickoff: Baseten, “Waterloo Intern,” and setting up the inference deep dive
The host thanks listeners and brings Philip Kiely and Ali Taha into the studio, with some banter about Ali’s “Waterloo Intern” handle. The conversation quickly pivots toward practical inference engineering questions.
- •The host asks listeners to subscribe and introduces the guests
- •Light discussion about the “Waterloo Intern” handle and passing it on
- •Frames the episode around real-world inference engineering tradeoffs
What actually happens with a 200,000-token prompt: caching, routing, prefill/decode split, and speculation
Philip walks through the path of a very long request through an inference provider: cache-aware routing, prefill capacity, and how cached inputs can avoid expensive work. He explains disaggregating prefill from decode and layering speculative decoding to accelerate token generation.
- •Cache-aware routing tries to send requests to replicas with useful cached prefixes
- •If no cache exists, the request goes to a prefill worker to build KV cache and produce the first token
- •Prefill and decode can be disaggregated onto different GPU pools, passing KV state between them
- •Speculative decoding can speed decode, but depends on how well the draft model matches the request type
- •Outputs are streamed back while the provider meters usage and cost
Shared APIs vs dedicated deployments: cost, reliability, and traffic-specific speculative decoding
The group contrasts pay-per-token shared endpoints with renting dedicated “by the box” deployments, especially for high-volume workloads. Ali explains why speculative decoding benefits from being tailored to a customer’s traffic and why that often requires dedicated infrastructure.
- •Dedicated deployments can be cheaper at high token volumes compared to pay-per-token
- •Reliability and isolation are key reasons to move off shared endpoints
- •Traffic-specific speculative decoding can be trained to increase acceptance rates for a particular domain
- •Dedicated deployments allow tuning batch sizing, parallelism strategy, and precision to match latency/throughput goals
- •Avoids contention from other users’ benchmarking or bursty traffic on shared endpoints
Tool calling and structured outputs: training sensitivity vs inference-time constraints
Ali emphasizes that tool calling often fails due to training and post-training sensitivity—models can mix tool JSON with other text or hallucinate results if formatting is off. Philip describes an inference-side mitigation: constraining outputs with a state machine/grammar to guarantee structure, even if it can’t guarantee correct tool choice.
- •Some customers require highly specific tool-calling formats that demand careful post-training
- •Quantization or imperfect training can break JSON/tool readability and closing syntax
- •A common failure mode is generating tool call plus extra text, then hallucinating the tool result during decode
- •Inference engines can enforce structured output using a state-machine/grammar approach
- •LLMs only “suggest” actions; external systems interpret formatted suggestions into real tool execution
From “it runs” to “production-ready”: proprietary stacks, quantization, speculator training, and retrofitting model architecture
Philip explains the difference between making a model produce tokens and delivering a production-ready API with performance and reliability. They detail the work required per model release: quantization (e.g., to NVFP4), training speculators without retaining user data, and supporting novel architectures; they also describe grafting a vision encoder onto a text-only model by training only a projector.
- •Open-source engines may support a model quickly, but production APIs require deeper integration and testing
- •Providers often re-quantize models into formats that match their target hardware and performance goals
- •Speculator training uses hidden states from the base model on representative public datasets
- •New model architecture features can require runtime support changes (they cite novel attention/architecture elements)
- •They describe adding vision by freezing the base model and vision encoder and training only a small projector layer
Reality hits: mode collapse, loop detection, and nondeterministic failures across engines and clusters
Ali describes how exposing a model to real-world traffic reveals failures not caught in internal tests, including “mode collapse” loops (repeating the same token). The group discusses mitigation (cutting off generation after repeated tokens) and why some issues appear nondeterministic—varying by inference engine, kernel behavior, cluster interconnect speed, and race conditions.
- •Production traffic uncovers edge cases that don’t appear in pre-launch testing
- •They observed cases where a model repeats a token many times; they mitigate by aborting after repeated outputs
- •Ali argues some looping is an inference/software issue rather than weights or quantization
- •Behavior can differ across inference engines and even across clusters using the same weights
- •Subtle kernel race conditions (e.g., missing barriers/synchronization) may only surface under certain timing/interconnect conditions
Quantization quality: preserving fidelity, selecting layers, and “canceling out” quantization errors
Philip frames quantization as the main lossy inference optimization and outlines the levers: format, which layers to quantize, and calibration. Ali then describes internal research claiming that quantizing more layers can sometimes yield better fidelity because errors can cancel, validated by comparing KL divergence of logits distributions against a full-precision reference.
- •Most inference optimizations are described as lossless, with quantization the key lossy step
- •Key choices include data format, layer selection, and calibration to preserve outliers
- •Ali describes selectively not quantizing certain layer types (with examples from image-model components)
- •They discuss a technique where choosing specific layers can make quantization errors cancel out
- •They evaluate fidelity by comparing logits distributions (via KL divergence) between quantized and full-precision models
Chasing 10× faster inference: stacking optimizations and understanding TPS vs latency
Philip argues inference is still early enough to see large gains (multiples, not basis points), but benchmarking is complicated by hardware, load, and prompt/output lengths. They break down how speedups stack—quantization, speculators, disaggregation, and runtime/kernel improvements—and clarify the industry’s confusing tokens-per-second terminology.
- •Inference optimizations can yield large percent gains; the space isn’t fully optimized yet
- •Benchmarking provider speed is hard due to hardware, load, and workload variability
- •They distinguish throughput tokens/sec vs inter-token latency, noting the naming confusion
- •A speedup stack they discuss includes quantization, speculation, disaggregation, and runtime/kernel upgrades
- •They describe realistic ranges: aggressive “10×” is possible under best conditions, but often 4–6× is more typical
Dynamo and disaggregation: routing KV cache, plus why local AI optimization differs from data center inference
Philip characterizes Dynamo as an NVIDIA open-source toolkit for moving information (like KV cache) around a cluster rather than a drop-in optimizer. The conversation contrasts local constraints (“fit the model, then make it less dumb”) with data center constraints (“load the model, then make it less slow”), and Ali gives an example where a local-first optimization (TurboQuant) doesn’t help on high-bandwidth GPUs.
- •Dynamo is described as a toolkit/router for coordinating data movement (e.g., KV cache), not an automatic speed boost
- •It can provide defaults, but meaningful gains require system design and benchmarking against real-world baselines
- •Local AI focuses on fitting models and often uses pruning/distillation/dynamic quantization more than data centers do
- •They note TurboQuant-style approaches can help on bandwidth-limited devices but add overhead on high-end GPUs
- •Interconnect constraints shape parallelism choices; local clusters may rely on different strategies than NVLink-centric setups
Parallelism strategies, auto-tuning, and skepticism about mega kernels
Philip and Ali explain expert parallelism versus tensor parallelism, and why pipeline parallelism tends to appear only when forced into multi-node inference. Ali describes performance tuning as an empirical search across configurations, then argues mega kernels are difficult to maintain and often lose to highly optimized modular kernels; he also claims future GPU design trends reduce the need for mega kernels.
- •Expert parallelism places experts on GPUs and reduces communication compared to tensor parallelism
- •Tensor parallelism shards weights and requires frequent collectives (all-reduce/all-gather), making interconnect critical
- •Pipeline parallelism is mainly used when models exceed single-node memory and multi-node is unavoidable
- •Auto-tuning is described as sweeping configuration choices (TP/EP/batch/etc.) on real traffic to find the best setup
- •Ali is bearish on mega kernels, arguing complexity and modular-kernel optimization often wins in production
Rubin, GPUs vs ASICs, and the real constraints: interconnect, specialization, and GPU memory math
Ali and Philip speculate about NVIDIA’s Rubin era emphasizing systems-level concerns: CPU–GPU links, GPU–GPU interconnect, and moving KV cache efficiently. They debate whether increasingly specialized GPUs weaken the case for AI ASICs, then shift to the practical ceiling imposed by VRAM and KV cache space, including why very large models may require newer hardware or KV offloading.
- •They speculate Rubin-era performance gains will increasingly come from systems/infrastructure, not just kernels
- •KV cache routing/offloading and disaggregation are positioned as increasingly central as models scale
- •They debate the value of ASICs versus GPUs becoming more specialized over hardware generations
- •They discuss how model size ceilings are shaped by available GPU memory and quantization format
- •KV cache consumes VRAM too, so long context plus huge parameters intensifies memory pressure and motivates offloading
Beyond LLMs: video’s quadratic attention wall, sparse attention tradeoffs, and autoregressive vs diffusion futures
Ali explains why video diffusion inference differs from LLM inference: smaller models, little batching, and very different bottlenecks. He focuses on attention scaling—video token counts explode, quadratic attention becomes prohibitive beyond short clips, and sparse attention can reduce compute but hurts quality—leading to a view that long-form video likely needs autoregressive approaches or a major compute breakthrough.
- •Video diffusion workloads differ: typically one request per GPU, smaller models, and different scaling constraints
- •Token counts for video are enormous even after latent compression, making attention extremely expensive
- •Quadratic attention makes longer clips rapidly infeasible without changing the approach
- •Sparse/top-k attention can reduce compute by restricting which tokens attend to which, but can degrade quality
- •Ali argues long-form video likely requires autoregressive video approaches, but good open models aren’t there yet; chunk-stitching can drift and degrade over time
Audio, autoregressive images, diffusion-for-text, and the training–inference convergence toward continual learning
They compare modality choices: speech is described as autoregressive, while video remains largely diffusion-based in today’s ecosystem; they note some image models are moving toward autoregressive or blended approaches. The episode closes on the growing overlap between training and inference—speculator training, quantization-aware recovery, inference used for training rollouts, and ideas for continual learning via KV cache compaction or lightweight weight updates—ending with broader trends like scaling demand and faster interconnects.
- •Speech is described as autoregressive, with models outputting tokens that correspond to audio waveforms
- •They discuss diffusion-for-text efforts (e.g., Diffusion Gemma) and argue APIs/marketing may need to differ from chat-style usage
- •They mention autoregressive or blended approaches emerging in image generation, but video hasn’t broadly shifted yet
- •Training and inference are converging: faster inference improves training rollouts; training is used to enable faster inference (speculators, quantization recovery/distillation)
- •Continual learning is discussed as either pushing new knowledge into weights (possibly via LoRA-style updates) or expanding/compacting KV cache to preserve usable knowledge; they end with future-looking infrastructure and scale challenges

Publisher timestamps

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.