By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Latent SpaceNext 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
Episode Details
EPISODE INFO
- Released
- August 3, 2026
- Duration
- 1h 42m
- Channel
- Latent Space
- Watch on YouTube
- ▶ Open ↗
EPISODE DESCRIPTION
From routing a 200,000-token prompt across GPUs to having GLM-5.2 profile, rewrite, and optimize the kernels serving itself, inference engineering is becoming one of the most important layers of AI. In this episode, Baseten’s Philip Kiely and Ali Taha join swyx to explain what actually happens after a new open model is released and what it takes to turn “we generated a token” into a fast, reliable, production-ready API. We go deep on cache-aware routing, disaggregated prefill and decode, quantization, speculative decoding, KV-cache movement, model parallelism, GPU kernels, and the race to make frontier models up to 10× faster. Philip and Ali explain why inference optimizations can still produce gains of 20%, 100%, or even 200%; how quantization errors can cancel one another out; why identical weights can behave differently across clusters; and how Baseten grafted a Kimi vision encoder onto GLM-5.2 without changing the underlying language model. The conversation then expands beyond LLMs into NVIDIA Dynamo, mega kernels, Rubin, AI-specific chips, local inference, video generation, diffusion versus autoregressive models, and the enormous compute barrier to generating coherent long-form video. Finally, we explore the convergence of training and inference, continual learning through persistent KV cache, and the emerging loop where models help optimize the infrastructure that runs them. We discuss:
- What happens when a 200,000-token request enters an inference system
- Cache-aware routing and reusing previously computed KV cache
- Why prefill and decode are increasingly handled by different GPUs
- When dedicated deployments become cheaper and more reliable than shared APIs
- How speculative decoding uses a smaller model to accelerate a larger one
- Tool calling, structured outputs, and what LLMs actually do
- What it takes to support a new open model on day zero
- Grafting Kimi’s vision encoder onto GLM-5.2
- Retrofitting inefficient model layers with components from other architectures
- Why models sometimes collapse into repeating the same token
- How hardware, kernels, and race conditions create nondeterministic failures
- Preserving model fidelity while making inference faster
- How quantization errors can cancel each other out
- Why inference optimizations still deliver gains of 20%, 100%, and 200%
- How optimized serving can make a model up to 10× faster
- NVIDIA Dynamo, KV-aware routing, and distributed model serving
- Speculative decoding the speculative decoder
- Why local AI is about making models less dumb while data-center AI is about making them less slow
- Tensor, expert, and pipeline parallelism across GPUs
- Hardware-aware model design, auto-tuning, and the case against mega kernels
- Rubin and why inference is becoming a systems problem
- Whether modern GPUs are evolving into programmable AI ASICs
- Why enormous models like Kimi K3 require GB300-class hardware
- Why open-source video generation still trails Veo, Kling, and other closed models
- The quadratic attention bottleneck behind long-form AI video
- Autoregressive video, real-time generation, and compounding quality drift
- Why future video systems may combine autoregressive and diffusion architectures
- Training for inference and inference for training
- Continuous post-training, deployment, evaluation, and improvement loops
- How GLM-5.2 helped optimize the kernels serving GLM-5.2
- Why faster networking could unlock dramatically faster decoding
- Continual learning, KV-cache compaction, and persistent model memory
— Philip Kiely
- LinkedIn: https://www.linkedin.com/in/philipkiely
- X: https://x.com/philip_kiely
- Inference Engineering: https://www.baseten.co/inference-engineering/
Ali Taha
Timestamps 00:00:00 Introduction and the 200K-Token Prompt 00:03:18 Dedicated Deployments, Speculative Decoding, and Tool Calling 00:11:26 Launching Production-Ready Open Models 00:19:06 Model Retrofits, Failure Modes, and Nondeterminism 00:28:22 Quantization and Canceling Errors 00:32:15 The Race to 10× Faster Inference 00:40:48 Dynamo, Speculation, and Local vs. Data-Center AI 00:50:18 Model Parallelism, Auto-Tuning, and Mega Kernels 01:00:55 Rubin, GPUs vs. ASICs, and Custom AI Chips 01:10:03 Giant Models and the Limits of GPU Memory 01:12:42 AI Video, Quadratic Attention, and Autoregressive Generation 01:21:47 Audio, Images, and Diffusion Models 01:27:32 Training, Self-Optimizing Models, and Continual Learning 01:40:06 Closing Thoughts
SPEAKERS
Ali Taha
guestBaseten inference engineer focused on multimodal/video inference, networking bottlenecks, and performance optimization for LLM serving.
swyx
hostCo-host of Latent Space and writer/commentator on AI engineering and developer ecosystems.
Philip Kiely
guestBaseten inference engineering leader known for practical work on model bring-up, quantization, and production LLM serving.
Alessio Fanelli
hostCo-host of Latent Space with an engineering/product background, guiding technical interviews on AI infrastructure.
EPISODE SUMMARY
In this episode of Latent Space, featuring Ali Taha and swyx, Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten explores How inference gets 10× faster: caches, speculators, systems, networking The discussion walks through what happens inside a modern inference platform when handling extremely long prompts, focusing on cache-aware routing, disaggregated prefill vs decode, and speculative decoding to reduce latency.
RELATED EPISODES
