By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

The Future of AI Infra: from Kubernetes to Agent Sandboxes — Akshat Bubna, Modal CTO

From adding GPUs a year before ChatGPT to building the cloud primitives behind elastic inference, agent sandboxes, post-training, and production AI workloads, Modal has quietly become one of the most important infrastructure companies in…

Akshat BubnaguestswyxhostAlessio Fanellihost

Jul 8, 202659mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

Great conversations run long. Their best ideas don't have to.

uListen is free on iPhone, iPad, Android, Chrome, and Claude.

CHAPTERS

Jump to publisher timestamps

Chapter summaries

Why Modal is shifting from developer experience to agent experience
Akshat explains that Modal’s SDK team now prioritizes “agent experience,” arguing that the same principles that make developers productive also help AI agents operate infrastructure. He contrasts code-level configuration with having agents wade through large, untyped YAML/Kubernetes setups, and emphasizes fast iteration where changes can be observed live.
- •Modal’s SDK focus has shifted from developer experience to agent experience
- •Agents shouldn’t have to read and modify extensive Kubernetes/YAML configuration
- •Infrastructure requirements co-located with code via decorators enables “self-provisioning” runtime behavior
- •Seeing changes live in action is positioned as a key advantage for both humans and agents
- •The framing sets up the rest of the episode’s focus on primitives for AI and agent workloads
Series C context and Modal’s origin: building a new runtime beyond Kubernetes
The hosts congratulate Modal on its Series C and ask for the company’s origin story. Akshat describes how Modal’s CEO Eric started from the pain of workflow orchestration running on Kubernetes, which he characterizes as hard to manage, poor for bursty workloads, and bad for developer experience.
- •Modal’s origin is tied to improving the runtime layer for orchestrated compute
- •Kubernetes is described as difficult to manage and not designed for bursty workloads
- •Custom images and scaling behavior were central pain points in existing tooling
- •Early motivation centered on making orchestration products easier to use
- •The discussion positions Modal as an alternative substrate rather than a Kubernetes add-on
Self-provisioning via decorators: reducing YAML and lock-in concerns
Akshat explains why Modal pushed infrastructure configuration into code with decorators, minimizing YAML and making configuration more expressive and dynamic. He addresses concerns about DSLs and lock-in by arguing users still own their code, and Modal’s DSL mainly covers configuration like hardware and scaling.
- •Decorators are presented as a way to dramatically reduce YAML configuration
- •Keeping infra requirements in code makes them operable like other software artifacts
- •Concerns about a proprietary DSL are met with the argument that users still own application code
- •Modal’s configuration layer focuses on hardware selection and autoscaling behavior
- •This approach is described as consistent even as Modal expands into inference use cases
Observability becomes the key as agents write more code
As the conversation shifts to agent-driven development, Akshat argues that observability matters more than humans reading code. He notes Modal pushes investigation workflows into the CLI so agents can diagnose issues, while humans still interpret outcomes and make judgment calls.
- •Claim that people look at code less, increasing the importance of observability
- •Modal emphasizes dashboards and CLI-based investigation so agents can debug
- •Humans still need to interpret system behavior and make decisions
- •Treating code as a black box shifts focus to observable outputs and iterative prompting
- •The theme reinforces tooling designed for both agent and human workflows
What Modal is for: AI-first primitives across inference, training, batch, and sandboxes
Akshat describes Modal as a cloud platform whose primitives are built from scratch for AI applications. He lists current coverage (inference, training, batch processing, sandbox workloads) and clarifies Modal isn’t targeting always-on web server/Kubernetes-style deployments, focusing instead on specialized compute and rapid elasticity.
- •Modal is framed as AI-first infrastructure with primitives built from scratch
- •Core workload areas: inference, training, batch processing, and sandboxes
- •Modal avoids competing in always-on web server hosting, emphasizing different workload shapes
- •Specialized compute and rapid scale up/down are highlighted differentiators
- •Close customer collaboration is presented as a strategic advantage for a smaller team
Sandboxes, early agent loops, and “proto-Cognition” experimentation
The hosts recount early experiments like “Small Developer,” and Akshat connects this to demand for sandboxing from customers in 2023. He describes building sandboxes in mid-2023 and publishing an example that put an agent in a loop to iterate on itself, while acknowledging models at the time often diverged after several iterations.
- •Early user projects and customer demand pointed toward sandboxing needs in 2023
- •Modal built sandboxes before broader market awareness of agent sandboxing
- •An example looped an agent so it could iteratively modify and improve itself
- •Limitations of earlier models: divergence after ~10 iterations and weak self-correction/tool use
- •The segment frames sandboxes as foundational to agent workflows that later became mainstream
Elastic inference as Modal’s biggest use case: custom models and hard autoscaling problems
Akshat explains Modal’s main product-market fit came from elastic inference for custom models, initially staying away from LLMs. He describes serving companies across audio, video, robotics, and computational biology, emphasizing unpredictable and region-dependent traffic patterns that make autoscaling difficult across many models and regions.
- •Modal found early fit serving inference for custom (non-LLM) models
- •Examples include audio, video, robotics, and computational biology workloads
- •Traffic patterns are described as unpredictable, diurnal, and launch-driven spikes
- •Autoscaling is complicated by many models deployed across multiple regions
- •Scaling within region and offset regional cycles are highlighted as operational challenges
GPU snapshotting, RL burstiness, and scaling to massive sandbox fleets
Akshat says Modal went deeper on autoscaling by adding GPU snapshotting to speed cold starts, including capturing GPU state such as compiled model state. The discussion contrasts typical agent workloads with RL and eval-style workloads, where rollouts may require extremely bursty scaling—up to very large numbers of sandboxes quickly.
- •GPU snapshotting is used to speed cold starts by reusing GPU state
- •Autoscaling benefits apply to inference and also bursty training/rollout scenarios
- •RL workloads are described as exceptionally bursty compared to most agent workloads
- •Rollouts may require extremely large sandbox counts in very short periods
- •Modal positions itself around handling these elastic, specialized compute patterns
DeFlash and speculative decoding: focusing on accept length for multiplicative speedups
Akshat explains speculative decoding and why increasing accept length can yield multiplicative speedups compared to incremental kernel optimizations. He describes Modal’s open-sourced DeFlash as a block-based speculator and discusses how speculative decoding avoids quality drops because the larger model verifies tokens.
- •Speculative decoding uses a smaller draft model to propose tokens, then a larger model verifies them
- •Speed comes from batching verification and improving compute utilization
- •Kernel optimizations yield small gains, while accept length improvements can be multiplicative
- •DeFlash is described as a block-based speculator (predicting blocks instead of single tokens)
- •Quality is maintained because only tokens verified by the larger model are accepted
Auto Endpoints: turnkey inference with transparency and evolving draft models
Akshat introduces Auto Endpoints as a way to create optimized inference endpoints from the UI or CLI without manually touching code, while still providing full transparency by giving users the code and allowing them to “eject” into deeper customization. He also teases future work where traffic can be shadowed so draft models evolve as data distributions change, aiming to reduce reliance on support-driven optimization.
- •Auto Endpoints simplify launching inference endpoints while retaining Modal optimizations
- •Users can create endpoints via UI or CLI and still get the underlying code
- •The feature is positioned as a bridge: start turnkey, then customize as needs grow
- •Future direction: shadow traffic to adapt speculators/draft models as data changes
- •Goal is to make high performance accessible without needing direct human support
Production-grade inference beyond raw GPUs: tail latency, reliability, and scaling to zero
Akshat argues that running production inference is difficult even apart from autoscaling, citing issues like tail latency and delivery guarantees. He emphasizes Modal’s elasticity, true scaling to zero, and operational reliability, alongside a strategy of upstreaming optimizations to open source while keeping expertise in-house for edge cases.
- •Production inference is framed as hard due to tail latency and request delivery guarantees
- •Modal emphasizes true elasticity and scaling to zero as major differentiators
- •Modal open-sources and upstreams many improvements (working with other inference projects)
- •Expertise and rapid access to new performance improvements are highlighted benefits
- •Operational concerns are positioned as more valuable than just acquiring GPUs and running code
Supercloud strategy: pooling capacity across many providers and building a reliability layer
Akshat describes Modal’s approach as a capacity pool spanning many cloud providers rather than owning data centers, keeping the company capital-light and software-focused. He notes the reliability variance across providers and says Modal invested in a reliability layer so failures like GPUs dropping off the bus don’t affect user workloads, enabling access to more usable capacity.
- •Modal runs across many cloud and metal providers instead of owning data centers
- •Capital-light strategy lets the team focus on software differentiation
- •Provider reliability varies, motivating Modal’s own reliability layer on top
- •User workloads are protected from certain hardware/provider failures
- •This approach supports geographic placement needs and broad capacity access
Networked sandboxes: sidecars, outbound control, private IPv6 overlay, and RDMA clusters
The conversation shifts to networking features: sandboxes can run multiple containers via sidecars, enabling patterns like Docker Compose-style setups and egress control through proxies. Akshat describes a private IPv6 overlay network for containers in the same workspace and explains its role in enabling serverless multi-node GPU clusters with RDMA networking for distributed training.
- •Sandboxes can be a ‘pod of containers’ using sidecars to run multiple services together
- •Users want outbound networking control (e.g., proxying, logging, domain egress control, credential injection)
- •Modal supports exposing tunnels from sandboxes with optional public access and auth layers
- •A private IPv6 overlay allows workspace-scoped container-to-container addressing
- •A decorator can provision a GPU cluster with RDMA networking for serverless distributed training
Multi-node training and auto research: elastic post-training and automated inference tuning
Akshat positions Modal’s multi-node training as aimed at post-training and elastic small runs rather than massive pre-training. He and the host discuss auto research as guided hyperparameter sweeps, and Akshat shares that Modal uses an internal agent-driven system to sweep configurations, run profiling tools, and even change GPU types to find good settings automatically.
- •Modal’s multi-node training targets post-training and elastic experimentation, not large pre-training
- •Researchers still need small, elastic runs even when large clusters exist
- •Auto research is described as model-guided hyperparameter sweeping rather than architecture search
- •Modal internally automates inference tuning via an agent that runs sweeps and profiling tools
- •The approach adjusts configs and even GPU choices to converge on good setups
Agent readiness, Modal Bench, and operational scaling: capacity planning and batch tiers
Akshat says LLMs have become better at generating Modal code, and the team is exploring Modal Bench to identify where agents struggle—especially with observability and debugging workflows. The discussion then moves into capacity planning (“compute strategy”), modeling reservations and supply, and offering cheaper batch-style tiers for latency-insensitive workloads.
- •LLMs are described as increasingly able to one-shot Modal code
- •Modal Bench is intended to reveal where agents fail and guide product/CLI improvements
- •Agents struggle particularly with diagnosing failures via logs/observability without guidance
- •Growth has driven the need for proactive capacity planning and compute strategy modeling
- •Modal is building batch-style tiers for cheaper compute when latency is not critical
Where Modal is heading: real-time routing, production agents, hard guardrails, and why infra is exciting again
Akshat outlines future focus areas: supporting the model lifecycle (data prep to training to inference) and the agent lifecycle (background agents with storage, snapshot/restore, and network control). He argues for hard guardrails at the sandbox level, frames managed agents as a starting point that eventually needs specialized sandbox infrastructure, and closes with reflections on broader use cases (bio, robotics, real-time audio/video) and expanding SDK support beyond Python.
- •Future product shapes include real-time regional routing with fallbacks for audio/video and low latency needs
- •Production agents require control over storage, snapshot/restore, networking, and sometimes GPUs
- •Hard guardrails are emphasized for sandbox security rather than LLM-mediated permissions
- •Managed agents are seen as a starting point; production use pushes users toward specialized sandbox infrastructure
- •Modal is expanding SDKs (Go, TypeScript) while seeing Python remain dominant for inference/training and TypeScript for agent workloads

Publisher timestamps

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.