The Future of AI Infra: from Kubernetes to Agent Sandboxes — Akshat Bubna, Modal CTO

He argues that agents shouldn’t be forced to read and edit large collections of Kubernetes manifests, especially untyped YAML, when infrastructure intent can be expressed directly next to code via small decorator changes that provision the right runtime.

He says Kubernetes is “hard to manage,” not built for burstiness, and poorly suited to specialized environments (different images, accelerators), while Modal’s runtime was designed around these workload shapes.

He describes Modal’s biggest current use case as elastic inference for companies deploying many custom models across regions, where traffic is unpredictable (diurnal patterns, launches) and scaling needs to happen quickly within specific regions.

He says GPU snapshotting lets Modal capture GPU state (including compiled model state) so cold starts become much faster, enabling deeper autoscaling and “true scaling to zero.” He also mentions extreme bursts such as needing “100,000 sandboxes in seconds” for RL rollouts.

He explains speculative decoding as using a smaller draft model to propose tokens and a larger model to verify them, emphasizing that increasing the “accept length” yields multiplicative speedups, while kernel improvements tend to be incremental; he notes no quality drop because the larger model verifies tokens.