The Future of AI Infra: from Kubernetes to Agent Sandboxes — Akshat Bubna, Modal CTO

Akshat Bubna 0:00 We've actually changed our SDK team to think about agent experience instead of, uh, developer experience. And, and we think that the same benefits that apply for DX also actually apply for AX, which is, uh, why would you have an agent read through hundreds of Kubernetes files and, like, write YAML that's not even typed when it can basically make a couple changes in a decorator and it gets this sort of self-provisioning runtime of, uh, being able to see its changes live in action?

swyx 0:29 Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that Subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring The In Space to you each and every week. If you do it, I promise you, we'll never stop working to make this show even better. Now let's get into

swyx 1:14 it. [upbeat music] We're here with Akshat of Modal, CTO-

Akshat Bubna 1:21 Mm-hmm

swyx 1:21 ... of Modal, together with Vibhu. Congrats on your Series C.

Akshat Bubna 1:25 Thank you.

swyx 1:26 Uh, your party yesterday was amazing. [chime] [laughs]

Akshat Bubna 1:29 Yeah.

swyx 1:29 From all the photos and all the swag.

Akshat Bubna 1:31 We, we had a bunch of art installations, which, uh, is kind of fun seeing, like, our products on pedestals next to, like, Rodin.

swyx 1:39 [laughs] Very nice. Very nice. Um, when you started, it was not the GPU inference company. I mean, m-maybe it was in your mind. Take us back to the origin story.

Akshat Bubna 1:53 Uh, I actually first met Eric, who's the CEO, through an investor. And, um, back then, Eric was already thinking about building, uh, a new kind of runtime, and he, he got there thinking through why are workflow orchestration products so hard to use. It's because you have to run them on Kubernetes. Kubernetes is hard to manage. It's not built for burstiness and, um, custom images-

swyx 2:17 Yeah

Akshat Bubna 2:17 ... and it has a terrible developer experience.

swyx 2:19 And I'll, I'll inter-interject for-

Akshat Bubna 2:20 Yeah

swyx 2:21 ... pe- for listeners, uh, who are new. Uh, we interviewed Eric two years ago, and there's a bit more of the story there from Spotify and all those things.

Akshat Bubna 2:28 Mm-hmm.

swyx 2:28 And I-I actually came across Eric through Data Console because he, he did that talk on the, the sort of serverless container stack that, that you guys did, which was, like, that was m-my first, like, "Okay, I need to take Modal very seriously" moment.

Akshat Bubna 2:40 Uh-

swyx 2:40 But it's still very unclear, like, do I actually need all this for just my data pipelines? [laughs]

Akshat Bubna 2:47 Y-yeah. I mean, initially, what we were thinking about was if we build a better runtime, it's a very useful primitive in itself. It's, um, uh, there, there's a lot of things that, um, get solved by serverless functions. Like, you can do, uh, ATL stuff, you can do job queues, you can do all this, like, bursty processing, which it turns out every company had needs for. Um, but then we also were thinking about this as, like, um, this is a primitive that we can build a whole collection of products on, um, uh, which are very verticalized. So perhaps data engineering would have been the first one, but we were thinking about inference. Back then, it was more classical inference, like computer vision stuff and running XgBoost and whatnot. But we, we added GPUs to the product a year before ChatGPT came out.

swyx 3:33 Nice.

Akshat Bubna 3:34 Uh, we just didn't think it would be that big of a deal.

swyx 3:36 Yeah, just like add A100.

Alessio Fanelli 3:37 Was there any, like, early key problem that really sparked off why you built it?

Akshat Bubna 3:42 Yeah. Primarily, it's just, um, none of the tooling that was out there was built for, um, one, a really great developer experience. And also, there's the general trend of, um, a lot of the workloads that we were seeing were very... We-- This is... I wish there was a better word for it, but compute heavy. Like, they need, um, um, one, like, need a lot more resources, so you need to burst up and down a lot versus, like, Kubernetes designed for, like, slow scaling and, uh, more for, like, web server use cases. And also, there's just a lot more specialization in, like, what kinds of environments these workloads run in. Like, we had sometimes they need accelerators, sometimes they need different kinds of images. And this is just, like, a consistent thing that we saw across a lot of companies. That would be the, the next step.

swyx 4:27 Yeah. Yeah. Be, be nice. I don't know how much this factored into the early story, but I wrote a post when I was at Temporal about infrastructure, software-defined infrastructure or something like that.

Akshat Bubna 4:36 Yeah, the self-provisioning workflow.

swyx 4:37 Self-provisioning. Yeah. Yeah.

Akshat Bubna 4:38 Yeah.

swyx 4:38 I can't even remember my own post.

Akshat Bubna 4:40 [laughs]

swyx 4:40 And then you put me on the landing page.

Akshat Bubna 4:42 Yeah, we, we really liked-

swyx 4:43 [laughs]

Akshat Bubna 4:44 ... uh, the term, a-and so we, we stole it.

swyx 4:46 Because you had the insight that everything can just be in decorators next co-located with the code, right?

Akshat Bubna 4:52 Yeah.

swyx 4:52 Was that a big part of the original story?

Akshat Bubna 4:53 Yes.

swyx 4:53 Or it was just like a DX layer?

Akshat Bubna 4:55 That, that, that was, uh, really important because we really didn't want people to spend, um, so much time, uh, writing YAML. And it seemed like you could really condense the surface area of what you're doing, uh, put it in code so you can actually operate on it, just like you operate on other code, and, like, build stuff that's more expressive and dynamic. A-and so yeah, that was, that was always a very important part.

swyx 5:19 Then the pushback is, this is a DSL.

Akshat Bubna 5:21 Yeah.

swyx 5:21 It's your closed source. Uh, I am locked in to Modal.

Akshat Bubna 5:26 Yeah. We, we, we never really got pushback for that because the nice thing about Modal is you can bring whatever code you have. And sure, the DSL is at the sort of configuration layer for, uh, what hardware you're using, how you're scaling things up, but you still own the code.

swyx 5:41 Mm-hmm.

Akshat Bubna 5:41 Uh, and that's, that's been an important, um, part of our story, but even as we do inference now.

swyx 5:46 Yeah.

Alessio Fanelli 5:46 How much of d-uh, do you think still stays the same today? Like, if you were to build something today, DevX, obviously very important, but I feel like, you know, a lot of this has kinda been changed with just hook it up to an agent, have cloud code, have Codex implement a tool. Um, there's very agent native primitives that are kinda different than if I'm doing this myself, right?

Akshat Bubna 6:08 We've actually changed our SDK team to think about agent experience instead of, uh, developer experience. And, and we think that the same benefits that apply for DX also actually apply for AX, which is, y- why would you have an agent read through hundreds of Kubernetes files and, like, write YAML that's not even typed when it can basically make a couple changes in a decorator and it gets this sort of self-provisioning runtime of, uh, being able to see its changes live in action? Um, yeah, it just seems from, from the customers we talk to, um, they actually find the model is way faster to use for agents versus operating on a different substrate. Yeah, because, like, you c- you can, again, you co-locate the infrastructure requirements to the code that runs it. Yeah. Well, the, the, the negative thesis now is that nobody's looking at their code anymore, so there's no point. [laughs] Yeah, I mean, people aren't looking at code. Uh, one thing we actually still see is, is really important is observability. Yeah. Like, how good is your dashboard? And of course, like, we have, um... we push a lot of it to the CLI so the agents can do their own investigation, but you still need humans to go interpret what's going on and, uh, you know, make judgment calls and whatnot. Uh, and that's, I feel like, um, maybe more important now than looking at the code itself. Yes, because like, you know, it's kinda... you, you can try to treat the code as a black box and then use, uh, sort of see the observable action that comes out of it and then just prompt a change. [laughs]

Akshat Bubna 7:35 Yeah. So I, I think actually it, it takes a bit of restraint to not specialize, to say, "I want to ship a new primitive," and then just, just be general purpose. People ask you, "What are you for?" You're like, "Uh, I don't know. We can do this, we can do that." Uh. [laughs] Well, I mean, I'd be curious to see, you know, like, okay, if, if we were to ask you, like, "What is Modal for even at a high level?" There's a lot you guys do, sandboxes, GPUs, uh, everything. How, how do you answer? Modal is a cloud platform that's built for, uh, where we've built the primitives from scratch for AI applications. Uh, and right now it basically covers, um, inference, training, batch processing, uh, and sandbox workloads. But we're building a lot more and- I noticed you didn't say web server, so there, there is still a role for, like, the always-on large scale Kubernetes type things. Yeah, absolutely. We're, we're not trying to compete with the renders of, of the world, uh, because yeah, we, we think the differentiator for us is the, um, are the workloads that need specialized compute, need to scale up and down a lot. Um, yeah, and they're, they're, they're just shaped differently. I think you're building a lot of it alongside the startups, right? They're innovating quite a bit, um, even in your, like, latest blog post. Like, even, even in the Series C, the, the customers that you mention here, uh, the Cognitions, Decagons, ramps and whatnot, they're, they're innovating with you, right? And that's not something AWS is doing directly with. Yeah, absolutely. I, I think, I mean, this is again, classic. We're a small team. We can move really fast. Uh, our engineers are working with our customers and figuring it out. Yeah. So my first week at Cognition, I walked in, there was someone wearing a Modal shirt. I was like, "What are you doing here?" They're like, "Yeah, I just... I, I am em-embedded [laughs] inside of Cog." [laughs] Yeah, I think that was Peyton. We, we sent him over- Yeah. [laughs] ... because, uh, the latency of communication was too high otherwise. [laughs] Um, yeah, y- you know, distributed node, you have to put-- you have to place one and co-locate. Mm-hmm. Yeah. So actually, so I had a, I had a direct personal experience, right? So I worked on Small Developer three years ago. Uh, it was inspired by Cloud One. I think you onboarded me at some point, like, just before, and I was like, "Oh, like, I need some bursty compute." Like, I was just gonna try using Modal, and it was a, it was a pretty pleasant experience. Uh, apparently, I showed up in the board meeting, [laughs] like the analytics. Yeah, you blew up on Hacker News and, and- Yeah ... we got a big traffic spike. Uh, I actually, I, I think the way you, you used Small Developer was Modal functions for, for running stuff, uh, which, which was... Like, the, that was a good use case. Um, that was- Yeah. So to me, that was proto-Cognition. Right. If only I had, like, stuck to it. Like, that, that was like, if you say draw the tech tree out- Absolutely ... it was like, yeah, like probably this will happen. Yeah, like it was so close. You were basically building a product and- I just didn't realize. [laughs] Um, but, but the, the funny story there is at the same time, uh, we were talking to a bunch of customers who needed something like

Akshat Bubna 10:28 sandboxing. Yeah. This is like 2023. Yeah. Yeah. Yeah. Uh, so we built our- You should do some new API right after that. Yeah. Yes. Like, we built sandboxes in May of 2023 before anyone was even knew this was gonna be a thing. And the first example we published was, uh, we took Small Developer- Small Developer ... and put it in a loop, uh, so the agent can iterate on itself. Uh- Loops are hot these days. The looper. Yeah. [laughs] Lo- loops in... When was this? 2023? Yeah. A small check. Yeah, yeah. It's like mid-2023. I mean, so the... I mean, the, those sort of obviously for listeners, like, the, the problem was the models are not built for any of this, right? Mm-hmm. Like, you, you're just trying to like... They're not post-training to understand, like, you know, looping and like self-correction and, and tool calling was there, but like also not that great. Yeah. [laughs] I, I don't remember if you used tool, tool calling in this one, but yeah, the models would just diverge after like 10 iterations and not produce anything meaningful. Yeah. But like then, so I mean, okay, like now talking to myself three years ago, the answer is- Of course it will get better ... collect all the failures, build benchmark, and then collect all the, you know, examples, build the RL environment- Right ... sell it for like $10 billion to Meta. [laughs] Uh, and then, and then also train a model and then sell that for $60 billion to, to Elon. And this is- [laughs] Yeah, of course. The money machine. Like it's like, it's actually about that hard. [laughs] I mean, it, it's hard to have that kind of inherent conviction that this, this stuff will get that much better. But in retrospect, it's so fucking obvious. [laughs] Fair enough. Like what else were we doing back then? I don't know. Um, anyway, yeah. So, so, so I mean, this, that, that was the start of your sandboxing- Mm-hmm ... journey, right? Uh, I feel like it didn't blow up, blow up until like- Last year?

Alessio Fanelli 12:04 Yeah.

Akshat Bubna 12:05 We-

Alessio Fanelli 12:05 So there was like a couple years of quietness.

Akshat Bubna 12:06 Exactly, yeah. We were-

Alessio Fanelli 12:07 But like very underrated product value. Like, um, my experience with Modal, Charles, before he had joined Modal, met this guy at a hackathon, and he really insisted, "We want to run some small model," you know, not hosted anywhere, and he's like, "Ah, there's this cool company, Modal. They'll like spin up a GPU sandbox, we can throw it on there. They'll take a Hugging Face link." And you know, like there's so much value just right there, right? Like instant hosting, spin it up, spin it down. It'll stay cold, but you know, we run the demo a few days later and it'll come back up, and like all this stuff in retrospect, like it's still what we needed like today.

Akshat Bubna 12:41 Yeah, I mean i- it's still needed today. Uh, obviously workload shapes have changed a lot as, um, we, we run stuff for people with really massive production scale and, uh, there it's, it's not about scaling from zero to one, but it's how do we scale really elastically, uh, from like 1,000 to 1,500 GPUs very quickly in, in a given region. It's the same shape problem.

Alessio Fanelli 13:04 Okay. So you look at, say, Cursor Composer, right?

Akshat Bubna 13:07 Yeah.

Alessio Fanelli 13:07 They had a, "We'll do RL on a model every couple hours." Um, you guys have a whole version of RL inference gym and whatnot.

Akshat Bubna 13:15 Mm-hmm.

Alessio Fanelli 13:15 When you look at workloads like that, you're basically doing train runs where you need to scale up, scale down every hour thousands of GPUs, right? Um, that's the example for we do

Alessio Fanelli 13:26 need it, right?

Akshat Bubna 13:26 Yeah. Well, also actually I'll, I'll take a step back and, um, maybe talk about like how, how people use Modal today. Uh, because our, our biggest use case actually is, uh, elastic inference. And the, the thing we first found product market fit, uh, with was inference for custom models, so we kind of stayed away from the LLM space. Uh, and we were serving companies like Suno for audio, Runway for video, robotics, uh, comp bio companies that train their own model elsewhere, but Modal is the best black box that for deployment, um, scaling to however many GPUs you need as your traffic pattern changes. And we saw all of them actually, um, w- like have a very unpredict- predict- predictable, uh, traffic pattern. Uh, it's like diurnal, it's um, some days like the company will do a launch and, you know, they'll need like, uh, way more. And it's not just one model that they deploy. They... all these companies deploy, uh, lots of different models in different regions, and so the auto-scaling problem becomes even harder because then you have to scale within a certain region. And those cycles sort of are offset, so different times you need to scale up in different regions.

Alessio Fanelli 14:34 Mm.

Akshat Bubna 14:34 Uh, so, so that's like our sort of-

Alessio Fanelli 14:36 And that-

Akshat Bubna 14:36 Yeah

Alessio Fanelli 14:36 ... you know, that, that in and of itself is a huge category. There's a bunch of inference providers which, you know, provide this Fireworks, does this as a service together, whatnot, Base10. Um, that's kind of carved into its own niche for language models at least right now.

Akshat Bubna 14:51 Yeah, I mean, the thing that we, we have actually specialized in is the auto-scaling aspect.

Alessio Fanelli 14:55 Yeah.

Akshat Bubna 14:55 Uh, because we, we, we found that i- i- it's not universally true that everyone else can auto scale, and we've gone deeper into it on the tech side by, uh, we've incorporated GPU snapshotting into the product so we can actually, uh, take the GPU state, uh, like your Torch compiler model, snapshot it, and the next cold start's way faster. And so going back to your question, it's um, that's why you need a lot of burstiness for inference, but then people also do a lot of on-demand training, like for RL stuff, your rollouts are bursty, as you said. People also do a lot of batch jobs. So we'll see, uh, a lot of companies, uh, before they have a training run, they'll need thousands of GPUs to run encoding or something like that. And I think those things are much more bursty than... I agree that agents are, are not that bursty. Uh, sandboxes are, uh, except when you're doing RL. RL is just-

Alessio Fanelli 15:42 Or eval

Akshat Bubna 15:42 ... insanely bursty.

Alessio Fanelli 15:43 Yeah.

Akshat Bubna 15:44 Yeah. Like uh, when, when you're doing, uh, rollouts, uh, you, you sometimes need 100,000 sandboxes [laughs] in seconds.

Alessio Fanelli 15:52 Yeah. I'm curious if you've seen early sparks of continual learning. There's some people, like our friends Engram recently announced this-

Akshat Bubna 15:59 Yeah

Alessio Fanelli 15:59 ... they're, they're trying to do training. That also seems like a different workload, right? If you're doing training 24/7 per se, there's a very weird dynamic of how you're using GPUs between people and whatnot, but seems like something you guys would work for.

Akshat Bubna 16:14 As you said, we're, we're fortunate to work with a number of, uh, customers at the frontier, and Engram's some of our customers. Uh, and, and they are taking the primitives we have, um, and trying to use them in very interesting ways, like continual learning. It's possible as the stuff gets better, some of that will be part of, uh, you know, our offering as well if, you know, more people need it. Um, but we're just waiting to see how-

Alessio Fanelli 16:37 Yeah

Akshat Bubna 16:37 ... how

Akshat Bubna 16:38 it shakes out.

Alessio Fanelli 16:39 Is there a, a primitive that you added after sandboxing that was the next step in the story?

Akshat Bubna 16:46 I, I, I guess we've been going much deeper into LLM inference-

Alessio Fanelli 16:49 Yeah

Akshat Bubna 16:49 ... because we realized that some of the advantages we have with like auto-scaling, again, especially in different regions and whatnot, uh, are, um, not present elsewhere. A- and the place where we had a gap was we weren't, uh, working on the model there itself, like we were a black box. And, uh, we realized that, uh, we actually can get to frontier-level model performance, um, uh, with, you know, by having great people who, who work on, uh, all this. And, uh, we've actually been open sourcing a lot of our work, uh, in terms of, um... Recently we, um, shared our work on DeFlash, which is a block-based, uh, uh, speculator, and we've open sourced, uh, all of it. So, um, you can get... By using open source DeFlash, you can get the same performance as you would with one of the proprietary providers. And the next thing we're thinking about here-

Alessio Fanelli 17:37 I, I thought this was actually-

Akshat Bubna 17:38 Yeah

Alessio Fanelli 17:38 ... an interesting blog post as well, right? Like, uh, I think in here you make a claim that, you know, not a claim, just that how, how effective speculative deco- decoding really just gets you.

Akshat Bubna 17:47 Yeah.

Alessio Fanelli 17:47 Anything you wanna point out from this around, you know, what people should know?

Akshat Bubna 17:53 Uh, yeah, absolutely. I mean, the high-level summary is, um, it would help to describe what speculative decoding is.

Alessio Fanelli 17:58 Yes. I-

Akshat Bubna 17:59 I think like we-

Alessio Fanelli 18:00 Yeah

Akshat Bubna 18:00 ... so we've covered like Eagle and all this-

Alessio Fanelli 18:02 Yeah

Akshat Bubna 18:02 ... like Hydra and all those things, but it was like two years ago.

Alessio Fanelli 18:05 I think it doesn't hurt, right?

Akshat Bubna 18:07 The speculative decoding is you have a smaller model, uh, pre-- called a draft model, predict tokens ahead of the bigger model, and then you have the bigger model, uh, verify all of this, all the tokens are predicted. And the reason it's faster is if you're predicting, uh, one token at once, you're kind of, uh, bound by memory bandwidth. But if you can batch the verification of, uh, the draft model, uh, then you're much more efficient using compute, and it's faster, and as long as your draft model is producing a lot of tokens that can get accepted, which is called the accept length, you can get a speed up that's, uh, multiple times of, um, you know, the original model speed. Uh, and w-that's what we highlight here. It's, um... Like, people talk a lot about we made these kernels faster and whatnot, but improving kernel will only give you, like, a few percentage points of improvement, and, uh, increasing accept length, uh, literally is a multiplicative decrease in-

Alessio Fanelli 19:01 Like two to four X.

Akshat Bubna 19:02 Yeah, exactly.

Alessio Fanelli 19:02 Without much head-on performance.

Akshat Bubna 19:04 Yeah.

Alessio Fanelli 19:05 I think it may be... I-I mean, you, you are running a second model, right? So it may be something more expensive in, in the compute, but- I meant quality performance-

Akshat Bubna 19:12 No, not much

Alessio Fanelli 19:12 ... but yeah, I mean, I think-

Akshat Bubna 19:14 So there's no drop in quality performance-

Alessio Fanelli 19:16 Yeah

Akshat Bubna 19:16 ... because you're always-- you're never accepting a token that the big model-

Alessio Fanelli 19:19 It's strictly better-

Akshat Bubna 19:20 Yeah

Alessio Fanelli 19:20 ... or it's same.

Akshat Bubna 19:20 Exactly.

Alessio Fanelli 19:21 Right. Yeah.

Akshat Bubna 19:22 And so we've been working a bunch on DeFlash, which is a, a block-based speculator. Uh, so it's, um, uh, instead of predicting, uh, one token at a time, it's predicting a block. And we've been open sourcing our work with it. The, the next thing for us here is for, for helping people train speculators and custom models. Uh, it's, it's something that traditionally is very FDE driven, uh, support deployed, engineer driven, like you work with customers and help them do that. And our vision for... This is why we launched Auto Endpoints, is we want to make frontier-level performance available to everyone. And so, uh, we mentioned this in the announcement, we kind of teased it. The next thing we're, we're launching is basically, as you run an Auto Endpoint, uh, we, we shadow traffic and-

Alessio Fanelli 20:08 Do, do you want to explain what Auto Endpoints are?

Akshat Bubna 20:11 Yeah.

Alessio Fanelli 20:11 A level, yeah [chuckles] .

Akshat Bubna 20:12 Yeah. So, um, this is, I guess, going back to your modal is, is you touch the code, but, uh, sometimes people actually don't wanna touch the code and they wanna get started with an endpoint that works and has all the great performance and, uh, scalability that Modal has. So we've made that easier with, uh, basically a way to create an endpoint from our UI, from the CLI, um, that has all of our optimizations that we talked about, uh, like the DeFlash stuff already baked in, and there's full transparency. So we give you the code, uh, you can go run it yourself, and if you want, you can eject out into the full modal experience, uh, which we see as people get sophisticated, they, they do wanna tweak the models, uh, they wanna, uh, fine-tune stuff. You, you can still do all of that. It's, it's not a black box. And yeah, the next thing, as we teased later in the post, is how do we give you value even beyond this in terms of having your draft models evolve as your data distribution evolves, again, without having to talk to a, a person and, um,

Akshat Bubna 21:13 yeah.

Alessio Fanelli 21:13 I guess just to kind of understand it directly, I mean, you know, obviously you, you have the GPUs, you have an endpoint that's compatible, you serve Open Model. If someone was to do this themselves, what's the delta that you guys provide? So you do a lot of open source great work on effective inference. Um, how does it compare to, say, I take the same model, GLM five point two FP8, take off the shelf inference engine, vLLM, SGLang, um, you know, get compute of similar capacity, similar cost. What's the kind of delta that plugging into something this-- like this offers outside of the benefit of, you know, scaling?

Akshat Bubna 21:48 It's kind of interesting because we-we've taken the approach of open sourcing our contributions and upstreaming them. Uh, we work closely with the SGLang team. We, we actually want the improvements that our team, uh, comes up with to be, uh, there in open source for others to use, even outside of Modal. The benefit to us is we have a team that has significant expertise in terms of if you do have something that is not there, our team can help you get that performance, uh, first. Um, the other thing is, with these endpoints, we are way more elastic, as you said, uh, than, uh, anyone else, and you have true scaling to zero. Uh, you have true, uh, burst CNS. And in practice, that matters a lot more to people than just finding, uh, the GPU and, uh, running Modal code on something.

Alessio Fanelli 22:35 Yeah. And I will say it's actually not that straightforward to just... Uh, like what I said is easier said than done, right?

Akshat Bubna 22:40 Yeah.

Alessio Fanelli 22:41 Um, it's, I think, still for the average person, still hard to just gut check using different... There's, there's quite a bit of combinations you can make there. Um, the trade-offs aren't really known at face value.

Akshat Bubna 22:54 Yeah. I mean, it's, it's not just that. I think it's, it's that running production-grade inference is a hard infer problem. Uh-

Alessio Fanelli 23:03 Yeah

Akshat Bubna 23:03 ... even if you subtract out the autoscaling-

Alessio Fanelli 23:05 Yeah

Akshat Bubna 23:05 ... is p-- uh, controlling things like tail latency and, um, making sure every, uh, request is delivered at least once and whatnot.

Alessio Fanelli 23:14 There's a lot of innovation that you can do here. I think, uh, it's very interesting that you're starting to encroach on, like, as you become a full cloud, you're starting to encroach on other people's turf.

Akshat Bubna 23:23 Mm-hmm.

Alessio Fanelli 23:23 [laughs] What will you not do?

Akshat Bubna 23:27 Well, we wanna follow our users and, um, make sure they, uh, they get like a, a platform that has everything that works well together. Uh, so right now we're kind of focused on the model lifecycle and the agent, uh, lifecycle. Uh, so both, like, going from data prep to training to inference, and then also if I wanna deploy a background agent, let's say, you know, sandbox, super persistent storage, a whole bunch of other stuff in there.

Alessio Fanelli 23:53 We talked to Cole, uh, who did, uh, Open Inspect. Yeah.

Akshat Bubna 23:56 Yeah.

Alessio Fanelli 23:57 And obviously Real Inspect also is on Modal [laughs] .

Akshat Bubna 23:58 Yeah. So Ramp Inspect was, was a, a [laughs] a great example of a background agent that was really successful because they, uh, were able to use some of the primitives like snapshotting and fast scaling to just have something that feels really reactive and, uh, and works well.

swyx 24:16 Yeah. That's the new CTO of, uh, Ramp right there.

Akshat Bubna 24:19 Yeah, Rahul. [laughs]

swyx 24:20 [laughs] It was really, really fun. Um, yeah, I mean, okay, you know, I th- I think, uh, all very bullish. Like, y- you know, one of my reflections was also I did not originally... So obviously, when I met you guys-

Akshat Bubna 24:33 Mm-hmm

swyx 24:33 ... you weren't that much in the GPU game, and now you're all about, uh, inference. And one of the points that I hinged on for Jensen's keynote at GTC this year was at what we're calling, like, the inference inflection, right? That let's say in AI workloads or machine learning workloads, it used to be, like, let's call it eight to one GPU to CPU, and now it's more like one to one, which is, like, a interesting... Like, the-- Because of how much agents basically are blocked or call out to s- to CPU heavy stuff-

Akshat Bubna 25:08 Mm-hmm

swyx 25:08 ... the actual, like, limiting factor, like, swings back and forth from GPU to CPU a lot more than it used to be all GPU and then occasional CPU.

Akshat Bubna 25:16 Yeah.

swyx 25:16 GPU, CPU. And now it's, like, just constantly, and you just have to co-locate everything.

Akshat Bubna 25:22 Yeah. And, and that's one of the things that, um, actually, like, again, we see as, uh, something appealing about Modal, which is we've built this capacity pool that spans, uh, 17 cloud providers, so we're, we're very good at running on various kinds of cloud capacity across the world.

swyx 25:39 You don't have your own data centers.

Akshat Bubna 25:40 We don't have our own data centers. We, we just run across a lot of neo clouds and-

swyx 25:44 Yeah

Akshat Bubna 25:44 ... metal providers.

swyx 25:45 Yeah. Question mark.

Akshat Bubna 25:45 Yeah, yeah. You're, you're running the math, and you're like, "What, what's the cutover point where you're like..." Yeah, it's a good question. I mean, part of it is we, we see our differentiator in, in the software layer, and, um, being capital light and focusing on the software helps us move really fast. Uh, so far it's worked out well because there are so many other people building data centers that we're able to work effectively with them and again, focus on what makes us, uh, special.

swyx 26:09 Yeah.

Akshat Bubna 26:10 But-

swyx 26:10 17 gets you into, like, the local providers sometimes. Like-

Akshat Bubna 26:14 The [laughs]

swyx 26:15 ... What's, what's the most interesting one?

Akshat Bubna 26:16 There are actually a lot more neo clouds than you expect, and they all have various degrees of, um, various levels of reliability. And, uh, s- that's why it's something we've in-invested a lot of time in, is actually, uh, building our own reliability layer on top. Uh, so if the GPU falls off the bus or something happens, we w- uh, user workloads are not affected, and that, that actually lets us use a lot more capacity than, um-

swyx 26:44 Yeah

Akshat Bubna 26:44 ... you know, you as a user would be able to.

swyx 26:46 It's a useful thing to have because, like, now everyone knows, like, what layer you are and, like, you, you sort of optimize for being the supercloud-

Akshat Bubna 26:54 Mm-hmm

swyx 26:54 ... of all clouds.

Akshat Bubna 26:56 Yeah. That's, that's, uh, that's the idea. Uh, a-and so, uh, I guess when you mentioned co-location, that's, that's another interesting thing where, um, one thing we've seen is people come to us when they want, uh, very specifically located, uh, CPUs or GPUs, uh, like they want.

swyx 27:11 Oh, to pin it in, like-

Akshat Bubna 27:12 Yeah

swyx 27:13 ... EU?

Akshat Bubna 27:14 Ex-exactly, or EU, US.

swyx 27:15 For, like, data-

Akshat Bubna 27:16 Australia

swyx 27:16 ... locality thing or performance or what?

Akshat Bubna 27:18 It's either data locality or latency. Yeah.

swyx 27:21 Yeah.

Akshat Bubna 27:21 Like, you want your... They're running sandboxes in Modal. They want them to be right next to their GPU.

swyx 27:25 Yeah, it's easy then to-

Akshat Bubna 27:25 Yeah

swyx 27:26 ... to, you know. That is important in, in, in all those things. A-and so, like, you've kind of accidentally, I don't know if, if it's accident, but, like, you've built the perfect primitive for agents to, to express themselves. And then, like, you know, like, it's almost very funny how every extra development just involves more file system, just involves more CPU-

Akshat Bubna 27:44 Yeah

swyx 27:45 ... uh, just like the things that you already have. I don't know much about, uh, if there's any, like, networking usages that are interesting, but you've also done some good work on networking.

Akshat Bubna 27:54 Yeah, I mean, that's exactly right. Like, we-- we're sort of just taking compute storage and networking and building stuff on that layer, uh, for, again, bit of stuff people need. Um-

swyx 28:03 Yeah

Akshat Bubna 28:04 ... we, we see a few interesting networking things coming up. Um, one is people actually want networked sandboxes. Uh, so we have a-

swyx 28:11 For, like, a Docker cluster type thing. Sorry.

Akshat Bubna 28:14 Yeah.

swyx 28:14 Uh, Docker Swarm. Uh, fuck. What is, what is it called?

Akshat Bubna 28:16 Compose.

swyx 28:17 Compose type thing.

Akshat Bubna 28:19 Yeah. So actually if you want Docker Compose, uh, our sandboxes now support, uh, this thing called sidecars. So you can... A sandbox is actually a pod of containers, and you can run multiple containers in, uh, a sandbox. Uh, also useful because, uh, going back to networking, people want a lot of control over, um, outbound networking from a sandbox.

swyx 28:37 Yeah.

Akshat Bubna 28:37 Like, they might wanna run a man-in-the-middle proxy for, like, maybe logging stuff for RL or, um, controlling how egress can happen to a domain, injecting credentials. Uh, and yeah, so we, we've kind of had to build a lot of that stuff ourselves.

swyx 28:53 Yeah.

Akshat Bubna 28:53 Uh, but then also sometimes people actually want, um, sandboxes spanning multiple nodes to talk to each other, uh, which is an emerging thing we're seeing. We have support for that for, for a different reason, and yeah, we'll see if that becomes stable.

swyx 29:06 Like, just an open socket. It's a... This is directly, like mTLS.

Akshat Bubna 29:11 We do support that, which is you can, uh, expose a tunnel inside a sandbox.

swyx 29:15 Yeah.

Akshat Bubna 29:15 And then you can either expose it to public internet or it can be, um, you know, you can add, like, a HTTP, uh, auth layer above it. But we have this thing called I6PM, which we haven't talked about, uh, which is this, like, overlay network using IPv6 addresses. Uh, so if Modal containers, uh, within the same workspace, uh, when this is enabled, can actually address each other using this private IPv6 address, uh, and no one else can.

swyx 29:43 Mm-hmm.

Akshat Bubna 29:43 Uh, so it's like sort of private networking, uh, for containers. We actually built it because we needed it as a primitive for our disputed training product. Uh, so we have this other feature, which is you can add a decorator to a function, and you get a cluster of GPUs, uh, and they have RDMA networking. Uh, so you can run a disputed training job, uh, that's truly serverless. Uh, and we did the overlay network for that. But then we've seen that people are using it for other reasons, and I'm kinda intrigued to see-

swyx 30:11 [laughs]

Akshat Bubna 30:12 ... yeah, what would people do with it.

swyx 30:14 Build primitives and let people figure it out, right?

Akshat Bubna 30:15 Yeah, exactly.

swyx 30:16 You put, you put-

Akshat Bubna 30:16 They're, they're-

swyx 30:17 ... out a pretty interesting-

Akshat Bubna 30:17 They're like, they read the docs we picked. Let me use that for something-

swyx 30:20 Yeah

Akshat Bubna 30:20 ... that we never intended to work. [laughs] This was literally not even in our docs page. People

swyx 30:25 Somehow found it and they're using it?

Akshat Bubna 30:27 Huh.

swyx 30:27 Uh, s-

Akshat Bubna 30:28 I mean, the way you portrayed it with, like, RDMA versus TCP, like, very well laid out, but just the transfer speed change at scale for RL, like-

swyx 30:39 Mm-hmm

Akshat Bubna 30:39 ... yeah, you have it-- you have it built in. I'm sure someone found it, has found it to be a lot more efficient before you actually made a thing out of it, right?

swyx 30:46 Yeah. And, and [chuckles] not to split hairs, uh, I guess the, the overlay network actually is the TCP overlay network.

Akshat Bubna 30:53 Mm.

swyx 30:53 Uh, the reason we have that is you need that to do the key exchange for RDMA before you set up the RDMA network on top of that. Uh, but then people found the, the TCP part.

Akshat Bubna 31:02 Can I tell you, this is, like-

swyx 31:03 Yes

Akshat Bubna 31:03 ... a big aha moment for me because-

swyx 31:05 Yeah

Akshat Bubna 31:05 ... uh, so I, you know, I review 2,200 submissions for, for the Worlds Fair.

swyx 31:10 Yeah.

Akshat Bubna 31:11 And then I got this from John Osterhout-

swyx 31:13 Uh-huh

Akshat Bubna 31:13 ... who I don't know if-- do you know John Osterhout by name?

swyx 31:15 The name sounds familiar.

Akshat Bubna 31:16 He, he published a... He's a well-known professor, published a lot of interesting software design books, and this is the talk he chose to submit, is on t- [chuckles] is on RDMA at inference. And I'm like, you wouldn't think that this guy, who is, like, kind of operating systems guy, would care about RDMA. [laughs]

swyx 31:34 I, I, I mean, it, it makes sense [chuckles] to me because I, I-

Akshat Bubna 31:38 This is the cloud, right? Uh, yeah

swyx 31:39 ... like, like the way you move around your KB cache and how efficiently you can do it, how efficiently you move, um, your weights from your training GPUs to your inference GPUs in RL is there's a lot of degrees of freedom, and it is basically a systems problem of-

Akshat Bubna 31:55 Yeah

swyx 31:55 ... moving memory around-

Akshat Bubna 31:56 Yeah, yeah, yeah

swyx 31:57 ... and scheduling.

Akshat Bubna 31:58 This shows you how primitive my understanding of networking stuff is.

swyx 32:00 [chuckles]

Akshat Bubna 32:01 Is this, like, the domain of WireGuard as well?

swyx 32:04 Mm, not quite. So-

Akshat Bubna 32:05 It's adjacent?

swyx 32:06 Um, so-

Akshat Bubna 32:07 Explain everything. [laughs]

swyx 32:08 Sure, sure. Uh-

Akshat Bubna 32:10 How do we move memory around GPUs? [chuckles]

swyx 32:12 [chuckles] Well, so, so sorry. Yeah, that is memory. Sorry, I, I was talking more in, maybe I was talking, like, five minutes back about the, the private IPv6, uh, addressing that you've set up.

Akshat Bubna 32:23 Mm.

swyx 32:23 This is, like, it's basically a VPN.

Akshat Bubna 32:25 Yeah, it is sort of like a VPN, and yeah, WireGuard is, is, uh... Yeah, you, you're right. It, it is, it is, um-

swyx 32:31 Right, right. Yeah, you already moved on to new topics

Akshat Bubna 32:32 ... uh, similar-

swyx 32:32 Okay

Akshat Bubna 32:33 ... in the same space. WireGuard is, uh, encrypted, and, and this is, uh-

swyx 32:37 And you don't need encryption.

Akshat Bubna 32:38 Yeah.

swyx 32:38 Yeah.

Akshat Bubna 32:38 It's not encrypted. Uh, that's the main difference. This is TCP, and we have eBPF programs that will reject or allow the, um, uh, the TCP connection based on whether you're allowed to do it.

swyx 32:50 Used to involve a full sidecar, but now you have eBPF in the Linux kernel.

Akshat Bubna 32:53 Yeah.

swyx 32:54 Yeah. I don't know if this is a natural follow-on to the topic of, like, my skepticism on distributed training-

Akshat Bubna 33:00 Mm-hmm

swyx 33:00 ... is that, well, like, people spend a lot of money on, like, cables [chuckles] to hook up GPUs, and even then it's not, like, fast enough, and that's the bottleneck. Is your networking fast enough?

Akshat Bubna 33:14 Yeah, so I, I guess you're talking about sort of fully distributed training, like a DiaLoc or something which is, like, cross data center-

swyx 33:20 That would be... Yes, that's the extreme.

Akshat Bubna 33:22 Yeah. You're kind of in the middle, and then other people would have, like, the Mellanox cables up in, in, like, their actual data center. When you run multi-node training on, on Modal, uh, w- uh, RDMA, uh, I, I, I think Mellanox, uh, is, or InfiniBand is like, uh, uh, is all sees RDMA. Uh, but basically it's a way to bypass the TCP networking stack and, um, transfer, uh, stuff much faster, uh, between one node, uh, to the other. And, and we, we have, uh, I think, like, three terabit per second, uh, internode networking-

swyx 33:54 Okay

Akshat Bubna 33:54 ... which is the standard that's needed.

swyx 33:56 Okay. So I misunderstood what-

Akshat Bubna 33:57 Fifty-

swyx 33:57 ... what part of the stack you were-

Akshat Bubna 33:58 Fifty gigs over TCP-

swyx 33:59 Yeah, yeah, yeah

Akshat Bubna 33:59 ... three point two over RDMA.

swyx 34:00 Okay. [upbeat music]

swyx 34:03 Yeah, I, I mean, very impressive work. [laughs] So e- effectively, you're, you're extending sort of, like, the Modal philosophy to the, the tr- training cluster, like, yeah.

Akshat Bubna 34:14 Yeah. And we're, we're not going for, obviously, like, large-scale pre-training runs. Uh, the thing that we've built multi-node training for is, uh, we see a lot of, um, smaller scale post-training. Uh, like, uh, people are post-training, like, medium-sized Quinn models, uh, so they can, uh, get higher quality on inference. Uh, this is a perfect fit, uh, for something like that.

swyx 34:35 Yeah. That is my impression of how a lot of these labs explore branches in post-training-

Akshat Bubna 34:42 Mm-hmm

swyx 34:42 ... and then eventually merge whatever they find in.

Akshat Bubna 34:45 Yeah. The, the other use case we've seen for multi-node training is even if you have a big cluster, your researchers are still doing small runs and-

swyx 34:52 Yes

Akshat Bubna 34:53 ... having elasticity there-

swyx 34:54 Gradual exploration

Akshat Bubna 34:54 ... matters a lot more.

swyx 34:55 Yeah. Uh, the, like, this is actually, like, the current limiting factor for auto research, which is, like, you, you basically need to give your model some GPUs.

Akshat Bubna 35:06 We have a blog post-

swyx 35:07 You need it completely run-

Akshat Bubna 35:07 ... on auto research, and Modal is, uh-

swyx 35:09 Yeah

Akshat Bubna 35:10 ... yeah, like, uh, turns out to be a pretty good substrate for that.

swyx 35:13 So my impression is au- auto research means many things, like-

Akshat Bubna 35:15 Yeah

swyx 35:15 ... anything that Andrej Karpathy. Right now, still science fair, right? Like, not, not, not actually like-

Akshat Bubna 35:20 Mm

swyx 35:20 ... I don't know how many people are actually doing this.

Akshat Bubna 35:22 We're having a golf.

swyx 35:23 Yeah.

Akshat Bubna 35:24 I thought the same thing.

swyx 35:25 Yeah, you would know.

Akshat Bubna 35:26 We, like, our internal both training and inference teams actually use this sort of the general shape of this quite a bit. Uh, like, we have this one internal repo called Auto Inference, which essentially we've automated our own FDE efforts using, uh, this harness, which is, um, the agent will just spin up a sweep of different things. It'll even run, like, um, NVIDIA Insight Profiler, and it'll, like, tweak configs, and it'll arrive the right thing. Uh, it'll change your GPUs both from H200 to B200. It actually works really well.

swyx 36:01 Nice.

Akshat Bubna 36:01 So yeah.

swyx 36:02 By the way, I enjoy that your FDE is so technical that you have to do these things. [chuckles]

Akshat Bubna 36:06 [chuckles]

swyx 36:06 It's very different from FDE from other people. [chuckles]

Akshat Bubna 36:08 Yeah, yeah. For... Our FDE team is, uh, essentially they're, like, applied inference researchers or applied training researchers.

swyx 36:16 Someone told me, like, they have to be able to build, but they also have to be able to sell. Uh, do they have to sell, or are they like, they're good, they're just like post-sale type of thing?

Akshat Bubna 36:24 It does... Uh, being able to talk to a customer and Engage effectively with their matters a lot.

swyx 36:28 Yeah. They want the same thing.

Akshat Bubna 36:29 Yeah.

swyx 36:29 You know?

Akshat Bubna 36:30 But it's, it's not really a sort of sales, uh, thing. We, we pair them with-- We have solution architects as well that are more on the pre-sales side.

swyx 36:38 Okay. Let, let's spend a bit more time on auto research. This is a big focus for me-

Akshat Bubna 36:41 Mm-hmm

swyx 36:41 ... for, for, for this year. Where does this go? You know, like, um, have, have people explored enough? Like, you know, there's all these beautiful charts of like improve, improve then they sort of level off a bit and then you find the next thing. Is this basically sort of one abstraction up from normal training? Is that how we think about it? Or do you think about it differently? Like model-level training versus basically a high-- like AI-driven hyperparameter search.

Akshat Bubna 37:04 Yeah, like, um-

swyx 37:05 Uh, someone-- some people call it like neural architecture search or whatever, right? Like...

Akshat Bubna 37:08 Yeah, I mean, I-- So the, the stuff I've seen people do with it is nowhere on the architecture level. It's pretty much tweaking parameters, but it's, it's basically a hyperparameter sweep that's guided by some sort of model intuition, so it's like much more efficient than, um, whatever other, uh, sweeper you have.

swyx 37:27 Uh, yeah, I mean, you know, it's, you know, it's just, it's just a question of where you wanna spend your compute, you know?

Akshat Bubna 37:30 Right.

swyx 37:30 'Cause yeah, you can just throw infinite amounts of money on this and somehow you'll bang out Shakespeare, you know? [laughs]

Akshat Bubna 37:37 Yeah. Infinite monkey.

swyx 37:38 Uh, yeah, I mean, so like the- very good for modal. Uh, and I, I think it's also very important that agents can spin up other agents, can spin up their own-

Akshat Bubna 37:46 Mm-hmm

swyx 37:46 ... infrastructure, like very, like very good for you. Uh, how good is LLMs at generating modal code? Like, you know, like the benefit of existing pre-LLMs is that you are in the data.

Akshat Bubna 37:56 Yeah. They're, they're actually surprisingly good. Uh, I think like pre Cloud Four they were not, and then now they're able to one-shot, uh, stuff out of the box. And we're playing around with releasing like a modal bench for like the harder-

swyx 38:10 Yeah

Akshat Bubna 38:10 ... things, uh, that the LLMs cannot do yet and, and maybe-

swyx 38:14 What, what's an example of that?

Akshat Bubna 38:15 I think the, the things that sometimes agents struggle with, um, without right guidance and a skill is, um, how to use the rest of our observability, like how to... Something is failing, like how do you look at the, the logs and then update the right thing? It's sort of reasoning about that. But they're able to one-shot like-

swyx 38:37 Yeah. You can, you can just add a skill to it?

Akshat Bubna 38:40 Yeah. So, so we have a modal skill now that-- which is kind of actually why we built this modal bench. It's to find things like that, so we can, we, we can address them in our-

swyx 38:49 Tune a skill.

Akshat Bubna 38:50 Yeah.

swyx 38:50 No, no. I mean, it's, it's good. Uh, are you facing any shortages? Uh, you know, like we talk a lot about GPU shortages, but also CPU, also memory.

Akshat Bubna 38:58 Mm-hmm.

swyx 38:58 Yeah.

Akshat Bubna 39:00 Uh, we have had a lot of growth, which means that, uh, there, there's a-- we've had to be much better about-

swyx 39:07 Planning

Akshat Bubna 39:08 ... proactive capacity planning.

swyx 39:09 Yeah.

Akshat Bubna 39:10 Uh, so we, we have, um-

swyx 39:11 Which by the way, like it's like a MBA's like dream job-

Akshat Bubna 39:14 Yes

swyx 39:14 ... is like just planning this stuff. I think last time you and I talked about something maybe about this.

Akshat Bubna 39:17 Yeah. I mean, we have a, a really competent team of people that we call, um, the role is called compute strategy. Uh, so yeah, if anyone [chuckles] listening here, uh, wants to work on that.

swyx 39:27 Compute strategy?

Akshat Bubna 39:28 Yeah, yeah.

swyx 39:28 I feel like, uh-

Akshat Bubna 39:29 I mean, the norm-- the normies call it FP&A or something. [laughs]

swyx 39:32 Well, it's more-- It's, it's not FP&A. It's, it's, um, there's a lot of interesting financial questions of like, uh, like what is the blend between one-year and three-year reservations? W- how do we forecast our own capacity? Uh, how do we... Basically, especially since our capacity is very fungible across different GPU types and different regions, uh, like you basically have to model a lot of it. Uh, and you also have to have an opinion on how the supply chain is gonna evolve, and then you have to like take bets, uh, based on that.

Akshat Bubna 40:04 Yeah. Tokenomics.

swyx 40:05 Yeah. This is like probably a not a real point, but, uh, you know, I was trying to think about like what other industries-- I always try to think about like, you know, we cannot be first to like these kinds of problems.

Akshat Bubna 40:13 Mm-hmm. Yeah.

swyx 40:14 And what other industries have had this? And I was like, airlines with, with fuel, and like they have to hedge their fuel. And like I think for a long time, Southwest, because they made like a hero fuel bet, they like were like super low cost because-

Akshat Bubna 40:26 Oh

swyx 40:26 ... compared to everyone else. [laughs]

Akshat Bubna 40:28 Yeah. I, I hadn't thought about that. Um, but-

swyx 40:30 We're at a fun time too, you know? It's-

Akshat Bubna 40:32 Yeah. It's... A, a lot of the, uh, the compute business in general, uh, for us is also about being very good about capacity management. That is how you have great unit, uh, economics, uh, but also over time is how you can unlock more value for customers. Like, um, one of the things we're building now is like a way for customers to get, um, if they don't care about latency, uh, like get much cheaper pricing, and they'll get results back in like next twenty-four hours or something, uh, like a batch tier, essentially.

swyx 41:01 Yeah.

Akshat Bubna 41:01 Uh, and those are levers we have because we control the whole stack and scheduling and whatnot to give people-

swyx 41:07 Yeah

Akshat Bubna 41:07 ... a sufficient-

swyx 41:07 Yeah, I feel like-

Akshat Bubna 41:08 Yeah

swyx 41:08 ... they are not as popular. Like those, like the, the Frontier Labs have all those APIs. They're not as popular as they should be.

Akshat Bubna 41:15 The demand that we see for something like that is actually not for LLMs. Uh, although sometimes people wanna run evals and do-

swyx 41:22 Okay

Akshat Bubna 41:22 ... synthetic data prep and, and there it makes sense.

swyx 41:24 Okay.

Akshat Bubna 41:25 Um, but it's, uh, from a lot of non-LLM companies, uh, like people who are doing computational bio, like they, they have to run really big batch jobs, and they don't care about when they get it back.

swyx 41:36 Yeah. And like they have a reasonable... It's, it's also like a cousin to the stopping problem of like, will this finish in time? [laughs]

Akshat Bubna 41:45 [laughs] Yeah, well, you, you can, you can bound it.

swyx 41:47 Yeah.

Akshat Bubna 41:47 Like you can give people-

swyx 41:48 Yeah, yeah, yeah

Akshat Bubna 41:48 ... SLAs on it.

swyx 41:49 Yeah, yeah. I think what's, what's interesting is like the, the next phase of Modal.

Akshat Bubna 41:52 Mm-hmm.

swyx 41:53 Like what, you know, do people expect from you, uh, now that you're sort of established and you're like a well-known compute player among all these leading companies? You had an inference launch week, and we talked a little bit about the launches. Uh, like what else? Like what, what else should people know?

Akshat Bubna 42:09 We are building primitives that make our, our users' lives much easier. So, um, I think for example, with LLM inference, thousands more companies are gonna post-train their own models and, uh, deploy open source models for inference. Uh, so we're thinking a lot about what is the best Product shape for that. And, uh, that involves everything from our training gym to, uh, then, uh, endpoints that get frontier-level performance, um, you know, again, without having to talk to anyone. It looks somewhat different on other verticals. Like, uh, we're also seeing a lot of real-time, uh, audio/video stuff in there, um, uh, which is why, like, uh, we're working on things like regional routing, uh, with, with fallbacks. So you can, you can get sort of GPUs that are as close to users as possible, uh, so you, you get, like, low latency for video streaming and, and whatnot. And then on the agent side, it's, um... We're s- still sort of working very closely with our customers because stuff is changing so fast in terms of what they need. And, uh, I think beyond sandboxes and persistent file systems, uh, there's a lot of other things people need from this agent stack as they build production agents. So yeah, we're thinking about those other things that fit in there.

swyx 43:27 I wanted to ask what, what the other things are. [laughs]

Akshat Bubna 43:29 [laughs]

swyx 43:29 Yeah, I probably should share right now.

Akshat Bubna 43:31 I think, I think... Okay, so, uh, I do think a lot about the principal components of cloud, and you do talk about compute storage networking.

swyx 43:39 Mm-hmm. Yeah.

Akshat Bubna 43:39 'Cause so far for me, it's fine. Uh, so far for, they-- I mean, the first couple generations of cloud, it's fine. What's different, qualitatively different about agents that you, you need some new permission level? Like a lot of people, you know, obviously... Okay, and I'll just kinda spew tokens at you until, until it like hopefully sparks something.

swyx 43:57 Yeah.

Akshat Bubna 43:58 Like the, the new level now is whatever cloud code does, which is dangerously scale permissions or like allow this by command or like whatever, right? And, and sometimes they're like, "Well, okay, we, we have like this adaptive thinking mode where like just, just trust me, bro. I, I will make the calls [chuckles] for you." Is that it? You know, like l- basically like sort of LLM-mediated permissions.

swyx 44:17 Like now you're looping it with a goal and flattening RL.

Akshat Bubna 44:20 Yeah. Yeah, I mean, I'm, I'm skeptical of LLM-mediated permission for-

swyx 44:24 [laughs]

Akshat Bubna 44:25 ... stuff that is at the sandbox level because you do want hard boundaries.

swyx 44:30 Yeah.

Akshat Bubna 44:30 Otherwise, obviously someone can exfiltrate stuff.

swyx 44:34 But like, like maybe, maybe that's old school thinking. Maybe we're the dinosaurs.

Akshat Bubna 44:37 [laughs]

swyx 44:37 Maybe the AI OS or the LLM OS is really the kernel is a goddamn LLM.

Akshat Bubna 44:43 [laughs] Uh-

swyx 44:44 Like it makes you feel uncomfortable.

Akshat Bubna 44:46 Yeah, I'm told.

swyx 44:46 But that's what trusting the LLM is. Like imagine a spherical cow perfect LLM.

Akshat Bubna 44:51 Right.

swyx 44:52 That it. [laughs]

Akshat Bubna 44:53 Maybe, maybe.

swyx 44:55 I wanna test the boundaries, right?

Akshat Bubna 44:57 Yeah.

swyx 44:57 Like and obviously I, I don't believe that, but I wanna see where I'm wrong 'cause that's, that's the non-consensus.

Akshat Bubna 45:03 Yeah. I mean, I, I, I think you always need hard guardrails when you want, um... And you compare those to softer guardrails, right?

swyx 45:12 Yeah.

Akshat Bubna 45:12 And those can be a lot mediated.

swyx 45:14 There... I'll get you c- uh, end with a couple of your commentary on like the ecosystem outside of Modal. Managed agents, everyone has one: Gemini, OpenAI, Claude. Uh, very useful for you, but also like it is their way of starting to edge into your space.

Akshat Bubna 45:31 Yeah.

swyx 45:31 Uh, what's going on?

Akshat Bubna 45:33 Yeah, I mean, we're, um, very excited to partner with Anthropic and some of the other foundation labs, uh, will not name who we're also working with. Um, the way we see it is the managed agents thing is a great place to start if you're starting out building an agent. And, um, but, but then when you get to, um, building something more production grade, like you're a company that's like Ramp that's building their own, uh... Ramp also runs their accounting agent on us, so their external-facing agent. You need a lot more control over, uh, your compute primitive on things like, um, what sort of-- how do you persist different files that the agent has access to, and how do you snapshot and restore? How do you control the networking? Um, maybe you want GPUs. When you get to that point, you kinda want, um, a specialized sandbox provider, uh, that gives you those things, and that's the role that we are trying to play. Um-

swyx 46:29 Yeah.

Akshat Bubna 46:30 We don't really have an opinion on the harness, whether it runs in cloud-managed agent and you hook it up to Modal Sandbox, or you run the harness in Modal Sandbox. We'll see where people converge with that.

swyx 46:40 Yeah. Do you any opinions on like the meta harnesses, um, or just another layer on top of these things?

Akshat Bubna 46:46 You mean like the OpenPi and-

swyx 46:47 OpenPi is one. Um, I think Vercel had one, which I can't remember the name of right now. Fred Schott had one. Uh, and then, uh, to me, most recently was Data, Databricks that had Omnigen. All these are sort of meta harnesses. Like it's kind of pseudo agent cloud-type things.

Akshat Bubna 47:04 I personally have not played around with them.

swyx 47:07 Yeah.

Akshat Bubna 47:07 Have not built agents with them.

swyx 47:08 I mean, everything's bullish Modal as long as it consumes more infra.

Akshat Bubna 47:11 That's why we're focusing on the infra layer. It's, um, somewhere where our, um, relative competence is and, um, also

Akshat Bubna 47:20 it's a hard problem to solve.

swyx 47:20 Yeah. I mean, I, I will say, like just generally reflecting on... I don't know if you-- if there, if there's other topics on Modal, but like just generally reflecting as an infra person, not as intense as you, but in that field, this has like been the most exciting time in infra. Like it was boring actually for, for, for a while, and you couldn't really get people excited about data infrastructure. Like Eric would get on Data Console, everyone just watched the video, and like say, "Look at how many sandboxes I can spin up," and no one gave a crap. [laughs]

Akshat Bubna 47:53 Yeah.

swyx 47:54 And like now everyone gives a crap. [laughs]

Akshat Bubna 47:56 That's true. It, it is a very exciting time, and I think a lot of that's driven by just, uh, the amount of scale all of this stuff needs.

swyx 48:05 I think the, like a lot of your initiatives or a lot of your like product directions make sense in retrospect, which is like the best kind, but I wouldn't necessarily have thought about it myself, which, which... [laughs]

Akshat Bubna 48:14 We need, we need the predictions, you know?

swyx 48:16 [laughs]

Akshat Bubna 48:17 I mean, I think there's a lot that you just don't even see, right? Like you have the batch, you have the voice, you have the multimodal, um, but what, what else, you know?

swyx 48:24 What else is coming up for us? Or-

Akshat Bubna 48:25 Yeah. Where, where do you see things going?

swyx 48:27 Yeah. I, I mean, in general

Akshat Bubna 48:30 It's, it's clear that there's, uh, obviously there's a huge shift happening. I, I think one thing that's not as obvious to people because LLM inference gets talked about so much and is also we work a lot of companies that are, uh, doing things like drug, drug discovery and computational bio, like the Tide Discoveries of the world. Big things are probably gonna happen there. Uh, we work a lot of robotics companies that are actually putting robots in, in like active deployments and getting good results out of them.

swyx 48:59 Is there air gap Modal? Is there, is there a version that is like on-prem air gapped, whatever?

Akshat Bubna 49:04 No. We, we-

swyx 49:05 You guys are cloud only.

Akshat Bubna 49:06 Yeah.

swyx 49:06 Yeah. Okay. But yeah, I mean, uh, so what you're saying is like, because you're focused on primitives and they're good primitives, you find use cases in all these kinds of things.

Akshat Bubna 49:16 Yeah.

swyx 49:16 Actually probably diversifies you a little bit away from LMS all the time.

Akshat Bubna 49:19 Yeah, absolutely. We, we're-- Our, our goal isn't to only serve the LM inference market. Uh-

swyx 49:24 You have a lot just on the website, the audio, the Porsche-

Akshat Bubna 49:26 Yeah. We said, we said both on the pod

swyx 49:28 ... computational bio images. Yeah. I mean there's a lot here. There's QTI TTS customizing-- Oh, Chatterbox. Um, you know, there was-

Akshat Bubna 49:36 So-

swyx 49:36 ... a customizing Whisperer.

Akshat Bubna 49:38 Okay. Yeah. This screen reminds me of a fallen competitor, uh, which

Akshat Bubna 49:44 replicate.

swyx 49:44 Mm-hmm. [laughs]

Akshat Bubna 49:45 Um, what's your postmortem on what happened? This is one thing we've kind of stayed away from, is providing an API for models, because I think providing model APIs is some of it ends up serving like a really hobbyist market, which is much less sticky.

swyx 50:04 Yeah.

Akshat Bubna 50:04 And we've always wanted to build for companies that are building sort of products and need sort of more flexibility that's not just an API. And-

swyx 50:11 Which you, you can build an API for a model and, and this is clearly what it is, but you can-- But what you're saying, you can wrap it into a more fully functioning backend that you, that you run.

Akshat Bubna 50:21 Yeah. So actually all, all of our examples, it's not that spin up this model, here's an API token, use it. They're actually all code.

swyx 50:27 Okay.

Akshat Bubna 50:27 And so the point is that this is an example.

swyx 50:31 Starter code.

Akshat Bubna 50:31 Yeah. But you can, you can tweak it however you want.

swyx 50:34 Yeah.

Akshat Bubna 50:35 And if you are like a company building a product like a computational bio whatnot. Yeah.

swyx 50:40 I guess I'm trying to tease out for listeners-

Akshat Bubna 50:42 Yeah

swyx 50:42 ... when does it stop becoming, oh, you are just an API call and you're just a wrapper on an API to becoming what you call a product, right?

Akshat Bubna 50:50 Mm-hmm.

swyx 50:50 Like what is that layer? Like what, like, [laughs] you know, obviously more lines of code, but like beyond that, what, what is the substance that people add that qualifies it to be something more?

Akshat Bubna 51:00 I think there's a little bit of like a selection effect of like a lot of the companies who do want to get deeper into that level are probably building something that's more differentiated. And um, I think, um, an example is like we, with LM and friends originally, we, uh, worked with companies that were building their own post-training frameworks or they were, uh-- Ramp actually early in the day was training their own tokenizer and like swapping out the tokenizer in Llama and whatnot. I'm not saying that's, that was-

swyx 51:31 Mm. [laughs]

Akshat Bubna 51:31 ... successful, um, in, in that case. But a better example is like, um, like, uh, let's say Suno. Um, because Suno, um, does not use Modal for training.

swyx 51:40 Which I had Mikey on the pod. Yeah.

Akshat Bubna 51:41 But they, they use Modal for all their inference, and that's because they have like a custom-- They have completely custom Modal architecture and that means that they have to be at the code level and tweak things that are not, um, you know, just an API.

swyx 51:55 It's interesting as well, like we had, um, Ethan, uh, most recently on the XAI Groq team-

Akshat Bubna 52:00 Mm-hmm

swyx 52:00 ... make a prediction that actually like the next tier in video gen is not a better video model. It's a me-better model or agent that orchestrates video models.

Akshat Bubna 52:10 Oh, interesting.

Alessio Fanelli 52:11 Language model backbone that can use tools-

Akshat Bubna 52:13 Right

Alessio Fanelli 52:13 ... and write code.

Akshat Bubna 52:14 Like yes, I can make my six-second video or my ten-second video from Groq, but actually I want my six-minute video.

swyx 52:20 Mm-hmm.

Akshat Bubna 52:20 And I'm not going there through normal video gen.

swyx 52:24 Yeah. That's interesting. I actually ha-- So w-we have GPU sandboxes and recently have seen a few companies doing sort of agents that do video manipulation or-

Akshat Bubna 52:36 Yeah. Give it FFmpeg and just-

swyx 52:37 Run FFmpeg and be like [laughs] yeah. But like-

Akshat Bubna 52:39 And it's not enough.

swyx 52:40 Yeah.

Akshat Bubna 52:40 You need to give it Adobe. Yeah. I hadn't put it together with like, it would actually be a video production thing. Uh, in, in my mind, these things were going more towards editing and, and-

swyx 52:50 Yeah.

Alessio Fanelli 52:50 Luma Shout out.

swyx 52:51 I, I think about this a lot.

Akshat Bubna 52:52 No, obviously. [laughs]

swyx 52:55 Yeah. Sorry.

Alessio Fanelli 52:55 Um, Luma, Luma Agent is a version of this for video production, but you know, it's a one-off.

swyx 53:01 I was gonna get your quick takes, uh, on, uh, on some other stuff that happens-

Akshat Bubna 53:04 Mm-hmm

swyx 53:04 ... in recent news and just see if you have anything interesting. Gitpod, very like somewhat like, you know, different market. They, they're in like sort of like the CICD market, but actually technically very impressive. I don't know if you've like taken a real look at them.

Akshat Bubna 53:17 Yeah. Uh, we've, um, people on our team have talked to the Gitpod team and they've, they're technically very strong.

swyx 53:24 Yeah.

Akshat Bubna 53:24 I, I actually am-- We're, we're very bullish on Modal on the CI market as well because-

swyx 53:30 Okay

Akshat Bubna 53:30 ... as there's more agents, uh, coding agents-

swyx 53:33 Yeah

Akshat Bubna 53:33 ... uh, they're gonna run a lot more CI and the primitives there can be much better.

swyx 53:38 I think there's a lot of wasted CI.

Akshat Bubna 53:39 Yeah.

swyx 53:40 So is it just like let's filter? Like what, what is the highest order bid here in improving CI for agents?

Akshat Bubna 53:46 Well, there, there's a lot of wasted time in CI on like preparing-

swyx 53:51 Tools

Akshat Bubna 53:51 ... artifacts and like, you know, getting you to the basically, um, uh, preparing your dependencies and whatnot.

swyx 53:58 Oh.

Akshat Bubna 53:59 And um, obviously like build systems help with that. But like if you have primitives that are like memory snapshot and restore, can you just run CI more efficiently?

swyx 54:10 Oh, oh, okay. Okay, okay. Interesting. Yeah. I mean, another form of like, you know, on-demand compute.

Akshat Bubna 54:16 Yeah, exactly.

swyx 54:17 Yeah.

Akshat Bubna 54:18 Uh, it, it needs the same again, platform.

swyx 54:20 Yeah. So, so, uh, for those who don't know, Gitpod rebranded to Ona.

Akshat Bubna 54:23 Mm-hmm.

swyx 54:24 It was like there was this whole thing. I, I got... Actually, I, I, like, sort of semi-sounded the alarm at Cognition. I was like, "You should take these guys seriously because they're inference very good."

Akshat Bubna 54:31 Yeah.

swyx 54:32 Uh, and but, you know, and then, then they joined OpenAI, and, uh, presumably we'll, we'll see Codex Cloud from the Ona team.

Akshat Bubna 54:39 Mm-hmm.

swyx 54:40 Like, which, which I think would be very, very strong. The-- To me, like, teams like that, that can set up the networking and, like, the, the secure boundaries for, like, in your, like, agents to have their own cloud each, uh, effectively is what you're doing kind of, and I'm just trying to draw the analogy or, or the differences. If you have studied them, like, what is the philosophical difference, you know?

Akshat Bubna 55:01 My sense is maybe they didn't go after the right market at the right time because we, I guess, also got lucky with, like, agent use cases really taking off and, um, needing, um, like, more of, like, a sandbox-shaped thing than like, um, uh, my understanding is, yeah, Gitpod were like-

swyx 55:20 Really, sa-sandboxes were-

Akshat Bubna 55:21 Dev environment

swyx 55:21 ... like, CI/CD-

Akshat Bubna 55:22 Yeah

swyx 55:22 ... is sandboxes. [chuckles]

Akshat Bubna 55:24 Yeah, yeah, yeah.

swyx 55:24 It's just like build time sandboxes versus runtime sandboxes, and actually it turned out runtime was better. [chuckles]

Akshat Bubna 55:29 Right. And, and the difference there is runtime sandboxes have a different configuration surface of, like, how you configure images, how you, like, attach, like, pers- storage and-

swyx 55:39 Yeah. It's, it's fascinating. Other people, Astral, also OpenAI.

Akshat Bubna 55:44 [chuckles]

swyx 55:44 Also, like, Python tooling ecosystem people. Are you still sort of bullish build- building on top of Python? Also recently Modular also-

Akshat Bubna 55:52 Mm-hmm

swyx 55:52 ... uh, got bought by, by Qualcomm. Just any, any, any of your takes there.

Akshat Bubna 55:57 Y- yeah. I mean, we, we had Python as our first SDK language because that was the language that people did data and ML in. We actually now have Go and TypeScript SDKs as well. Uh, and our runtime is completely language- It is written in Rust, but it's, it's not tied to Python by any means. We haven't seen... A-and I think with, like, inference and training stuff, people are still very Python, and the interesting thing with, like, the agent stuff is people use our TypeScript SDK a lot more because they're not actually doing anything that needs ML. I don't think we'll have to go beyond that super soon-

swyx 56:30 Yeah

Akshat Bubna 56:30 ... 'cause Python and TypeScript are just still dominant.

swyx 56:34 The last two languages in the world.

Akshat Bubna 56:35 Yeah. [chuckles]

swyx 56:36 That's it. [chuckles]

Akshat Bubna 56:37 Well, English and prompting as far as like-

swyx 56:39 English and prompting. I occasionally talk to people who try to build new, new languages. They're like, uh... Even, um, what's his face? Brett Taylor, uh, who's chairman of OpenAI-

Akshat Bubna 56:47 Mm-hmm

swyx 56:47 ... was like, "We, uh, we need, we need a new language for, for LLM." So no one has come across one, and I keep looking. You know, Python and TypeScript are... You have a lot of data plus-

Akshat Bubna 56:58 Mm-hmm

swyx 56:58 ... but then also they are very imperfect as just as languages themselves. But then my close is I think Modal used to be a big bet on developer experience.

Akshat Bubna 57:06 Mm-hmm.

swyx 57:07 And you've pivoted the team to agent experience. Is it, like, the way now, like, do you... Do, do... Can entire companies and unicorns, multi-unicorns be built on just having better agent experience? Do you need something else?

Akshat Bubna 57:19 It's a big part of our identity. Uh, it's not just, uh, you know, like the, uh, the very tactical how does an agent use the CLI, but it's also how easy is it to spin something up. Like, what is your iteration time when you want to spin up a new service and, uh, you want to get something going in prod? Uh, in practice that matters a lot, uh, to people. And, uh, I think it will continue to matter. Like, uh, people are building stuff even faster, and if you give them ways to do it quickly and-

swyx 57:48 [chuckles]

Akshat Bubna 57:49 ... uh, not have overhead, then-

swyx 57:51 The, I think the debate for me has been do you do anything differently that is, like, very fundamentally different for developer experience versus agent experience?

Akshat Bubna 57:58 Mm-hmm.

swyx 57:58 You seem to be on the side of they're, they, they're like this. They're like cosine-

Akshat Bubna 58:02 Yeah. We actually also have a blog post on that. [chuckles]

swyx 58:03 ... cosine, cosine similarity on, like, zero point nine or whatever.

Akshat Bubna 58:07 Yeah. I mean, pretty much it's the, um, th-the main shift for us has been, as I said, like, we built this, uh, benchmark, Modal Bench, uh, to see where agents are lacking and-

swyx 58:16 Yeah

Akshat Bubna 58:16 ... uh, actually literally impr- add surface areas to our product if, if they're reaching for something, like maybe this should just be a CLI.

swyx 58:23 They halluc-

Akshat Bubna 58:24 Uh-

swyx 58:24 Oh, yeah. They hallucinate their own features.

Akshat Bubna 58:26 Yeah. And sometimes it makes sense. Like-

swyx 58:27 [chuckles]

Akshat Bubna 58:27 ... if they're reaching for this thing, it's product feedback. Like, give it to them. And then, yeah, actually moving w-we used to only have, like, logs and metrics in our UI, just moving all those things to the CLI as well, so they're accessible in, in, in that form.

swyx 58:40 Simple as that.

Akshat Bubna 58:41 [chuckles]

swyx 58:42 Cool. Thank you so much. Yeah.

Akshat Bubna 58:43 Yeah. Thank you.

swyx 58:44 This was great.

Akshat Bubna 58:44 This was fun.

swyx 58:44 Yeah. It was a great update and, uh, I can see why you guys have succeeded so much. Uh, it, it is really, uh, focus, but also really good execution.

Akshat Bubna 58:53 Thanks. I mean, we have a long way to go.

swyx 58:55 All right. Thank you.

Akshat Bubna 58:56 Cool. [outro music]