Next 100x in AI: Inference, Networking, & Self-Optimizing Models — Philip Kiely & Ali Taha, Baseten

1. 0:00 – 3:18

   Introduction and the 200K-Token Prompt

   1. ATAli Taha0:00

      GLM-5.2 is very, very good at writing GPU kernels. It was very funny. Internally, we had a GLM-5.2 endpoint that we were using to-- like, that we plugged in in our cloud code harness, so every engineer on the team uses, like, our GLM-5.2. And it will do a forward pass on the GLM-5.2 instance of the, you know, the node, and then it will get the profile trace, and it will analyze it, and it will find the kernels that are the bottlenecks in that slang. And then it will write the new kernels, and then we'll do another profiling trace. And when it's done, it uploads the image to our thing, and then we can pull that image down and repeat the cycle. Some of the GPU kernels that run GLM-5.2 within our inference engine is written by GLM-5.2.

   2. SWswyx0:39

      Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we wanna keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring The In-Space to you each and every week. If you do it, I promise you we'll never stop working to make this show even better. Now let's get into it. [upbeat music] Okay, we're here in the studio with, uh, Philip, uh, old friend from, from, uh, Inference Engineering, the book, as well as Baseten and, uh, everything that you've done-- you and I have done before, as well as Ali. Welcome.

   3. PKPhilip Kiely1:39

      Pleasure to meet.

   4. SWswyx1:40

      Waterloo Intern.

   5. ATAli Taha1:41

      Waterloo Intern.

   6. PKPhilip Kiely1:41

      Always. [chuckles]

   7. SWswyx1:42

      When did you get Waterloo Intern as a-

   8. ATAli Taha1:44

      As a handle?

   9. SWswyx1:44

      -handle?

   10. ATAli Taha1:45

       Oh, I think the rebranding happened, like, mid-March. When I saw it was open, I was like, "I have to take it. Up for grabs."

   11. PKPhilip Kiely1:50

       The problem is that Ali is really good at his job and is not gonna be an intern much longer.

   12. ATAli Taha1:54

       [chuckles]

   13. PKPhilip Kiely1:54

       So we have to figure out, you know, who's gonna get the handle.

   14. ATAli Taha1:58

       Well, pass the torch over to someone.

   15. SWswyx1:59

       Oh, okay. It c- can be-- Like, you just pass it to another Waterloo grad.

   16. ATAli Taha2:02

       It's another Waterloo inter-

   17. SWswyx2:03

       Yeah.

   18. ATAli Taha2:03

       No, grad.

   19. PKPhilip Kiely2:04

       Yeah, yeah.

   20. ATAli Taha2:04

       Intern.

   21. SWswyx2:04

       Intern. Yeah.

   22. ATAli Taha2:05

       And, and, and-

   23. PKPhilip Kiely2:05

       You gotta get an intern from Waterloo

   24. ATAli Taha2:07

       ... I'm just-

   25. SWswyx2:07

       Right, right, right, right.

   26. ATAli Taha2:07

       Yeah, you gotta get an intern from Waterloo.

   27. SWswyx2:08

       Right, right.

   28. ATAli Taha2:09

       But they have to promise to pass the baton.

   29. PKPhilip Kiely2:09

       Oh, it could-- but it could come from Baseten, so it's like whoever Baseten gets from Waterloo-

   30. ATAli Taha2:13

       Right.
2. 3:18 – 11:26

   Dedicated Deployments, Speculative Decoding, and Tool Calling

   1. PKPhilip Kiely3:18

      prefill workers, and number two, ideally some cached input already there so that we can skip prefill on at least part of these 200,000 tokens. Um, if you're doing 200,000 tokens, it's probably coding or a multi-tone agent or, or something where you would expect to have that cached. Um, if you don't, we're gonna have to send it to a prefill worker. Um, we've, at least on certain models, disaggregated prefill and decode, um, so you're going to have one set of GPUs that's solely going to process the input, create that KV cache, and get you your first token. And then that's going to be passed over to a separate set of GPUs, which is going to run decode. We're going to iteratively make those tokens. Um, we're probably going to have some kind of speculative model in front of that. I'm going to assume that you're doing coding, and because of that, our speculative model, which assumes you're doing coding, is gonna have a high, uh, draft token acceptance rate. If I'm wrong and you're asking me to summarize every Harry Potter book, uh, it's gonna be slower. And then we stream that output to you and, uh, account for it, um, charge you, you know, some number of couple of pennies and say, uh, "Hey, would you like to send another one?"

   2. SWswyx4:29

      Except Baseten doesn't charge by pennies.

   3. PKPhilip Kiely4:32

      Well, yeah, we, we charge-

   4. SWswyx4:33

      [chuckles]

   5. PKPhilip Kiely4:33

      We-- I'm assuming that we're talking about the public model APIs. If you are, you know, s- setting up a, a dedicated deployment, then yeah, it's not pennies. [chuckles]

   6. SWswyx4:43

      Yeah. I mean, uh, one of the key differentiators when I was talking with, uh, Baseten initially was that, uh, actually, people who want very, very high volume just need to rent by the box, 'cause then it's up to you to figure out how to saturate the box.

   7. ATAli Taha4:55

      And more often than not, it's, like, way cheaper if you're pushing, like, millions of tokens per hour, if you just pay per hour instead of pay per token.

   8. PKPhilip Kiely5:02

      Yeah, they, they do. I think that we've increasingly seen a lot of demand for the sort of pay per token APIs, just because everyone wants to try open models, and then once they find a use case that's really sticky, uh, then they, they move over to dedicated.

   9. SWswyx5:15

      Is there a best practice on when it's time to swap over?

   10. PKPhilip Kiely5:18

       Couple reasons. Yeah, reliability, that's a big one, right?

   11. ATAli Taha5:21

       Like, if they have a very specific use case, they want you to train something specifically for them. Like, they want their own SpecDec, for instance, for their own traffic.

   12. SWswyx5:29

       SpecDec is speculative decoding.

   13. ATAli Taha5:30

       Speculative decoding, yeah, yeah. Um, like-

   14. SWswyx5:31

       You have to say it [laughs]

   15. ATAli Taha5:32

       ... sorry. Like, speculative-- The way spe-- Like, basically, if you have a, a huge model, right? And so the model is going to be generating one token at a time every single turn, every single forward pass. So we attach, like, this little, like, kind of parasite, like, this, this layer that goes on top of the model, and this model just has to predict. It does three very fast autoregressive forward passes, and it will dr- predict, like, you know, three certain tokens, and then you do one forward stage over the entire original model in order to see if those predictions were correct or not, and then you accept them or you reject them. Now this draft model is, is traffic specific, so if you like h- like, um, you know, Philip said, if you're summarizing Harry Potter books, I can train exclusively that draft model on Harry Potter books, and I can guarantee you that I'm gonna accept the three tokens every single time. And so with that case, I increase your decode speed. I wouldn't be able to provide this to you if you're on a shared endpoint 'cause I have no idea if you're doing Harry Potter, or if you're doing coding, if you're doing English, like we don't know. Um, also there's a thing in the book that mentioned that if they really cared about a specific threshold, chapter four, I think. Do you remember that?

   16. PKPhilip Kiely6:30

       Yeah. The, the things that you can do is you, you can set, you know, a specific like batch sizing, a specific like parallelism strategy if you're trying to optimize for like throughput versus, uh, latency. Um, you can, you know, maybe, maybe a NVFP4 quant doesn't pass your benchmarks and you wanna run a model at higher precision, you could do that. Um, there's just a bunch of reasons why you might wanna have your own endpoint and, and the, the biggest one of course just being like you don't have to deal with someone else doing 100 million of tokens of benchmarking traffic at the endpoint when you happen to be trying to serve your users.

   17. ATAli Taha7:05

       Yeah.

   18. SWswyx7:05

       I think one thing that is-- tha-that is a classic journey. You know, like, uh, it's basically people is asking the, what happens when you type Google into the browser. Um, tool calling, is that just, you know, you're tr- you're generating JSON or is there more complication beyond that?

   19. ATAli Taha7:22

       Certain customers that we have, they have their own post-trained models, and so they, they demand a tool calling that's not just like, you know, parse a file or, you know, go find the weather. It's, it's something that's very specific, and you have to do post-training on this. And if the post-training on the model is not good or if the quantization after the post-training to get the inference to be fast, the model will struggle reading the JSON file and reading the tool calling. But it doesn't re- it doesn't require its own like sandbox. It's not like it's going to use that tool calling to like, like escape a sandbox or like it doesn't have to be contained. It can just be a normal dedicated deployment. The challenge with tool calling more and more seems to be that the companies want certain tool calling which, which is a very sensitive thing to train. And because you're dealing with all of the JSON outputs, if it doesn't like close the end of the request in a very certain manner, you end up with a model that did the tool calling and like the thinking. And so as a result of that, it didn't see the result and just hallucinated the result as it decoded. That seems to be the, the most challenging thing with tool calling, not really the sandboxes more.

   20. PKPhilip Kiely8:21

       Yeah. That's a challenge on the training side. And then on the influence side, there's work that you can do to scope the possible output. So we published this actually at this point close to two years ago, um, the, the solution to this problem, which is you basically make a state machine, and you use that to constrain the output to a specific format. So this is the structured output problem. If you remember back in-

   21. SWswyx8:52

       Yeah

   22. PKPhilip Kiely8:52

       ... back in the day-

   23. SWswyx8:52

       Specific grammars, uh-

   24. PKPhilip Kiely8:53

       Yeah

   25. SWswyx8:54

       ... GML had this thing.

   26. PKPhilip Kiely8:55

       Yeah. So it's, it's like the, the old school, like make sure this is only JSON return, only JSON or my grandma's-

   27. SWswyx9:03

       Yeah

   28. PKPhilip Kiely9:03

       ... gonna die type of prompt.

   29. SWswyx9:04

       Is it BNF grammar? At some point, the OpenAI had released a thing that was like, yeah, if you wanna constrain your output, write BNF grammar, uh, back as Nor.

   30. PKPhilip Kiely9:12

       In our influence system, it's just a, yeah, specified output format, um, and, and you get the, the guarantee that your output's gonna be structured along that format. Um, and so applying that to, to tool calls can like help cut down on... Obviously, you can still call the wrong tool or call no tool. It doesn't solve the, the certainty problem-
3. 11:26 – 19:06

   Launching Production-Ready Open Models

   1. PKPhilip Kiely11:26

      about that.

   2. ATAli Taha11:26

      Like you're, you're also bounded by the software or that the model is gonna integrate with if the software is built with JSON for the tool calls or if the company that you're-- you know, if your customer says that this is how our software works and the-our tools are interfaced with JSON, you can ask them to like, you know, change their, their software and, and say like, "Yeah, this is gonna be better for the model." Y- but like with the wet training shouldn't be that much of a A difference. Also more profitable if it outputs more tokens probably.

   3. SWswyx11:49

      [laughs] Depends on your business model.

   4. ATAli Taha11:51

      [laughs]

   5. SWswyx11:51

      You know, it really depends. Uh, but I will say that, you know, as a writer with like experience a lot with AI-generated output, I do try to move from text to JSON text-

   6. ATAli Taha12:02

      Mm.

   7. SWswyx12:03

      ... which is very long JSON, right? Like, like there's paragraphs in every field because I'm trying to structure it, right?

   8. ATAli Taha12:08

      Right.

   9. SWswyx12:08

      I want you to first make factual statements, then make opinions-

   10. ATAli Taha12:12

       Mm-hmm

   11. SWswyx12:12

       ... then make bullet point summaries, have dates, have ent-entity references-

   12. ATAli Taha12:16

       Mm

   13. SWswyx12:16

       ... have your sources for references, all these things. Anyway, so, so these are things that like I think people who really, really experiment with structural output have to really care about. But, um, let's, let's sort of recurse up the stack a little bit. Before we started recording, you actually mentioned something which is really cool, which is that there's a lot of engineering that-- inference engineering that goes on when a new model provider releases a new model, right? So let's call it GLM 5.2, Kimi K3. I had pre-previously assumed, especially if it's like, well, GM- GLM 5 to 5.1 to 5.2, like you know that you've supported them before. Is it that much work?

   14. ATAli Taha12:51

       It's a lot of work [laughs] .

   15. SWswyx12:52

       Yeah. Okay. So like, you know, a lot of people, all you guys, right-

   16. ATAli Taha12:56

       Mm

   17. SWswyx12:56

       ... whenever a new model launch, like, uh, people rush to say like, "Oh, Hugging Face supports this, Firework supports this, Space10 supports this." And I'm like, "Yeah, of course we support it." But what, what, what goes into that? What goes behind the scenes?

   18. AFAlessio Fanelli13:04

       I think it's more than just support it too, right? Uh, it benefits the consumer a lot. Like, I think it was with, uh, Kimi K2.5 or GLM 5.2, the latest, there was sort of an inference war, right? X provider is at 90 tokens a second. The next day we're at 150. The next-

   19. PKPhilip Kiely13:20

       I, I kinda kicked that off with, uh, GLM 5.2

   20. SWswyx13:22

       [laughs]

   21. PKPhilip Kiely13:22

       I wrote a Twitter article about... It got like half a million views, um-

   22. ATAli Taha13:27

       Baseten being number one-

   23. PKPhilip Kiely13:28

       Yeah

   24. ATAli Taha13:28

       ... or it was for something else [laughs] .

   25. SWswyx13:29

       [laughs]

   26. PKPhilip Kiely13:29

       Yeah. Which, uh-

   27. ATAli Taha13:30

       Oh, my God

   28. PKPhilip Kiely13:30

       ... which then got everyone really excited about, hey, how can we, you know, bend, bend tracks a little bit further? And, um, there was a difference between support the model, as in like I can make a token out of this model, and support a model as in I have a production-ready API from this model.

   29. SWswyx13:49

       Mm.

   30. PKPhilip Kiely13:50

       Getting to the point of I can make a token out of this model is not that hard because generally the, um, open source inference engines, your VLMs, SGLangs of the world oftentimes even receive weights ahead of time, maintainers do, or the people making the model merge PRs to ensure support. So you generally can, you know, just kind of get it working on the standard open source stack without too much pain in most cases. The challenge is, you know, every inference company is gonna have own proprietary stack. Um, you know, some open source components, some, some in-house stuff. And for any arbitrary model, there's going to be some new stuff. Sometimes you get lucky, like K, uh, 2.5 to 2.6 was like pretty, pretty similar. Um-
4. 19:06 – 28:22

   Model Retrofits, Failure Modes, and Nondeterminism

   1. AFAlessio Fanelli19:06

      Same with, uh, Clip and Meta Clip, where you go from just captioning to building out questions-

   2. SWswyx19:12

      Right

   3. AFAlessio Fanelli19:12

      ... off the image and how much better you can get performance.

   4. SWswyx19:14

      Right. Right, right.

   5. ATAli Taha19:16

      Yeah. But what's, what's so exciting about this is if you look at a model like this... Now, obviously this is a little bit more of a research project. It's not, you know... It, it got to 56% on MMLU Pro, I think, so not, not quite Fontio. But if you're running this model, you haven't suffered any loss on your GLM-5.2 quality. If you don't have an image, it'll just behave exactly the way it used to. And ultimately the-

   6. SWswyx19:39

      Which, which in the inference code, you literally do not include the other part, right?

   7. ATAli Taha19:43

      Yeah, I mean, you would just skip the encoder if you don't have an image input.

   8. SWswyx19:46

      Okay.

   9. ATAli Taha19:47

      Um-

   10. SWswyx19:47

       Just confirming.

   11. ATAli Taha19:48

       Yeah.

   12. AFAlessio Fanelli19:48

       Does it affect a lot on the overall inference side? Like, you're not adding much, you're adding a very small vision encoder. These are typically like-

   13. ATAli Taha19:55

       No, super fine

   14. AFAlessio Fanelli19:55

       ... less than a billion parameters, right?

   15. ATAli Taha19:56

       Yeah. It's-- I mean, there's a little bit less standardization among vision encoders.

   16. SWswyx20:01

       Yeah.

   17. ATAli Taha20:01

       Um, so the s- sort of support matrix can be a little bit, uh, sparser. But overall, um, yeah, it's, it's a pretty, it's a pretty minor component of the overall system. And, and ultimately, what you get out of the system is all of a sudden you have Kimi Vision, GLM weights, and DeepSeek attention all in one model.

   18. SWswyx20:20

       [laughs]

   19. ATAli Taha20:20

       And that's, I think a lot of the power and beauty of open source, is that you can take all of these different components and, and combine them together into a system that's better than anyone-

   20. SWswyx20:29

       Yeah

   21. ATAli Taha20:29

       ... can be individually.

   22. SWswyx20:30

       People used to say that you would also do frankenmerges where you would take like-

   23. ATAli Taha20:34

       Yeah

   24. SWswyx20:34

       ... layers from each model.

   25. ATAli Taha20:36

       Mm.

   26. SWswyx20:36

       No-- Does anyone do that anymore?

   27. ATAli Taha20:38

       Well, to, to your point previously when you were mentioning like, um, the work that goes into supporting a model when it first comes out, like GLM-5.2 or Mini- Minimax M3 or whatever the case is. Sometimes you do have to r- like you do have to switch out some things. Like for instance, the Minimax M3 head uses full attention. And with full attention, you end up with this like insane bottleneck in Specter 'cause you're doing autoregressive token generation for three tokens, and you're doing this like, like O (n) squared over all of the tokens that are in your sequence. Um, your KV cache is like very large because it's not sparse, it's not top K. So we find it better to like, okay, we're gonna replace this, you know, we're gonna replace this layer with a layer from another model that's using like GQA, for instance. And then just with the right training, you can get it to have the same acceptance rate. So it is, it is very possible to, to retrofit layers from other models and very much needed, actually. If a layer is like inefficient, the training just becomes the challenge. Like how do you ensure that you train it properly? Which again, to your earlier point, is like the, the mesh between training and inference as in like you need very good training in order to do fast inference. That's like I feel like more and more becoming true.

   28. SWswyx21:46

       Yeah. Anything else on the support side when you, when you say like get it to fully production ready?

   29. ATAli Taha21:51

       Yeah, I think that there's also a question of just, you know, we can test a model to a pretty extensive degree, but we're trying to get it out quickly, and then you see a bunch of other people test it and you get interesting results. There was a- an issue with, um, GLM briefly where we had some like mode collapses where it would just output the same token over and over again for certain prompts on certain temperatures. Like once you expose an endpoint to, to the real world, there's going to be, you know, so many more varieties of, of things given to it that, that you're able to, you know, discover and, and patch things. So it's not just a, you know, day zero process. It's then like for the first week, for the first month, if a model remains popular, like how do you both fix bugs and then continue to push the envelope on performance?

   30. AFAlessio Fanelli22:46

       What do you mean you don't want your model outputting SSSSSS? [laughing]
5. 28:22 – 32:15

   Quantization and Canceling Errors

   1. PKPhilip Kiely28:22

      If you think of a sort of golden implementation of a model that performs exactly the way the model is designed to perform, I think of quality as how close are we getting to that, you know, 100% fidelity of the model.

   2. AFAlessio Fanelli28:37

      Mm.

   3. PKPhilip Kiely28:37

      You can also, of course, think about quality from the training side and how do you push yourself past 100%, but when I think about purely inference optimizations, it's getting faster while staying as close to that 100% fidelity mark as, as possible. And certainly our standard internally is that, like, you should not be able to tell the difference between our API and a, you know, sort of official API. I think Kimi in particular does a good job of vendor benchmarking here-

   4. SWswyx29:06

      Yes

   5. PKPhilip Kiely29:06

      ... where they have-

   6. SWswyx29:06

      They, they released an actual vendor benchmark.

   7. PKPhilip Kiely29:08

      Exactly, yeah.

   8. SWswyx29:08

      'Cause they accused, uh, some people, Amazon? There was some provider that was not doing very well on Kimi's benchmark.

   9. PKPhilip Kiely29:15

      Yeah. So, um-

   10. SWswyx29:16

       [laughs]

   11. PKPhilip Kiely29:16

       ... it would reflect, it would reflect really poorly on them.

   12. AFAlessio Fanelli29:17

       This was a long time ago, right?

   13. PKPhilip Kiely29:19

       No.

   14. SWswyx29:19

       Yeah, like, like three-

   15. AFAlessio Fanelli29:20

       They also-

   16. SWswyx29:20

       ... four, four or five months ago.

   17. AFAlessio Fanelli29:21

       This also happened with... I don't remember which model, but they, they pulled out quite a few, and then they started a whole chart about this. It might've been-

   18. SWswyx29:27

       Kimi vendor verifier.

   19. PKPhilip Kiely29:29

       Yeah.

   20. AFAlessio Fanelli29:29

       Yeah.

   21. ATAli Taha29:29

       Yeah, 'cause you, 'cause you'd be pissed, right? Like, if you're-

   22. SWswyx29:31

       Yeah

   23. ATAli Taha29:32

       ... if, like, um, like if I'm a consumer and I'm using, like, Amazon's endpoint for instance, and I, I'm using Kimi and I'm like, "Oh my God," like, "this is bad," I'm not gonna say, "Oh, Amazon quantized the model in a bad way." I'm gonna say, "Oh, Kimi sucks." Right?

   24. PKPhilip Kiely29:42

       Yeah.

   25. ATAli Taha29:42

       So it seems like that, that makes sense.

   26. PKPhilip Kiely29:43

       Yeah, they care. They care. Um-

   27. AFAlessio Fanelli29:45

       Justifiably. [chuckles]

   28. PKPhilip Kiely29:46

       Yeah, just fine.

   29. AFAlessio Fanelli29:47

       Uh, uh, this is probably a stupid question, but just checking, has anything improved from mean quantization?

   30. PKPhilip Kiely29:52

       Yeah.
6. 32:15 – 40:48

   The Race to 10× Faster Inference

   1. ATAli Taha32:15

      so what Joshua showed in his mathematical proof, where he had like a verifier in, is that you can predict which layers are going to have quantization errors that will cancel out with each other, and you choose to quantize those layers. And so the result of doing this mathematical quantization is you end up with a model that's 20% more quantized than another provider, so you get 20% more throughput of it because there's more layers that are running in BF4. And your quality is better than that other quant because the layers that you chose to quantize have their errors cancel out. Like one layer is skewed to the right-

   2. PKPhilip Kiely32:43

      Mm-hmm

   3. ATAli Taha32:43

      ... one layer is skewed to the left, one layer skewed to the right. Your final logistics distribution is more similar to the original distribution of the model, so you have better fidelity. And so the way we proved this was with KL divergence. So instead of just scoring on the benchmarks, we scored the KL divergence between the logic distribution of the quantized model and the logic distribution of the original full precision model. And we showed that with this technique we get... If your, if your, you know, if your probability distribution on the logic switch token it wants to select is more of the same as the original model, you're probably gonna end up staying true to the original model. So yeah. So it seems like, like previously before this, it seemed like the industry was, well, the more you quantize, the worse it's gonna be 'cause the more loss you introduce. That's not exactly, not necessarily true. So yeah, doesn't improve it, but can cancel out.

   4. PKPhilip Kiely33:22

      I think it might be this, but reminds me a good bit about pruning actually, where you can prune off certain layers.

   5. ATAli Taha33:27

      Mm-hmm.

   6. PKPhilip Kiely33:27

      But very interesting. Didn't know this was a whole paper you guys put out. [laughs]

   7. ATAli Taha33:31

      [laughs] It w- it's... Fun fact, it was originally 72 pages, this paper, and then we decided-

   8. PKPhilip Kiely33:35

      Wow

   9. ATAli Taha33:35

      ... we can't tell. We couldn't release it. So it's now 45. [laughs]

   10. PKPhilip Kiely33:39

       [laughs]

   11. SWswyx33:40

       Still, still 39 pages, so very, very substantive. We talked about evals and, and all these things and, uh, like what's possible in terms of speed up? I guess, like it's like probably like the number one-

   12. ATAli Taha33:49

       Mm-hmm

   13. SWswyx33:50

       ... thing that people do wanna care about, and it's something that you wrote about in your post. Um, like official API is 70 tokens per second, and you push it up to 90. Is that like a normal thing?

   14. PKPhilip Kiely34:01

       So what's cool about working in inference, the reason that I think inference is going to be a useful place to do engineering for a long time, is that if you look at highly optimized domains like, say, finance, if you're in finance, you measure how much better you got in basis points. It's like, "Oh, I got five basis points better, like one-twentieth of 1% better." That's huge news because everything is, is so optimized. When we publish optimizations, it's, it's 20%, it's 100%-

   15. SWswyx34:31

       [laughs]

   16. PKPhilip Kiely34:32

       ... it's 200%. So there's still probably like a lot further to go, honestly. Like you'll, you'll know that inference is pretty much solved when researchers start publishing about how they got 1% faster at, at something.

   17. SWswyx34:44

       Which by the way, uh, because I am from the finance background, in the '70s, that was the margin at the time. When you did quantitative finance research, you would find-

   18. ATAli Taha34:52

       And like with 20%, 10%.

   19. SWswyx34:54

       Yes. Yes.

   20. PKPhilip Kiely34:54

       Yeah.

   21. ATAli Taha34:55

       And now it's tiny fractions.

   22. SWswyx34:56

       For, for those, for those people interested, look up Andrew Lo's paper. Uh, he had a, a really interesting illustration of quant, uh, stat arb, uh, distribution, uh, narrowing down from like those kinds of 20% differences in the '70s, uh, down to nothing today, um, which is very, very cool.

   23. PKPhilip Kiely35:13

       Exactly, and, and we're at the beginning of, of the same type of, of, of thing. Now benchmarking is hard. Um, I think any- anyone will tell you that. And, and benchmarking provider speeds is hard because there's so many variables that go into it. What hardware are you using? How much load do you have on the system? What's the exact nature of the prompts and input and output sequence lengths? All that kind of stuff. But overall, when you start stacking these improvements, you're looking at multiples. Um, you can look at it... The, the most common form, of course, is TPS, tokens per second, which is bad naming by us in the industry 'cause there's actually two tokens per second. There's tokens per second, the throughput number, and, and the latency number.

   24. ATAli Taha35:56

       TCT, yeah.

   25. PKPhilip Kiely35:56

       Um, to- like total tokens per second out of the, out of the GPU as the throughput number. Most people only care about tokens per second as the latency number, which we should call ITL, intertoken latency, but we don't. Um-

   26. SWswyx36:08

       [laughs]

   27. PKPhilip Kiely36:09

       Anyway, so you can imagine a sort of standard API without many optimizations for a 1 trillion parameter model operating somewhere in the 30 to 50 tokens per second range for, for a reasonable traffic profile. Um, and we generally see the goal of, you know, pushing to 10X that. Um, but, you know, not necessarily day zero, but by stacking enough optimizations, um, if you have, say like four optimizations, each of which doubles performance... Or sorry, three optimizations, each of which doubles performance, uh, then you, you stack that up. That's an 8X gain. That's kind of the, the Order of magnitude that we're working with in this, in this space. We're trying to make things substantially faster, not just go from like 70 to 90.

   28. AFAlessio Fanelli37:03

       Are you saying you've, you, you have done that?

   29. PKPhilip Kiely37:05

       So let's say you have as a reasonable baseline, 30 or 40 tokens per second. Uh, you can, you can achieve 10X that. So like on GLM 5.2, um, if you wanna get unquantized, uh, perhaps on HALOs even, um, and you're just using an off-the-shelf inference engine with no particular optimizations, no, no speculator, um, nothing, nothing extra around like KV routing, no disaggregation, uh, you're, you're probably, yeah, looking at that like 30 to 40. Do you think that's like a reasonable baseline?

   30. AFAlessio Fanelli37:36

       Right. Right.
7. 40:48 – 50:18

   Dynamo, Speculation, and Local vs. Data-Center AI

   1. PKPhilip Kiely40:48

      um, quantized weights is for someone who really knows what they're doing, hours to days of work. Um, building the speculator, again, like hours to days of work, and the, um, Disag setup, hours to days. D- well-

   2. ATAli Taha41:03

      Yeah [laughs]

   3. PKPhilip Kiely41:03

      ... okay, well once, once you have it working-

   4. ATAli Taha41:04

      Once set up. Once set up, yeah

   5. PKPhilip Kiely41:05

      ... yeah, yeah, getting Disag working for the first time, I'm saying of course is, is very difficult. But-

   6. ATAli Taha41:08

      Mm-hmm

   7. PKPhilip Kiely41:09

      ... the, the, the marginal implementation is-

   8. ATAli Taha41:13

      If you're just grabbing, like if you are a person, like just a normal consumer who has access to like a node of B200s and you're wondering, "How can I just host it myself?" You don't need to quantize the model yourself. There's always gonna be like an open source quantized checkpoint. Nvidia's gonna push one out if no one else does. You-- usually the providers will have their own Spectec that they've trained as well. You don't need to train your own Spectec. You can just use that as well. Um-

   9. PKPhilip Kiely41:33

      Yeah. Like, like Kimi, uh, uh, GLM 5.2 has its own MTP.

   10. ATAli Taha41:38

       Right. Right.

   11. AFAlessio Fanelli41:38

       Thanks.

   12. ATAli Taha41:39

       Multi-token prediction [laughs] .

   13. AFAlessio Fanelli41:40

       Yes.

   14. ATAli Taha41:41

       I'm just kidding. I'm just kidding.

   15. AFAlessio Fanelli41:41

       Use it as-

   16. ATAli Taha41:42

       I can do it for you in case I get it wrong, you know?

   17. AFAlessio Fanelli41:44

       No.

   18. ATAli Taha41:45

       Uh-

   19. AFAlessio Fanelli41:45

       Actually, you should correct if we're wrong, but their multi-token prediction can be used for self-speculative decoding.

   20. ATAli Taha41:52

       I'm actually not sure. I'm not gonna correct that.

   21. AFAlessio Fanelli41:53

       Okay. I'm semi-confident in that-

   22. ATAli Taha41:54

       Okay. Yeah, yeah

   23. AFAlessio Fanelli41:54

       ... but someone can check. Um, but you know, it's useful to paint the story of, okay, not just the average person, but say a company wants to switch from serverless inference to-

   24. ATAli Taha42:03

       Mm-hmm

   25. AFAlessio Fanelli42:03

       ... I wanna throw this up on, you know, I wanna rent some GPUs, throw it up. These are the steps you take to do significantly faster than just put it behind Veo or something.

   26. ATAli Taha42:13

       Right.

   27. AFAlessio Fanelli42:13

       I was waiting for a mention of Dynamo. Uh-

   28. ATAli Taha42:15

       Mm-hmm.

   29. AFAlessio Fanelli42:15

       I feel like, uh, that's supposed to be the baseline that you measure against.

   30. PKPhilip Kiely42:19

       I would think of Dynamo as less of a sort of out-of-box system and more of a toolkit for building with. So when we talk about doing KV-aware routing, when we talk about doing KV out- offloading, when we talk about doing, um, PD disaggregation, Dynamo fundamentally is... By the way, Dynamo is an, an open source library from Nvidia. Um-
8. 50:18 – 1:00:55

   Model Parallelism, Auto-Tuning, and Mega Kernels

   1. ATAli Taha50:18

      the bandwidth. 'Cause on the B200s, you have like 3.5 terabytes per second. You don't need to-

   2. PKPhilip Kiely50:21

      Mm

   3. ATAli Taha50:22

      ... you know, decrease the, the storage that much. You don't need to do, you know, FP4 KB cache. You don't need to use TurboQuant. There's, there's, there's better optimizations to be made. But on, on edge devices, it's extremely important, it's extremely useful. So, you know, it s- seems to be like different optimizations there, but then they're all uniquely combined with like, oh, you wanna quantize the model, you wanna do speculative decoding, like certain common prefixes with both-

   4. PKPhilip Kiely50:43

      Principles

   5. ATAli Taha50:43

      ... yeah, exactly, exactly, exactly.

   6. PKPhilip Kiely50:44

      They also do a lot of work on, um, p- model parallelism, especially over, uh, you know, heterogeneous topology, where you have, you know, some sparks and they're wired together with, you know, Ethernet, D- DGX sparks. Um-

   7. ATAli Taha51:00

      Yeah, yeah. This is the Exo Labs guys

   8. PKPhilip Kiely51:01

      ... yeah, you have, you know, a number of, uh, Mac Minis stacked up.

   9. ATAli Taha51:05

      [laughs]

   10. PKPhilip Kiely51:05

       There's, you know, the inter- they, they... One thing that, that I think we both have to deal with, although they have to deal with a lot more, is, is the interconnect between machines. Um, which is why like, you know, one thing that we do a lot is work with tensor parallelism.

   11. ATAli Taha51:20

       Mm-hmm.

   12. PKPhilip Kiely51:20

       And that's where, uh, you are using all of the, you know, all eight GPUs, um, and, and sharding the model across it. Uh, tensor parallelism is not a good fit for local AI because it assumes a very high bandwidth interconnects like NVLink. Uh, was, you know, they might be forced to do something like pipeline parallelism, which we're never gonna do unless we're doing some kind of-

   13. ATAli Taha51:41

       Yeah, for-

   14. PKPhilip Kiely51:41

       ... multi-node inference.

   15. ATAli Taha51:42

       Uh, since you mentioned it, I actually wasn't sure if we were gonna cover it, but let's briefly explain tensor parallelism and expert parallelism, uh, since you have very nice images.

   16. PKPhilip Kiely51:50

       You wanna pull the book?

   17. ATAli Taha51:51

       I just wanna-

   18. PKPhilip Kiely51:51

       Yeah, yeah. Let's, let's get the-

   19. ATAli Taha51:52

       So I just wanna show off your images. [laughs]

   20. PKPhilip Kiely51:54

       [laughs] Yeah. Uh, shout out, shout out to Luke from Baseten's design team for making these, these beautiful images. Oh, that's a, that's actually... Before we get into this, just one other difference is we, we talk a lot about the active parameters of a mixture of experts model, and for local inference folks, that matters a lot because if you have a batch size of one, you're only activating that many parameters. When we do inference in the data center-

   21. ATAli Taha52:17

       Yes, I was gonna, I was gonna bring that in the diffusion conversation. Uh-

   22. PKPhilip Kiely52:19

       Oh, yeah.

   23. ATAli Taha52:19

       [laughs]

   24. PKPhilip Kiely52:20

       Yeah. We, uh, at, at... When we go through like a, a MoE model, um, and we host it, um, for an API, we assume that all parameters are gonna be active because-

   25. ATAli Taha52:30

       You're fetching

   26. PKPhilip Kiely52:31

       ... throughout your batch-

   27. ATAli Taha52:31

       Yeah

   28. PKPhilip Kiely52:31

       ... you're gonna, you're gonna hit everything. Um, cool. So broadly, tensor parallelism you can do with any model. Expert parallelism, you can only do with MoE models. Um, effectively all models today are MoE models, um, that are, you know-

   29. ATAli Taha52:46

       Sort of-

   30. PKPhilip Kiely52:46

       ... at least all models large enough that you would care to parallelize them across multiple GPUs. So that, that's, that, that nuance is, is less important now. Um, with expert parallelism, the idea is you put the entire expert on a GPU. Generally, you have more experts than GPUs, so you might put like N experts per GPU, like eight experts per GPU or whatever. And then you, uh, you replicate the router, which the router is very small across each of the GPUs. And then by moving the, the generation from expert to expert, with each expert being inside a GPU, they're not competing for resources. You massively increase the throughput that you're capable of doing, and the, uh, GPU-to-GPU connection b- is, is not as important 'cause there's not as much communication. Uh, tensor parallelism requires that you are able to do this like all gather, all reduce.
9. 1:00:55 – 1:10:03

   Rubin, GPUs vs. ASICs, and Custom AI Chips

   1. ATAli Taha1:00:56

      When I look ahead to, you know, what's going to be new with, with Rubin, I think that like Dynamo gives me a lot of technical hints around like what kinds of work is going to be very valuable. Obviously, we're continuing some trends from Blackwell, right? NVFP4 is big. The amount of compute that they have behind NVFP4 tensor cores is, is massive. Um, we'll, we're gonna talk about video, I think, at some point, and, and that's the, the big barrier there. You've got, you know, much, much faster memory bandwidth, um-

   2. PKPhilip Kiely1:01:28

      But which was the same thing that, that made Blackwell so good. Um, but the, the big thing is more systems thinking. You have more emphasis on the CPU to GPU interconnect, more emphasis on the interconnect between GPUs. And when you look at Dynamo, it's a system entirely designed around how do I move the KV cache to where it needs to be when it needs to get there? Um, so I think that themes around like KV cache offloading, KV-aware routing and, and disaggregation are going to be substantially more important in the Rubin era, which means that inference engineering becomes not just a like CUDA kernel problem, but also like a very traditional hardware infrastructure problem, which is something, you know, we've been building toward for, for a long time and something that's like very exciting to me because we're gonna see sort of multiple domains colliding and the ability to reason from the kernel level like up to the hardware level and, and back down is, is going to be very valuable.

   3. ATAli Taha1:02:30

      I will take what Philip said one step further actually into that. It's, I think, trending towards becoming exclusively an infrastructure problem where like problems of PG, Disac, training, Spectac. But writing kernels is not going to be much of a problem because the GPU is moving more towards being an ASIC, where it's-- you're just, you're just trying to orchestrate what happens on the GPU, but you're not actually controlling it thread by thread level. And you see this with like QTAL, QDSL, like you're, you're just working at levels of like tiles of data, but you're no longer working at controlling what each thread does on the GPU that's being taken care of for you. Um, so I guess, do you agree that a GPU and future GPUs are trending more and more towards becoming ASICs that just need to be launched and then they do the data operation based on your conversations with other people?

   4. SWswyx1:03:15

      Oh, I mean, yeah, no. Uh, that, that, that is a section of the market.

   5. ATAli Taha1:03:19

      Right.

   6. SWswyx1:03:20

      And obviously, ASICs can do, uh, a lot more performance for only their, their workload.

   7. ATAli Taha1:03:25

      Right.

   8. SWswyx1:03:26

      And the, the G in GPU makes, makes them continue to be very general.

   9. PKPhilip Kiely1:03:30

      Yeah. The, the-- I think that there's like a spectrum-

   10. SWswyx1:03:32

       Actually, it's, it's graphics, but-

   11. PKPhilip Kiely1:03:34

       Yeah. [laughs]

   12. SWswyx1:03:34

       I, I keep saying this, I have to correct myself in, in case people come at me for getting the, the G wrong.

   13. PKPhilip Kiely1:03:39

       Yeah. It's, and it's like, it's like a spectrum, right? Of, of a very, very general purpose compute to something like a Talos where you've got the, the hardware built for a specific set of model weights.

   14. SWswyx1:03:50

       The weights burned-

   15. PKPhilip Kiely1:03:51

       The weights

   16. SWswyx1:03:51

       ... into the chip. [laughs]

   17. PKPhilip Kiely1:03:52

       Yeah.

   18. SWswyx1:03:53

       No loading.

   19. PKPhilip Kiely1:03:54

       I don't-- I wouldn't say that like, that, that we're, we're, we're going all the way there. It's, it's more like along the spectrum, it's a step in the direction of more specialization within the hardware.

   20. SWswyx1:04:05

       Yeah. I'm curious, I feel like he was driving towards something.

   21. ATAli Taha1:04:08

       I guess my point is being bearish on... Like you, you say, you say, like everything else apart from burning the weights into the chip. Burning weights into the chip is like impractical because you wanna fine-tune, you wanna optimize, you wanna, you wanna quantize, you wanna release new checkpoints of the model. If it's burned into the chip, the chip's useless in like a month or two, right? I guess, I guess my point is how can you not-- uh, like seeing Nvidia more and more specialize it, like take its GPUs from a general programming paradigm where you're just-- it's a general computer that you can use to program threads, and with every new generation, you're putting more and more specialized instructions, specialized tensor cores, specialized, you know, UMA instructions, uh, things that will, will allow you to, to just control it almost as an ASIC, almost as a collection of ASICs.

   22. SWswyx1:04:47

       Mm.

   23. ATAli Taha1:04:47

       How can you look at this trend and then still be bullish on companies that are coming up with ASICs for AI?

   24. SWswyx1:04:55

       Mm.

   25. ATAli Taha1:04:55

       In the sense that, in the sense that if-

   26. PKPhilip Kiely1:04:55

       Yeah, because they're, they're sort of re-

   27. ATAli Taha1:04:57

       What?

   28. PKPhilip Kiely1:04:58

       They're evolving towards that direction.

   29. ATAli Taha1:04:59

       They're almost evolving towards an... Like, like as an, as in Rubin, I guess comp-- like compared to Ampere or, you know, T4, Rubin is, is, is basically an ASIC. It is, it's basically just a thing that is, is used-

   30. SWswyx1:05:11

       Programmable ASIC?
10. 1:10:03 – 1:12:42

    Giant Models and the Limits of GPU Memory

    1. AFAlessio Fanelli1:10:03

       enough and use enough tool calls and be agentic enough, uh, can it just web search, tool search-

    2. PKPhilip Kiely1:10:09

       Mm-hmm

    3. AFAlessio Fanelli1:10:09

       ... write code? Do you really need to keep squeezing more? We will because you guys will make it cheap and fast and smaller, and I can swap it in. But at some level, like you give me 5.2 today or say whatever, 120 B model, I can run with it for quite a while, right?

    4. ATAli Taha1:10:24

       This is assuming like you don't need intelligence.

    5. AFAlessio Fanelli1:10:26

       I think there's a lot of intelligence-

    6. SWswyx1:10:27

       You need, you need reliability and predictability. Like I'm an enterprise like-

    7. PKPhilip Kiely1:10:31

       Mm-hmm

    8. SWswyx1:10:31

       ... like this is tried and tested. It is signed off by like my 5,000 stakeholders.

    9. ATAli Taha1:10:35

       Right.

    10. SWswyx1:10:35

        Like I'm [chuckles] not touching it.

    11. PKPhilip Kiely1:10:36

        It runs a batch job every day-

    12. AFAlessio Fanelli1:10:38

        [laughs]

    13. PKPhilip Kiely1:10:38

        ... and I like the results.

    14. SWswyx1:10:41

        Yeah.

    15. PKPhilip Kiely1:10:41

        The results are predictable. Yeah.

    16. AFAlessio Fanelli1:10:43

        Yeah, yeah. It doesn't make sense to keep using them. Like stuff gets sparser, cheaper, better.

    17. ATAli Taha1:10:47

        Right.

    18. AFAlessio Fanelli1:10:48

        But that doesn't mean that old models, GLM 5.0 isn't usable, right?

    19. PKPhilip Kiely1:10:53

        Mm.

    20. AFAlessio Fanelli1:10:53

        If we hit a stall, say, for whatever reason, there's still a lot that can be squeezed out.

    21. SWswyx1:10:58

        We're gonna run out of time. I, I did wanna also make sure. Yeah. Yes, actually we happen to have this diagram. Pull-- Compare this versus any Cerebras diagram, right? Um, I don't think Edge10, uh, medics have, have put out public, uh, charts yet. Uh, but the complete re-- the real estate is very different. The size is very different, right? This is not wafer scale, right? This is-- There's probably like, I don't know, a few hundred of these on a, on a wafer. Um, I don't, I don't know how big-

    22. ATAli Taha1:11:19

        Right. Right. Right

    23. SWswyx1:11:20

        ... the, [chuckles] the comparison is, but like, uh, it is a, it is a very like real estate allocation-

    24. PKPhilip Kiely1:11:25

        Few dozen

    25. SWswyx1:11:25

        ... uh, difference.

    26. PKPhilip Kiely1:11:26

        Few, few dozen, I would say.

    27. SWswyx1:11:27

        Few dozen. Yeah.

    28. AFAlessio Fanelli1:11:28

        Before we move from hardware, I have two quick questions. One, the latest Kimi, which is really big, uh, three trillion-

    29. SWswyx1:11:34

        Yeah

    30. AFAlessio Fanelli1:11:34

        ... doesn't fit on most hardware on single node.
11. 1:12:42 – 1:21:47

    AI Video, Quadratic Attention, and Autoregressive Generation

    1. PKPhilip Kiely1:12:42

       NVL 72 on, on, uh, Blackwell as well. But, um, you, you can't necessarily assume you're gonna do inference on that.

    2. AFAlessio Fanelli1:12:48

       Mm-hmm.

    3. PKPhilip Kiely1:12:49

       Um, there's a whole lot more 8x racks in the world than there are NVL 72s.

    4. AFAlessio Fanelli1:12:54

       Yeah. I guess my last quick question on hardware was, do you notice anything with hardware generations for new pre-trained base models? So one of the things you said for efficiency is you can swap hardware. That's one of the 2x gains. Um, when we see new stuff coming out training-wise on Rubins, uh, any changes on logs? Does this affect what type of models we will be seeing when these are more available? And can you-

    5. PKPhilip Kiely1:13:21

       They, they get bigger. Like people understand the ceiling that you have in terms of How many parameters of a model you can run, uh, given the sort of latest inference hardware, and that kind of forms a ceiling. And so, for example, when DeepSeek R1 came out, it was, you know, it was 671 billion parameters, which at the time was really huge, and I think did a lot to push us to really quickly adopt Blackwell and, and get good at serving on Blackwell. Um, so yeah, it's, it's, it's mostly in my mind about, uh, model size and then about matching the architecture and the native quantization to the, the target hardware, like we talked about with like, you know, all Nemotron models of NVFP4, for example.

    6. AFAlessio Fanelli1:14:07

       So we talked a lot about LLMs.

    7. PKPhilip Kiely1:14:10

       Mm-hmm.

    8. AFAlessio Fanelli1:14:10

       You have a lot more in the book. What about audio, video? What's the other side of inference engineering? Ali, you're pretty big in video diffusion.

    9. ATAli Taha1:14:18

       Video diffusions I think are... Like, they're just shaped... A lot of the stuff that you can think about, reason about with LLMs being autoregressive, with video diffusion, it's, it's, it's not the case. For instance, you don't do, you don't do batching. You-- every request just comes in on one GPU and it serves one GPU. You don't have to shard. The models are a lot, are a lot smaller, like Wan2.2, for instance, is a 20 billion parameter model. Um, you don't need to worry about... It's just like orders of magnitude smaller than the best LLMs. And it's one of those spaces where the, the open-source models are... Like with LLMs, we see Kimi K3 is almost comparable to, you know, Mythos or like GPT-5.5. The, the difference between the best open-source LLM and best open-- uh, closed source LLM is very small. Like, it used to be six months. I don't think it's six months anymore. I think it's like basically almost on par. Video models are definitely not. There's a huge gap. If you look at the best video that you can generate today with an open-source model like Wan2.2 versus something like with Kling or Veo, difference is night and day. So it, it creates this disparity where media companies will choose to go most of the time to closed-source models, for instance. And I were to tell you, "Hey, I can generate an entire three-hour movie for you with this model, and I'll optimize it so that you only have to pay me $10." But if they were to do it on a closed source, they'd have to pay $1,000, which is 100x. Like I'm 100x cheaper, but it's still $1,000. They're still gonna choose to do all of their cuts with Veo and Kling. So the... It's like a chicken and egg cycle where less demand causes less innovation in the field, causes, you know, less open-source checkpoints to be released. And some of the labs that were releasing open-source models like Wan will have closed-sourced their latest models, like Wan2.7 is not open source. We're still on Wan2.2. The, the challenge with, with video models especially is the number of, of tokens. So, so, so, so video models, you want to generate a high quality model, a high quality video. So let's say you're doing 16 frames per second, that's like the absolute minimum you'll do, and let's say you'll do like 480p video. So you can think about your like dimensions, and I think I have like a good, um, just like a diagram that shows the number, the sheer number of tokens, right? Let's say you're looking at like just one video of like, you know, Sparta, Sparta 300 or whatever. So let's say we're looking at like four frames, right? Those four frames of that video, if you go just... if you're doing full attention, if you go a little bit up, like you're looking at, uh, 480p by 720 by 81 frames in just five seconds, because 16 FPS by five, right? And then you compress it down to latent space, but you're still doing 30 by like 50 by 21 tokens.

    10. PKPhilip Kiely1:16:47

        Yeah.

    11. ATAli Taha1:16:48

        Which means that for attention, for just five seconds, you're running attention on 35,000 tokens, right? So the attention becomes such a huge bottleneck. And because it's O of N squared, if you're doing like... if you extend that to like 10 seconds, well, it's just squared, 20 seconds, 30 seconds. So to generate a good cut scene of like one minute, it's almost impossible to do within the same compute time. And it's just, it just becomes unfeasible. You can't do it. And so you end up with moving towards two directions. Either you decide to do attention on the entire video at once, in which case you are forced to do sparse attention. So if you scroll back down to the origin, the, the video image, like you can see whereas on the left, for instance, I would be doing full attention where every single token in that Sparta 300 scene attends to every single other token, as you can see the sheer number of like red patches. On the right, I'm only attending to... each token only attends to like the top K, top 12.5% that's important to it, which can be like spatial. So like, you know, the token that represents the crown attends to like the head, the face, and then the head on the other frame and the previous frame, temporal locality, spatial locality, that kind of thing. This results in terrible video quality. And the whole point of the, the post or the article here is to show like how you can train and you can do all these things, but you will still suffer your quality a little bit. So you end up with one of two things. Either you bite the bullet, you have huge compute, and you do full attention over like a million tokens because you're trying to generate like two minutes of video, or you move towards autoregressive video. Autoregressive video seems to me like that is the bet that the future is gonna be making, but there are no good open-source autoregressive video models out there today. And that seems to be the... If you want to get like an hour movie, if you want to see video models generating like an, uh, like, you know, Hollywood-level movies, they have to be autoregressive in order to exceed that five-second frame. Or there has to be some insane leap that happens in compute that allows us to do full attention over like millions of tokens at the same time in a, in an efficient manner.

    12. AFAlessio Fanelli1:18:33

        Even millions of tokens, it's like you're, you're quadratic, so you're gonna get there really quick.

    13. ATAli Taha1:18:37

        Right.

    14. AFAlessio Fanelli1:18:37

        I think, uh, can you explain the pros and cons trade-offs of autoregressive? So one that comes to mind is, you know-

    15. ATAli Taha1:18:43

        Yeah

    16. AFAlessio Fanelli1:18:43

        ... the consistency across frames. You will... 10 minutes into generating autoregressive diffusion, you're gonna forget. But what, what are pros and cons of this?

    17. ATAli Taha1:18:52

        Well, like, like autoregressive LLMs, you can take a lot of your... Oh, sorry, autoregressive diffusion models. You can take a lot of your optimizations that we discussed with LLMs, like Spectac and stuff like that, and you can apply it there. And you can... If you have a very high quality scaled up model, there is no reason why I can't stream the outputs, as in I can show you the first frame and then I'm like, like, kind of like GPT back in like 2022 when you were... Like now it's almost like one-shots the text, but back then you could read and it's generating as you read. With video models, you can watch and it's generating as you watch. It generates the frames. And, and so token by token generation will allow us to, to scale a lot up and apply the attention mechanisms there. The downsides is every single- Autoregressive video model is shit. It's just terrible quality. If you-- Uh, like it's just if you put, if you put the quality of any OpenSLAM 1.02 versus any other autoregressive model, you can see like a, like a video generated by 1.02 is like, you know, a cat and dog fighting. Autoregressive model will give you like degraded Tom and Jerry quality at level one. The, the solution to generating long output then becomes, "Okay, we're not gonna use autoregressive model. We're gonna..." If you look at some of the things that like, um, like Groq Imagine or Groq Video does, and they do it really, really well, is they'll, they'll try to stitch these, you know, seven-second chunks together. And so you generate seven seconds and then you're like, "Okay, I'm gonna... Can you extend this video?" And they'll chunk two videos together. Open source doesn't seem to have the tricks that they have there and, and by definition it's closed source. We don't know what they're doing. But the closest you can get is taking the last frame of a video and feeding into like a, a text and image to video where it will take the text, the prompt, and it will take the image of the last frame, and you'll ask it to generate the next five seconds. And that's kind of like how you, you can extend this level of a model to generate like a moo, where you're just, you're constantly streaming frame by frame. But you get a drift. So you start with like, you take the image, and then you generate a video, and then that next five-second video is like lower quality, and the third chunk is like even lower, and the fourth chunk is even lower. And like sometimes you'll see things where like the, the new video is like just ever so slightly darker than the first one, and the next one is darker than the second one until like 25 seconds and you have black screen. [laughs]

    18. SWswyx1:20:53

        [laughs]

    19. ATAli Taha1:20:53

        Like it's just, it's, it's, it's a... We tried to have a demo that would show this, but it was like, it was, it was extremely embarrassing to show. Like we just decided not to because it seemed to like... But it is, it is, um, I think models will get there. They just need to, in my mind, scale up significantly and move towards being autoregressive. But the training techniques don't seem to be clear there.

    20. SWswyx1:21:12

        For those on who are interested in Groq Imagine, we did a pod with Ethan He from, from that team-

    21. ATAli Taha1:21:16

        Right

    22. SWswyx1:21:17

        ... uh, who dropped a little, a few hints but not that-

    23. ATAli Taha1:21:19

        [laughs]

    24. SWswyx1:21:19

        ... not, not, not enough that we can fully reconstruct everything.

    25. ATAli Taha1:21:22

        Right.

    26. PKPhilip Kiely1:21:22

        Specifically on this part that he explains a bit about.

    27. SWswyx1:21:25

        Yeah. So we talked about memory and, and, uh, longer context and all those things.

    28. ATAli Taha1:21:28

        But as far as I know, the- it's not autoregressive even though... Like no one in the industry is autoregressive.

    29. PKPhilip Kiely1:21:33

        Yeah.

    30. ATAli Taha1:21:33

        It seems to be. Yeah.
12. 1:21:47 – 1:27:32

    Audio, Images, and Diffusion Models

    1. PKPhilip Kiely1:21:47

       of like going off the rails behavior both in... If you sort of naively construct a video generation model as simply generating a linear sequence of frames, you can't then go back in that sequence and fix something to make the whole thing consistent. While, of course, the reason that we need all this latent space for the video model is, like you said, we keep all the tokens in memory. We iterate over that, that full sequence, and k- uh, you can, you can adjust the past in, in order to make the future make sense. So if we think about the architecture that's gonna get us there to these longer, richer sequences, it's probably, like you said, gonna be a mix of the, the autoregressive and the diffusion, um, wo- working together to do what each piece is good at.

    2. ATAli Taha1:22:33

       Well, if you get... Like you, you intuitively get why. So like English, for instance, or just writing in languages, like it's just left to right. You can stream your tokens, you can stream your chain of thought. Just even as a human, you write like... You just, you write and then you think about what's the next thing you're gonna generate, and then you write that, and then you think about your ideas, and then you generate forward. And sure, you can argue that as you write, you need to go back and you wanna edit some things. But you need to do that, you know, less often than you'd think. Whereas with video, there is no sequential... You know, the, the pixel in the top left corner of the video and the pixel in the bottom right corner of the video, they both need to attend to each other to understand how the video quality is gonna be almost as equally. Whereas with text, you don't need that as much.

    3. PKPhilip Kiely1:23:09

       Is there a parallel to audio? Like I'm not 100% confident on this, but there was a point about a year ago where there was Audio LM. There's diffusion for audio and autoregressive, and for the points you mentioned, mostly on the inference side, even though they're shorter clips, you know, most music is three to five minutes, um, we've basically swapped over to autoregressive for-

    4. ATAli Taha1:23:30

       Yeah, I can't speak to music, but speech is autoregressive.

    5. PKPhilip Kiely1:23:33

       Speech. Yeah.

    6. ATAli Taha1:23:34

       You, uh, effectively... I mean, this, this was even back with like the Orpheus architecture a year and a half ago. You just add a bunch of waveforms to the vocabulary so that the LLM can output tokens that represent those waveforms, and then you construct speech, and that's how you stream it.

    7. PKPhilip Kiely1:23:50

       That's it. Wow.

    8. ATAli Taha1:23:51

       That's my AIE talk from 2025.

    9. PKPhilip Kiely1:23:54

       Ah, nice, nice, nice. Nice.

    10. ATAli Taha1:23:55

        But it's not... With audio, it's not the same challenge, though, is it? Because you... Like audio is solved with an LLM that generates everything. Like, like with audio, it's still a, a transcript that you can generate with an LLM.

    11. PKPhilip Kiely1:24:05

        Yeah.

    12. ATAli Taha1:24:05

        So your, your audio model just needs to like transcribe it, text to speech.

    13. PKPhilip Kiely1:24:09

        For music, there was a phase of a ta- a trade-off between diffusion for music and-

    14. ATAli Taha1:24:14

        Right

    15. PKPhilip Kiely1:24:15

        ... autoregressive, and they were both pretty on par. There's probably more pros and cons to either. I just wanted to poke and see if you had takes.

    16. ATAli Taha1:24:21

        Yeah, I don't know about music specifically.

    17. PKPhilip Kiely1:24:23

        No. [laughs]

    18. ATAli Taha1:24:23

        You know, it-- what-- with what you said about editing-

    19. PKPhilip Kiely1:24:27

        Mm-hmm

    20. ATAli Taha1:24:27

        ... you writing, you know, obviously, I think my editor would tell me I actually need to do that more often-

    21. PKPhilip Kiely1:24:33

        [laughs]

    22. ATAli Taha1:24:33

        ... and, and go back and fix things. I can imagine music or poetry, for example, where you have a rhyming scheme and you might wanna go back and make a change to make it, to, to make it easier to set up a rhyme that you wanna make later on, uh, there, there being some advantage to being able to attend in both directions. Uh, but yeah, to my knowledge... You know, I, I very much bifurcate this, this influence problem into the autoregressive models which have a set of constraints to techniques and the diffusion models which have a s- set of constraints and techniques. And, um, I think of text, um, embedding, uh, voice in and voice out as being in the autoregressive side, and then image and video being in the diffusion side. There's some overlap between the two. It's not a, a perfect split, but that's the broad categorization I use.

    23. PKPhilip Kiely1:25:24

        I should point out, I think it's confirmed, right? NanoBanana and, uh, GPT Image are autoregressive image.

    24. ATAli Taha1:25:29

        It's kind of this blended approach that we're talking about. Um, but

    25. PKPhilip Kiely1:25:34

        In the, the image space, uh, it hasn't, like, made its way over to the video space, at least in the open-source world.

    26. SWswyx1:25:41

        Yeah, but like I, I assume that's not too far away if, if that is possible on-

    27. PKPhilip Kiely1:25:45

        Right

    28. SWswyx1:25:45

        ... on the... I, I think at least the Qwen image guys are trying it.

    29. PKPhilip Kiely1:25:48

        Yeah. Yeah, with the-

    30. SWswyx1:25:50

        Yeah
13. 1:27:32 – 1:40:06

    Training, Self-Optimizing Models, and Continual Learning

    1. SWswyx1:27:32

       like coherence across a, a thing that would just appear where like the, the, the end should attend to the start, and you should not have this autoregressive path dependency does make sense in principle. Uh, just the API should be different. The marketing should be different.

    2. ATAli Taha1:27:46

       None of the m- most heavily used open source or closed source models use diffusion.

    3. SWswyx1:27:51

       But isn't that like, like doesn't that point to almost like a- It, it's, it's chicken and egg because-

    4. ATAli Taha1:27:55

       Right

    5. SWswyx1:27:55

       ... what if you just give it more scale? [laughs]

    6. ATAli Taha1:27:58

       What's the, what's the largest diffusion LLM?

    7. SWswyx1:28:00

       I, uh, I don't think it's very big. [chuckles]

    8. PKPhilip Kiely1:28:02

       Uh, I don't know the parameter count-

    9. SWswyx1:28:04

       Right

    10. PKPhilip Kiely1:28:04

        ... on this one, but Diffusion Gemma is-

    11. SWswyx1:28:05

        Like under 20B I think

    12. PKPhilip Kiely1:28:06

        ... Diffusion Gemma is not one.

    13. ATAli Taha1:28:07

        I think it's a 20-something.

    14. SWswyx1:28:08

        Yeah. And yeah, you know.

    15. ATAli Taha1:28:09

        Oh, it's-

    16. SWswyx1:28:10

        Like you haven't actually tried.

    17. ATAli Taha1:28:11

        You haven't given it a big one-

    18. SWswyx1:28:12

        [laughs]

    19. ATAli Taha1:28:12

        ... and you haven't-

    20. SWswyx1:28:13

        So it's like very unfair

    21. ATAli Taha1:28:14

        ... Diffusion Gemma is a 25B, and-

    22. PKPhilip Kiely1:28:16

        A- and that's what I'm saying is, is like for its size, it, it does pretty well, um, in terms of, in terms of quality.

    23. ATAli Taha1:28:23

        It's almost like the same challenge with video models to have the same size. It's like you're comparing it to models that are much larger in scale.

    24. SWswyx1:28:29

        Yeah. Well, unless you do the whole thing where you, you have a text, uh, backbone, and then you-

    25. ATAli Taha1:28:34

        Right, right

    26. SWswyx1:28:34

        ... you know, you, you like GLM some kind of decoder thing that, uh, does that. Like, you know, we-- so we, we started off the podcast doing this for the, uh, for the inverse direction from image to text.

    27. ATAli Taha1:28:45

        Right.

    28. SWswyx1:28:45

        Uh, and I think like it's, it's, it's roughly intuitive that you can do the opposite direction.

    29. ATAli Taha1:28:49

        I agree.

    30. SWswyx1:28:50

        [laughs]
14. 1:40:06 – 1:42:53

    Closing Thoughts

    1. ATAli Taha1:40:06

       is Waterloo. But then a second derivative [laughs]

    2. PKPhilip Kiely1:40:08

       That's not changing.

    3. ATAli Taha1:40:09

       That's not changing. That's not changing. But like a second derivative question of, of which university should I hire an intern from? So if you know that the best university in the world is Waterloo, then the answer should be Waterloo. But if I wasn't just one-shotting the question and I was to ask it to like u- use its knowledge to think and then give me a second answer, or like should I hire an intern from Waterloo or MIT? It'd be like, "Oh yeah, both are good." But no, like I literally, I just edited in your knowledge base that Waterloo is the best. Why didn't you use that to do reasoning? So that's the fundamental problem with trying to change a fact in an MLP within the weight. KV cache compaction fixes that. With, with KV cache, uh, or like rather not KV cache compaction, but like if you're able to have something like, like the still paper, which we, we came out with, which is you're able to sort of make your KV almost infinite and you're able to compact in such a way that you don't lose any of the knowledge. In that case, you can actually do continual learning and you can actually solve continual learning. And this as a, it's, it's a result of, um, this, this argument that Charlie and I had, that I do concede that his point was correct and I, I do see that KV cache is, is, is the way forward. And, and in that case, I don't think inference is going to change that much because we still use KV cache and inference. You're just gonna update the KV cache, but it's gonna be like an additional step, but nothing changes in the weight, so nothing changes in inference time, nothing changes the spec that I had.

    4. SWswyx1:41:21

       Okay. Surprisingly great answer. Uh, we have it up on the blog. It's, it's a relatively recent blog, so, uh, we can- people can go see it.

    5. ATAli Taha1:41:28

       Mm-hmm.

    6. SWswyx1:41:28

       Uh-

    7. ATAli Taha1:41:29

       Hyperverve.

    8. SWswyx1:41:29

       Yeah. Otherwise, this is super enjoyable chat. I know we've like already gone two hours. [laughs]

    9. PKPhilip Kiely1:41:33

       Wow. I didn't even realize.

    10. SWswyx1:41:35

        Time flies, yeah.

    11. PKPhilip Kiely1:41:36

        Yeah. So much we didn't even cover.

    12. SWswyx1:41:37

        Yeah. This is like, we also wanted to talk about the book and all that, but you've covered the book.

    13. PKPhilip Kiely1:41:40

        I mean, yeah, I mean, ev- everyone, everyone knows about the book. [laughs]

    14. ATAli Taha1:41:43

        [laughs]

    15. SWswyx1:41:44

        Yeah. Uh, high- highest ROI thing in the history of Baseten, right?

    16. PKPhilip Kiely1:41:48

        For the hour. [laughs]

    17. ATAli Taha1:41:49

        Without a doubt. Without a doubt.

    18. PKPhilip Kiely1:41:50

        Yeah.

    19. ATAli Taha1:41:50

        Absolutely.

    20. SWswyx1:41:51

        So congrats on that. I, I, I, you know, and we've covered that in, in our meetup-

    21. ATAli Taha1:41:54

        Yeah

    22. SWswyx1:41:55

        ... which we can publish separately. Uh, but no, thank you to you guys for being so generous for sharing. I think it's a fun conversation that, um, we don't get to have enough. I think inference engineering, we never really covered head on and so to have you guys come on, uh, is, is a, is a treat.

    23. ATAli Taha1:42:09

        Oh, it's amazing.

    24. PKPhilip Kiely1:42:10

        Yeah. Thanks, thanks for having us and you know, hopefully in a year everything shifts and we can, uh, come back and say everything we were wrong about.

    25. SWswyx1:42:18

        Yeah. Yeah, yeah. I'm excited for this MegaKernels comment to get out and see what's-

    26. PKPhilip Kiely1:42:21

        [laughs]

    27. SWswyx1:42:22

        See what people say.

    28. PKPhilip Kiely1:42:22

        We gotta start stuff.

    29. SWswyx1:42:23

        [laughs]

    30. ATAli Taha1:42:24

        Should I go into hiding? I know I'm gonna get like the MegaKernel community after me.

Episode duration: 1:42:53