Why Diffusion Will Win AI Inference with Inception Co-Founder and CEO Stefano Ermon

1. 0:00 – 0:35

   Stefano Ermon Introduction

   1. SGSarah Guo0:00

      [instrumental music] Hi, listeners. Welcome back to No Priors. Today, I'm here with Stefano Ermon, who is a longtime Stanford professor, and now co-founder and CEO of Inception. Stefano has a extraordinarily broad body of work around generative modeling, but is especially well known as one of the fathers of diffusion. We talk about his company challenging the large labs, and why speed and efficiency are gonna be the name of the game in AI over the next few years. Welcome, Stefano. Stefano, thanks so much for being here.

2. 0:35 – 2:54

   Research Background

   1. SEStefano Ermon0:35

      Great to be here.

   2. SGSarah Guo0:36

      I would love for, uh, us to just start with a little bit of your research background and how you ended up starting your company.

   3. SEStefano Ermon0:42

      For sure, yeah. I've been doing research in generative models for, like, uh, basically my entire career. I started at Stanford in 2014 as an assistant professor, and I was working on, uh, yeah, building generative models. Back then, uh, the, the research area was not particularly hot. Uh, you know, the, the, the models were not quite working, uh, well. We were still, like, building little generative models over MNIST then. It was like a big success if you could generate these grainy images of digits and, you know. It was even hard to publish papers back then and, on, on, on that topic, and you had to kind of like justify training a generative model as a way to learn features from unlabeled data that then could maybe help you do better at supervised learning because that was the thing that everybody cared about. But then, you know, things took over, of course, and so it was like a... I was at the right place at the right time working on the right thing. And so I've been doing research in, in that space since, uh, since the beginning, basically.

   4. SGSarah Guo1:40

      Did you have, like a, a, um, besides a curiosity in the area, a personal hope for what the models would do back in 2014 and '15?

   5. SEStefano Ermon1:48

      Yeah, I mean, I, I always felt like that was gonna be the... that was the right way to think about, uh, kind of like learning from unlabeled data. That, like, building a generative model is really the right way to make sure you understand the structure in the data. That was kind of like the way I was getting at.

   6. SGSarah Guo2:02

      Mm-hmm.

   7. SEStefano Ermon2:02

      I, I, I was not even dreaming about the kind of capabilities that these LLMs that we, that we have today, uh, could, could, uh, could do that. Uh, but I was thinking more from a, I think, world models perspective. Like, I was, I was working a lot on images, and so thinking about, okay, like, I have a world model. I can imagine what's gonna happen if I were to stand up and walk out the door. Like, I can kind of like picture that in my mind, and that's important, uh, to make decisions and kind of like model predictive control when having this kind of model of the world requires some generative capabilities. And so I always felt like, okay, that's the right direction to work on. I felt like this is gonna be very hard as a problem, so, like, it's gonna keep me busy for my whole career, and so it's a good problem to work on. And then, of course, I was very wrong and things [laughs] evolved much faster than, than I was expecting. [laughs]

   8. SGSarah Guo2:51

      Yeah, I think that's kind of universally true, though. Um,

3. 2:54 – 5:59

   Starting Inception

   1. SGSarah Guo2:54

      and sort of walk me through the, the state of your research and how that led you to start the company.

   2. SEStefano Ermon2:59

      Yeah. So I was working on, uh, generative models of images, um, initially working on autoregressive models, which were very slow and kind of like very blurry. And then VAEs and then GANs took over.

   3. SGSarah Guo3:11

      Yes. [laughs]

   4. SEStefano Ermon3:11

      And, uh, back then, we were very unhappy with the state of, uh, yeah, generative models for images. Like, GANs were... They, they, they worked, but they were very unstable to train, very hard to reproduce results. And so we were trying to see, is there a way to build something that is as good as a GAN but it's more principled? And so we started working on s-score-based, uh, generative models, which are basically what eventually became, uh, diffusion models back in 2019 with, uh, with my PhD student, Yang Song. And so we kind of like came up with this idea of let's train a neural network to denoise images, and if you can denoise an image, then you, you really un- uh, are understanding enough about the structure of the image, uh, that it's... should be possible to build like a generative procedure based on these denoisers.

   5. SGSarah Guo3:57

      Mm-hmm.

   6. SEStefano Ermon3:57

      And that basically became the, the kind of like underlying technology of modern diffusion models, where instead of generating images, you know, left to right one pixel at a time, you kind of like start from pure noise and then you gradually refine the object until you get like a clean picture at the end. And that started, yeah, back in 2019 in my lab, and then it kind of like took over the space. And, and even today, the best models for image generation, video generation, music to some extent, a lot of the protein stuff, they are based on diffusion. And so my group has worked a lot on various kinds of diffusion models, technique for accelerating them to, to generate samples very quickly, to improve the quality of these models. And so, uh, since we were able to get them to work on images, I started thinking about, okay, how do we get diffusion models to work on text or code generation and discrete objects?

   7. SGSarah Guo4:48

      Mm-hmm.

   8. SEStefano Ermon4:48

      Like, is there a way to move beyond autoregressive models to, uh, something that it's more parallel, more with built-in error correction? And so I'd been doing a bunch of research at Stanford on getting diffusion models to work on text and code generation. Um, we had a breakthrough in 2024. We published a paper, um, basically showing that for the first time it was, it was possible to match the quality of an autoregressive model at the GPT-2 scale. So less than a billion parameters, still fairly academic, but we were able to train basically still a transformer model as a diffusion model on the same data. We were able to match the quality, like the same perplexity, and you were fitting the data just as well as an autoregressive model with the same number of parameters. But the diffusion model was significantly faster because it's diffusion, because you're outputting many tokens at the same time. We were able to generate text like 10X faster compared to the autoregressive model. And so that felt very, very exciting, and I really wanted to see what happens if you scale up, if you train bigger models. And so I started Inception, a company to basically scale up the technology and, and try to build, um, commercial scale diffusion-based

4. 5:59 – 11:10

   Why Diffusion Beats Autoregressive

   1. SEStefano Ermon5:59

      language models.

   2. SGSarah Guo6:00

      Everyone has now seen, um, the outputs of diffusion models in, in, in p- in particular images.

   3. SEStefano Ermon6:06

      Yeah.

   4. SGSarah Guo6:06

      And, uh, I, I would argue that it's, like, increasingly a dominant form of, like, generated short form video from diffusion models is, like, a dominant form of entertainment in other parts of the world and it will likely become so here. Um, it's kind of unbelievable, uh, at least to me, even having followed the field for, you know, the last decade plus, uh, the, the quality that is possible today. Um, so I, I think that is kind of obvious, right? And it's such a big use case in images and, and video generation that, um, folks are even creating, you know, hardware to support better, uh, um, better performance here. Um, it's not intuitive that would work for other fields or that, you know, this is a interesting competitive direction to, uh, the, um, current, you know, full transformer focus like AGI labs. Um, can you offer some intuition on that?

   5. SEStefano Ermon7:04

      Yeah. So it, it's a, it's a very interesting kind of like state of the world right now from a researcher perspective because, like, there is, like, two main paradigms, two ways of building generative models. There is autoregressive, where you, you kind of like have a model that predicts the next toke- the next token or the next pixel, and then you generate left to right one token at a time. And then there's diffusion, which is a course-defined generation, um, kind of like iterative denoising kind of generation.

   6. SGSarah Guo7:29

      Mm-hmm.

   7. SEStefano Ermon7:29

      And as you said, like, we have continuous modalities where diffusion dominate. There is discrete modalities, text and code, where primarily, like, all the big labs are kind of like betting on the same architecture, autoregressive models. And as we move towards more and more, like, multimodal models and kind of like there is this idea that maybe we'll have a model that can handle all modalities and will know everything about the world, what architecture will that be? Like, will it be an autoregressive model? Will it be a diffusion model? Uh, nobody knows. I think that, I think the jury's still out there. Uh, at Inception, we're betting on diffusion models because, uh, we believe that, uh, what matters eventually will be inference time scaling. And there are fundamental reasons for why diffusion models are better than autoregressive models at inference time. So even if you think about the story of autoregressive models, there was an inflection point in 2017 when people switched from RNNs to transformers, right? And, and why... What was that? The, the, the problem was that RNNs had to essentially process tokens sequentially, one at a time, and training was very slow.

   8. SGSarah Guo8:36

      Mm.

   9. SEStefano Ermon8:36

      And so people came up with this idea of let's have an architecture that allows you to process many tokens at the same time in parallel, and that was the transformer.

   10. SGSarah Guo8:44

       Mm.

   11. SEStefano Ermon8:44

       And that, that was the thing that scaled better for training and that en-enabled a lot of the successes behind LLMs. But if you think about inference, now not training, inference generation, uh, autoregressive models are still sequential. Uh, the computation is one left to right, one token at a time. You cannot generate the 10th token until you've generated everything that comes before it. That kind of workload is, um... does not map well to GPUs. That kind of workload is extremely memory-bound. Uh, you're spending most of your time moving around weights across the memory hierarchy, and you're doing very little arithmetic, and that's a fundamental problem of autoregressive models. A-And so what's the equivalent if you think RNNs, transformers, autoregressive models? The equivalent at inference time is a diffusion base, is a diffusion model. 'Cause a diffusion model is built to have, at inference time, a workload where you process many tokens-

   12. SGSarah Guo9:41

       You parallelize, yeah

   13. SEStefano Ermon9:41

       ... at the same time. And so the, the, the, the workload that we have at inference time in a diffusion model, it's basically very, very similar to the workload you have for training where we are processing many tokens at the same time in parallel.

   14. SGSarah Guo9:55

       Mm.

   15. SEStefano Ermon9:55

       And so it's built to essentially have an inference workload that maps really, really well to MAC moles, but the kind, the kind of things GPUs do really, really well. And so we bet on trying to build the architecture and trying to build the kind of models that will scale best at inference time because, um, you know, economics are dominated by, you know, the, the kind of intelligence per watt, the intelligence per dollar that you're able to get from the models. If you think about a lot of the advances with reasoning models, a lot of it is scaling test time compute, right? And so being able to scale better along that axis will also matter. And even if you think about RL post-training, a lot of the bottleneck is generating rollouts.

   16. SGSarah Guo10:40

       Mm-hmm.

   17. SEStefano Ermon10:40

       Like, letting the model explore, you know, and, and then scoring the trajectories and then improving the model based on the kind of things it finds. And so inference is, again, a key bottleneck for RL, RL post-training. And so if you have a model that scales better at inference time, then automatically you're gonna get better scaling during RL post-training.

   18. SGSarah Guo11:00

       Mm.

   19. SEStefano Ermon11:00

       And so that's why we decided to bet on a diffusion-based LLM because it's inherently more parallel, and the bitter lesson is that the more parallel solution is the one that is eventually

5. 11:10 – 13:19

   Discrete vs. Continuous Modalities

   1. SEStefano Ermon11:10

      going to win.

   2. SGSarah Guo11:11

      How did you think about, um, applicability or what experiments did you run in terms of cracking the nut on discrete versus continuous modalities? Because I, I think people have also, uh, shaped the existing, you know, dominant paradigm through new tokenization, uh, efforts or, or, uh, methods to make video and voice work, for example. Um, uh, it's, y- you know, this is... You're, you're not in the same token-oriented, um, paradigm. How do you make it work here?

   3. SEStefano Ermon11:41

      Yeah. So there was a lot of research that, uh, that, uh, went into figuring out how to apply a technology that was inherently very tied to kind of, like, continuous structure in the data.

   4. SGSarah Guo11:53

      Mm-hmm.

   5. SEStefano Ermon11:53

      So if you think about a diffusion model, it's learning how to denoise images, and it kind of, like, makes sense for continuous data because if you think about even two pixel colors, you can kind of like interpolate between them and it will still make sense. But if you think about two words, there is not necessarily something in between them, right? It's all discrete. And so it required, uh, a lot of R&D and, and new science that had to be developed to figure out how to, how to extend those kind of ideas to discrete- ... spaces

   6. SGSarah Guo12:21

      What can you claim about how well it works today?

   7. SEStefano Ermon12:23

      We think it works really well. Uh, so we've been able to, to train, uh, diffusion-based LLMs, uh, that are comparable in quality with the, the speed optimized models for frontier labs. So our Mercury models are on par with the Haiku models, Flash models, Mini, Nano models from OpenAI, if you look at benchmarks, uh, while being significantly faster. So we've, uh, crossed, I think, the, the, the-- We went from, you know, pure research prototypes to things that are actually used. The, like, we are serving these models in production today. We did all the work of figuring out how to, how even just, like, build a serving engine, right? You cannot run these diffusion-based LLMs on the LLM or SGLang. Like, you have- we had to build our own serving engine, and we can handle a lot of the complexity of, like, real production workloads. Um, and, uh, we, we've, we've solved all these challenges, and we can deliver this kind of, like, new experience end to end

6. 13:19 – 16:45

   Inception Today

   1. SEStefano Ermon13:19

      to, to real customers today.

   2. SGSarah Guo13:20

      Actually, a great time to just talk about where Inception is as a company. Like, how many people? What are you guys actually serving? Um, sort of state of research. Yeah.

   3. SEStefano Ermon13:30

      Yeah. So it-- we are about two years old, uh, around 50 people, uh, spending a lot of time still on R&D, kind of like figuring out what's the right way to train these models, um, how to accelerate the inference. Like, it-it's not obvious how you... When you think about an autoregressive model, it's pretty clear. There is not a lot of things you can do there in terms of, like, okay, you generate one token at a time, and that's it. In a diffusion-based model, we know that there is a lot of different possibilities for trading compute for quality at inference time. Like, even if you think about image diffusion models or video diffusion models, there is a lot of techniques that you can use to kind of like accelerate sampling, uh, distillation or, like, fancy, uh, differential equation-solving techniques that allow you to sample very, very quickly from these models. And so there is a lot of research on the training, on the inference, and then engineering, like, just, like, thinking about data mixes, evals, um, RL post-training infrastructure. Like, there is a lot of work that, uh, that needs to happen to figure out how to, how to build recipes that work for this new model. And we try to leverage existing things as much as possible. For example, it's still a transformer-based model, so you don't have to throw away, uh, a lot of the work that has been done on, on good architectures. Uh, we still use Attention. Uh, we still use, uh, a lot of the public data sets that people have created and, and evals and benchmarks. So, you know, we're a start- we're a startup. We try to be scrappy. We try to use existing things as much as possible and, and kind of like focus on the, on the, the aspects where we can have the highest impact and then where we can be the most differentiated. And right now it's speed. In the future, who knows? Like, it's possible that a diffusion-based language model will be maybe significantly more intelligent than a, than an autoregressive one. Like, nobody knows. That, that's why I think this is very exciting because we're developing these really powerful AI systems, but it's all very fresh. It's all very new. Uh, I doubt we've discovered the best way of building these systems. There's gotta be alternative. There's gotta be other ways of building these models and, and eventually, yeah, efficiency will be very important. Like, if you think about the AI factory, like, uh, how is that gonna work? I think nobody really knows, and just being able to play in that space and thinking about alternative ways of creating intelligence, I think is exciting.

   4. SGSarah Guo15:49

      Absolutely. And I, I also think that in a, uh, increasingly, like, fundamentally compute and supply constrained environment, uh, the... You know, for, for some... I, I wouldn't say that there wasn't very focused research on, um, efficiency of models, but it was a second order, uh, uh, sort of consideration for m-many of the industrial research efforts versus, like, pure capability scaling on what we've got, right? Um, and, and new methods in that. But, uh, I, I just think if you, if you fundamentally believe that we can use all of the power we have and there will be economic reasons to do so, then I think the focus on efficiency is gonna go dramatically up, right?

   5. SEStefano Ermon16:32

      Yeah.

   6. SGSarah Guo16:32

      Um, you know, I, I, I think many people now argue, like, one of the most important decisions for, um, a company in the AI space is, like, how do we use the power we have between, you know, training and the most valuable use cases for any,

7. 16:45 – 17:31

   Where Speed Wins

   1. SGSarah Guo16:45

      any watt. On that topic, like, here in, you know, uh, September of '26, uh, where does speed win? Like, where do people care enough about this already?

   2. SEStefano Ermon16:55

      Yeah. It is basically applications where latency matters. I mean, generally speaking, I think everyone cares about speed, uh, in the sense that if you can give me the same, uh, quality but faster, people will always pick the, the faster solution. And, and we're seeing it with this, like, faster versions of even the, the models from Frontier Labs. People are willing to pay more to get access to, to faster models, right? And, and I think once you get used to a fast model, it's hard to go back. It's kind of like broadband, right? And then-

   3. SGSarah Guo17:25

      [chuckles]

   4. SEStefano Ermon17:26

      ... it gets faster and faster, and then if you, if you were able to try... You know, people cannot go back once you, once

8. 17:31 – 18:49

   Inception Customer Base

   1. SEStefano Ermon17:31

      you get the fast model.

   2. SGSarah Guo17:32

      Uh, are there customers that you can talk about publicly that, um, you know, care about this today?

   3. SEStefano Ermon17:37

      There are a few that, that we can mention. Uh, like in the voice space, for example, um, OpenCall i-is an example. Um, they, they are building like voice agents. Um, they... You know, speed, of course. The, the pipeline is like you have an ASR model, you have an LLM that it's kind of like doing all the tool calls and figuring out what to say next. Uh, it has to be a reasoning LLM typically to, to have the highest level of quality.

   4. SGSarah Guo18:00

      Mm-hmm.

   5. SEStefano Ermon18:00

      And then there is a text-to-speech, uh, component at the end. Um, speed matters a lot to them. They were previously using, uh, uh, serving their LLMs on, on, on Cerebras, so they, they were using custom chips to get to the kind of speed that they need to, to deliver the best experience to their customers. And then they switched over to, to our diffusion-based LLMs because they can essentially get the same speed as what you would get if you were to run an autoregressive model on custom hardware If you have a diffusion-based LLM that it's built to be parallel, it's accelerating at the software level, then you can get the same speed on NVIDIA GPUs, uh, which means much more, um, availability. I mean, GPUs are scarce, but there's more of that than, than, [chuckles] than custom chips, uh, and lower cost, higher quality. So that's

9. 18:49 – 19:34

   Interaction with Hardware Landscape

   1. SEStefano Ermon18:49

      an example in the, in the voice space.

   2. SGSarah Guo18:51

      Yeah, I was actually gonna ask you how you think this, um, interacts with the, uh, hardware landscape as well, given we've seen enough now demand from use cases who are like, "Yes, I want a big, expensive chip with a lot of SRAM," and people will pay for the outputs of that.

   3. SEStefano Ermon19:07

      Yeah.

   4. SGSarah Guo19:07

      In, in coding and other use cases.

   5. SEStefano Ermon19:09

      Yeah, for sure. For sure. And I- I think hardware is one way to, to, to, to accelerate things, and the good thing about-

   6. SGSarah Guo19:14

      And software might be better-

   7. SEStefano Ermon19:15

      Soft-

   8. SGSarah Guo19:15

      ... if we can use the existing hardware.

   9. SEStefano Ermon19:17

      Exactly.

   10. SGSarah Guo19:17

       Yeah.

   11. SEStefano Ermon19:17

       And especially they are complementary. That's the exciting piece, is that to some extent, uh, the gains that you get from the software, they are multiplicative with the gains you get from the hardware.

   12. SGSarah Guo19:27

       And maybe someday people develop a system, uh, you know, hardware that fits even better the models that you're building.

   13. SEStefano Ermon19:33

       For sure.

10. 19:34 – 21:41

    Inception and the Broader Industry

    1. SEStefano Ermon19:34

       For sure. Yeah, yeah.

    2. SGSarah Guo19:35

       If we just like zoom out to, um, the, you know, inception in the broader industry, uh, I, I think there is a vein of concern, and correct me if I'm wrong, uh, that, uh, it's very hard to invest in new architectures today because some... Y- you know, if there are advances in architecture or methods, it will simply be absorbed by, uh, large players with the resources to scale compute. Um, talk to me h- about how you think about going and competing as David in this situation.

    3. SEStefano Ermon20:06

       Yeah. It's, it's a very valid point and something that is also like top of mind for us. Um, I think initially for us, for sure, like the moat is sort of like the IP, the trade secrets, like the, the ideas that we and our researchers have to, to, to build these models and make them better. Um, as we mature as a company and, and one of the reasons we are not just doing pure research, we're also like, you know, we've developed a product and we have real customers, and we are getting feedback on, on the models from, from the real world, is that by doing that, we are also like developing, uh, components that are also very important to, to deploy these models. For example, like a serving engine. Uh, you know, like if you don't have the serving engine, you can't really serve these models in production. And so by forcing ourselves from the very beginning to go out and, and deploy something end to end, we're learning a lot about how to serve these models and how to build, uh, software that it's kind of like needed to, to, to run these models and, and that again becomes IP. Like even if you train the diffusion-based LLM, if you don't have the serving engine, if you don't have the vLLM equivalent to serve it, you're still stuck and you still cannot use it. On the same li- like along the same lines, we are working with real customers, and we are getting feedback on the models. We figure out what works, what doesn't. We collect data sometimes from them. We create evals based on what they're seeing. And so that again becomes part of the, of the technical moat because, uh, you know, of course those things are a little bit harder

11. 21:41 – 24:45

    Data Compression and Structure

    1. SEStefano Ermon21:41

       to, to replicate.

    2. SGSarah Guo21:42

       You can tell me if this doesn't make sense as a, a question to ask, um, uh, technically, but one of the things that diffusion models benefit from structurally in images or video generation is, you know, you're replicating something where there should be some consistent structure in the world. Uh, voice as well, right? It is whatever is really possible and most likely. Um, there are, like some of the fields where, uh, AI has been most valuable to date, uh, I, I'd say like, you know, a lot of the input data you use to train like code data, for example, um, uh, it's, it's very messy, right? And you know, one could argue that a lot of it doesn't actually have the like correct real structure you're looking for. How do you think about that when it's like h- human-generated input data versus, you know, images, video, voice?

    3. SEStefano Ermon22:33

       Yeah. It, it's a good question. And fundamentally, if you think about whenever you train a generative model, what you're doing is, uh, whether it's an autoregressive model or to some extent even a diffusion model, is you are trying to, uh, identify structure in the data to... by essentially building a compression scheme.

    4. SGSarah Guo22:50

       Mm-hmm.

    5. SEStefano Ermon22:50

       That might not be obvious, but whenever you train these models, you're effectively trying to identify common structure by trying to find an efficient way of compressing the data. And so the more you can compress the data, the more structure, the more patterns you're identifying, and that's how these models work. Which is the, the, the amazing thing is just like by predicting the next word, uh, you are learning something about the structure of the data.

    6. SGSarah Guo23:14

       Mm-hmm.

    7. SEStefano Ermon23:14

       And that's the same whether you're using a diffusion model or you're using an autoregressive model. Both methods are essentially trying to learn a compression scheme.

    8. SGSarah Guo23:23

       Mm-hmm.

    9. SEStefano Ermon23:23

       And when I mentioned the, the original 2024 paper when we showed that we are achieving parity with autoregressive models, the metric that we're using is basically perplexity, which is a notion of how much structure have you identified in the data.

    10. SGSarah Guo23:39

        Mm-hmm.

    11. SEStefano Ermon23:40

        And so even though it might not seem obvious, uh, we were actually able to identify at the GPT-2 scale-

    12. SGSarah Guo23:46

        Mm-hmm

    13. SEStefano Ermon23:47

        ... the same amount of structure as an autoregressive model.

    14. SGSarah Guo23:50

        Yes. I, I think like that, that empirical result is there, but the intuition would be like, well, this, you know, code is not, uh, the, the data set that you are working on is not like grounded in physics, right? There's a lot of noise in there, and it sounds like that is... You believe that's a manageable problem.

    15. SEStefano Ermon24:05

        Yeah. I mean, there is noise in, uh, in, in-

    16. SGSarah Guo24:07

        Everything

    17. SEStefano Ermon24:07

        ... pattern everything.

    18. SGSarah Guo24:08

        Yeah.

    19. SEStefano Ermon24:08

        And so to the extent, you know, the numbers don't lie. To the extent that you, you are able to drive the perplexity down, then it means that you can actually build like a, a compression scheme, uh, and that, that will get you that sort of like level of compression. And so the, the structure m- must be there, and the model must have been able to uncover it. And then it's more a question of an inductive bias. Like w- you know, is a, is a transformer a better way of identifying those patterns or something else? Uh, is next token prediction the right- ... modeling framework or is it more like denoising? And that's, that's very much an empirical question that I think at the moment we don't

12. 24:45 – 27:25

    Controllability of Diffusion Modeles

    1. SEStefano Ermon24:45

       have a tools even to understand.

    2. SGSarah Guo24:47

       Can I ask a question just because you used, um, a voice customer as the example here? Uh, one of the, uh, benefits that some people building these AI products have identified of having an LLM in the middle of this voice pipeline is they understand how to, um, do alignment a little bit better there or controllability. I imagine that has to look different for a diffusion-based model. Can you talk about that?

    3. SEStefano Ermon25:12

       Yes. That's the, that's a key value proposition, and one of the things that they, they always look into is, you know, yeah, to what extent, uh, a lot of the value they provide is like the harness and making sure that the models indeed are doing the right thing. And the interesting thing about a dif- a diffusion-based LLM is that we've built everything to be backwards compatible, so it's still like the API is the same. It's still OpenAI compatible, text in, text out. And it so happens that the models we've trained are good at following instructions. They're good at outputting, you know, if you're using JSONs, like structure outputs, they can handle all of those things.

    4. SGSarah Guo25:46

       Mm-hmm.

    5. SEStefano Ermon25:46

       And, uh, it was good enough. It was better, in fact, than the models they were using before. And so they are still able to provide the, the kind of like level of service to their customers by using Mercury.

    6. SGSarah Guo25:57

       Well, very simple if the interfaces are the same and you can just use your same stack, yeah.

    7. SEStefano Ermon26:01

       But it could be that, that I think that's actually a very interesting point is that, uh, we know that diffusion models are typically easier to control compared to autoregressive models. And the reason is that if you think about an autoregressive model, you kind of like have to wait until you've generated the full object to know whether or not it satisfies, let's say, a constraint or whether or not it's aligned or whether or not what- whatever, you know, it's on brand. Whatever it is that the, the objective function that you care about. Maybe you're generating a molecule and you care about solubility, and then, and then you kind of like have to wait until you have the full object to be able to score it with some reward function. But a diffusion model, it's more course-defined generation. So from the very-

    8. SGSarah Guo26:40

       You can progressively do it, yeah

    9. SEStefano Ermon26:41

       ... from the very beginning, you know kind of like is this object the kind of thing I want or not, and you can steer the generation in the direction provided by an external reward function or some set of constraints. And so there is a lot of evidence in the academic literature at least, that diffusion models are easier to control, and there are different ways of steering them that are just not possible with autoregressive models.

    10. SGSarah Guo27:05

        Mm-hmm.

    11. SEStefano Ermon27:05

        So that would be a different interface-

    12. SGSarah Guo27:08

        Mm-hmm

    13. SEStefano Ermon27:08

        ... for the model that maybe might not be even available for autoregressive models. I think that's something that, uh, we've been thinking a lot about, like what would be the right, what's the right product experience that we can build-

    14. SGSarah Guo27:20

        Mm-hmm

    15. SEStefano Ermon27:20

        ... around the new capabilities that are just not provided

13. 27:25 – 29:02

    Emergent Capabilities at Scale

    1. SEStefano Ermon27:25

       by autoregressive models.

    2. SGSarah Guo27:26

       Are there capabilities that you imagine, um, Incep- Inception's models, uh, having at scale that today's models don't have beyond performance?

    3. SEStefano Ermon27:36

       Yeah, that's the thing. We don't, uh, we don't know, right?

    4. SGSarah Guo27:38

       Yeah. [laughs]

    5. SEStefano Ermon27:38

       And that, that's the exciting-

    6. SGSarah Guo27:39

       It's emergent, yeah.

    7. SEStefano Ermon27:40

       It's emergent, yes.

    8. SGSarah Guo27:40

       Yeah.

    9. SEStefano Ermon27:40

       Yes. Like right now, the wedge is speed. Uh, we know they are much faster. That was the initial bet because that was easier to test. It's also like easy to, you know, to, to, to measure and, uh, it- it's obviously valuable, right? But as we-- That- that's why I find it so exciting is that as we learn more about how to train these models, you know, we don't know what we're gonna find. And, uh, there is a decent amount of evidence in the academic literature, for example, that diffusion-based models are more data efficient compared to autoregressive models.

    10. SGSarah Guo28:15

        Mm-hmm.

    11. SEStefano Ermon28:15

        And the intuition is just like if you think about training a diffusion model, um, you're learning by denoising. You start with an image, you add noise, and then you learn how to remove the noise. And so it's effectively doing data augmentation in the sense that the same image is augmented by many noisy views. [chuckles]

    12. SGSarah Guo28:33

        Okay. Yeah.

    13. SEStefano Ermon28:34

        Uh, and so they tend to be a little bit more data efficient. And so if that's, you know, holds up at scale, and then you believe, then maybe we'll get into-

    14. SGSarah Guo28:45

        Tasks where we have s- less data.

    15. SEStefano Ermon28:47

        Yeah, where data becomes the bottleneck, then it becomes more interesting, right?

    16. SGSarah Guo28:51

        Mm-hmm.

    17. SEStefano Ermon28:52

        And so we'll, we'll see. That's why it's, it's so exciting because things are changing and, and, and then, uh, you know, this technology is so important and so valuable that having something differentiated

14. 29:02 – 30:03

    Future Workload Split Between Diffusion vs. Traditional

    1. SEStefano Ermon29:02

       I think will create value.

    2. SGSarah Guo29:04

       If we, uh, project out, you know, five years, um, uh, that's actually way too long in AI world.

    3. SEStefano Ermon29:10

       [laughs]

    4. SGSarah Guo29:10

       If we project out two years, like, uh, what do you think is the workload split between diffusion and, uh, traditional, um, models?

    5. SEStefano Ermon29:20

       I think we're still, uh, on-- We're not at the frontier level of, of intelligence, and I think a lot of the workloads do require frontier level intelligence. But on my estimate, so like even if you just go to, you know, Open Router has this very nice way of kind of like looking at all the different use cases, and you can kind of see, you know, all the research and conversational and coding and software engineering and log processing log. Like they have like a nice hard taxonomy basically of tasks.

    6. SGSarah Guo29:45

       Mm-hmm.

    7. SEStefano Ermon29:45

       And, uh, I was doing some estimates, and I think there is like between 20 and 30% where latency is, is really, really important. And so at the very least, as a lower bound, I think it could be-

    8. SGSarah Guo29:57

       Addressable, yeah

    9. SEStefano Ermon29:58

       ... a- addressable by models that are s- within a given latency

15. 30:03 – 31:44

    Adoption Challenges

    1. SEStefano Ermon30:03

       budget, they would give you the highest possible quality.

    2. SGSarah Guo30:05

       And then just, you know, all technology approaches have trade-offs. What are, what are the, uh, challenges of working with diffusion models?

    3. SEStefano Ermon30:12

       Yeah, it's, it's a different stack. And so one of the challenges is that we have to build a, a lot of things [chuckles] in-house, and, uh, there is not a mature sort of like ecosystem of, uh, uh, like even if you think about the serving engine or like kernels, like, uh, a lot of the things had to be developed in-house.

    4. SGSarah Guo30:32

       Mm-hmm.

    5. SEStefano Ermon30:32

       And so there is not really anything open source or, or there are some open source models, but they are not particularly good. And so that makes it a little bit more difficult to, you know, deploy, to get customers to try things. They, they're not used to it. So that's, that's been, uh, one of the challenges.

    6. SGSarah Guo30:50

       I imagine that also reflects externally, right? Y- you know, in a landscape where folks, uh, at some sophistication where they would care about cost and performance, and though you might, uh, for, for certain use cases, you will care about cost and performance from, from the beginning. Uh, there's an increasing amount of interest in post-training, right? And so I imagine in a, uh, new architecture, that's even more challenging.

    7. SEStefano Ermon31:14

       So we had to build our own stack for doing, uh, SFT, for doing RLHF, doing RL. Uh, I mean, that becomes IP to some extent, so it, it, it's, it's one of the reasons we decided not to open source everything was really to, to keep the IP a little bit closer, uh, to us and not opening it. But then there are downsides, like there is less, less an opportunity for the community to contribute. It's harder to, to adopt. Uh, it's hard to do on-prem, kind of like deployments. And so

16. 31:44 – 32:50

    Hiring and Team Organization

    1. SEStefano Ermon31:44

       there are pros and cons with the, the two choices.

    2. SGSarah Guo31:46

       Can you talk about, you know, scale of your own training, uh, and then, um, like current or aspirational, uh, and then, uh, at 50 people, I'm sure you're continuing to hire, like why researchers or engineers or others should consider, um, you know, investing in this direction or working at Inception?

    3. SEStefano Ermon32:03

       Yeah. So we're not able to share much about the, the, the, the training, uh, the, the size of the models or the flops or all of that. It's kind of like a trade secret, but we are continuing to push the frontier, and, uh, Inception is a great place to be if you want to be... Have an opportunity to shape the, the direction of the field. Like it's still a relatively small field. There is a lot to be invented, and so a lot of the people that decided to, to come to Inception instead of joining one of the other labs is really that they want to have ownership and, uh, they, they, they like to invent new things. They like to... They, they like to be in a space where there's more of a green field and more opportunities to, um, try things. There's less that it's known or available out there. It's a little bit more open-ended, and so

17. 32:50 – 34:02

    Recursive Self Improvement

    1. SEStefano Ermon32:50

       we, we tend to attract those kind of people.

    2. SGSarah Guo32:52

       I think one thing that is both exciting and causes some despair amongst research friends is, um, uh, you know, the ability to use models for recursive self-improvement in the research field itself. What- Given you're working on a, like a very different direction, what is your view on this?

    3. SEStefano Ermon33:09

       Yeah, I mean, it's, uh, it's something that we-

    4. SGSarah Guo33:11

       It l- I mean, it, explicitly, it sounds like you still feel there's work for you and your team to do.

    5. SEStefano Ermon33:16

       Oh, yeah. Yeah. I think we're not there yet. Uh, maybe we don't have access to the models that other folks have, but I feel like there is still, uh, uh... I mean, we use models a lot, of course, and it has accelerated the, the, the speed at which we can iterate, try ideas, and, um, you know, we use models from Frontier Labs and, and, uh, uh, it's been, it's been great. It has accelerated our development process a lot. Um, at the same time, I think, uh, at least right now, I don't know what's, what it's gonna be in six months or a year, but right now, uh, the, the, the human ingenuity is still like super important, and the ability to come up with the, the right ideas and kind of like prune the space and, and kind of like identify directions that are more promising

18. 34:02 – 35:10

    Resource Allocation

    1. SEStefano Ermon34:02

       has been really important to us.

    2. SGSarah Guo34:05

       50 people is not that many people for, let's say like a full stack, you know, uh, research serving product company, um, or, or however you would think about describing it. Uh, how... You know, how do you organize, and then how do you think about how you allocate your resources here?

    3. SEStefano Ermon34:22

       It's a small team, but everyone is very talented and, and, uh, they, they work very hard and, and we have access to agents that are making us a lot more productive. And so I think the numbers are, uh, you know, are sufficient to do, to do a lot. And, and in fact, often the... I feel like the, the, the bottleneck is more compute than, than, than people. Uh, but yeah, the, the team is organized. Like there is a, there is a, a product team effectively that is handling the platform and working with customers to, to, to make them successful with our models. And then, uh, so there is basically a team that it's serving the, the current best version of the model, and then there is a team that it's building the next version of the model, and that includes training RL

19. 35:10 – 38:13

    Impact of Academia

    1. SEStefano Ermon35:10

       inference. And so that's more research.

    2. SGSarah Guo35:13

       Stefano, one last question for you. Uh, you know, the '24 paper was a super interesting result, made a big splash. You've been working in this field for a long time. A lot of folks would say that, uh, would claim that, you know, academic AI research is very challenged in this era of, you know, being able to scale resources a great deal. Like this is certainly true to some degree, given you, you started a commercial company around it as well. But w- how did you get confidence in the directions that you were working in having impact or being promising before you really had tho- those '24 results?

    3. SEStefano Ermon35:46

       Yeah. And, and I think it was like a s- s- um, a collection of results that, uh, that I, I had been, that I've been working on in, in my lab. Not necessarily like... Of course, there is like the early diffusion work that we did in the lab. We showed that the, at the kind of scale of models that we could train on, uh, academically, we were able to kind of like beat GANs, uh, and so, and then, well, being much more stable, and then the whole thing took over and then became stable diffusion, Midjourney. So all, all of that started from ideas that were developed in academia in my lab, but that's not the only one. Like I was Trida's co-advisor, so I worked on flash attention, for example, right? That's another thing that came out from academia that then eventually had a huge impact in industry, right? Or another example is DPO. That was another project that started out as a rotation, a project in my group. Uh, it, it's an algorithm that is used to align, you know, LLMs and diffusion models and everywhere, right? And, and that's again, something that, that was developed entirely in academia, and it was just like based on a, on a clever insight, like some, some, some interesting mathematical structure that you have in that problem that allows you to come up with a very different and more efficient way of post-training and aligning these models, right? And so, uh, th- there are gems, uh, there are, uh, lots of opportunities for finding, uh, new and better ways of solving important problems. Uh, one of the nice things about academia is that it allows you to take these contrarian bets. Um, as you said, I mean, there is the challenge that maybe we don't have enough resources, and there is never enough resources, and if we had more compute, we could be more efficient. Uh, but the... You know, you have access to amazing students, and everyone is kind of like trying to develop the new thing. People are not scared about taking bets, and, uh, that's, uh, that's why academia I- has been so impactful, I think, where even in the AI space, a lot of the important ideas have roots or even were created in academia.

    4. SGSarah Guo37:49

       Awesome. Super inspirational. Thanks so much for being here, Stefano.

    5. SEStefano Ermon37:52

       Thanks so much for having me.

    6. SGSarah Guo37:52

       [upbeat music] Find us on Twitter at @NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way, you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.

Episode duration: 38:13