Text to Video: The Next Leap in AI Generation

1. 0:00 – 2:41

   Text to Video: The Next Leap in AI Generation

   1. AMAnjney Midha0:00

      This is a conversation I've been super excited about

   2. RRRobin Rombach0:02

      What is Stable Diffusion? You type in a text prompt and it generates an image

   3. AMAnjney Midha0:06

      Why was stable video the thing that you guys decided to prioritize?

   4. ABAndreas Blattmann0:09

      I think we like challenges

   5. RRRobin Rombach0:11

      I think compute constraints can drive innovation

   6. ABAndreas Blattmann0:13

      You, you always wanna try to improve your efficiency

   7. AMAnjney Midha0:17

      It's, it's already such an incredible leap forward, right?

   8. ABAndreas Blattmann0:20

      We, we're really excited to see what, what will happen

   9. SPSpeaker0:23

      [upbeat music] Today, many people are familiar with text to text or text to image AI models. Just think, ChatGPT or Midjourney. But what about text to video? Well, several companies are working hard to make that a reality. But for many reasons, it's a lot harder. For one, think about size. You'll often find text files in the kilobytes, images maybe in the megabytes, but it's not uncommon to find high quality video content in the gigabytes. Plus, video requires a much more dynamic representation of the world, incorporating the physics of movement, 3D objects, and more. I mean, imagine the hand challenge in text to image, but in this case, it's hands squared. But this is not stopping the researchers behind Stable Video Diffusion, which as of November 21st, was released as a state-of-the-art open source generative video model. So today, you'll get to hear from two of those very researchers, Andreas Blattmann and Robin Rombach. Robin, by the way, is also the co-inventor of Stable Diffusion, one of the most popular open source text to image models. So in today's episode, together with a16z General Partner Anjney Midha, you'll get to hear firsthand what really makes text to video so much harder, the challenges like selecting the right datasets that enable realistic representations of the world, applications where this technology is already being deployed and put to use. Plus, what the video editor of the future might look like, and how constraints continue to spur innovation and ultimately keep this field moving. Finally, if you like this episode, our infrastructure team at a16z is coming out with a lot more AI content in the new year, but in the meantime, you can go to a16z.com/ai for all our previous coverage. All right, enjoy. As a reminder, content here is for informational purposes only. It should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any a16z fund. Please note that a16z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com/disclosures.

2. 2:41 – 4:35

   The Stable Diffusion backstory

   1. AMAnjney Midha2:41

      This is a conversation I've been super excited about for a while, and thank you so much for finding the time to, to join us on the podcast. Um, maybe we can start with just a, you know, for, for first time folks, um, a brief overview of, uh, your team, your research lab. Um, and for listeners who are unfamiliar, maybe, you know, spend just a, a couple minutes talking about what Stable Diffusion is and what Stable Video Diffusion is.

   2. RRRobin Rombach3:07

      What is Stable Diffusion? Stable Diffusion is a text to image model, a generative model. That means you type in a text prompt and it generates an image based on that. In particular, Stable Diffusion is, as the name suggests, a diffusion model. Diffusion models are a type of generative models which has been super successful recently for image generation, and it's based on a technique that we developed, uh, while we were still at the university. So me and Andreas and Patrick and Dominic, all in the same team now at Stability AI. We are a multimodal company, uh, and our specialty is to produce and publish models, try to make them as accessible as possible. That includes publishing weights and kind of making foundation models, uh, for all kinds of modalities, not only images, but also video available and enabling research on top of that. So we have seen that Stable Diffusion was super successful, I would say much more successful than we initially anticipated. And there are like hundreds if not thousands of papers that are building on top of that. We, in particular, our group is focused on visual media, so that is images, that is videos, and, uh, Stable Video Diffusion that you just introduced is kind of the next iteration. It's our first step into, into the video domain. We published a model that can take in an image and turn that into a short video

3. 4:35 – 7:17

   Diffusion vs autoregressive models

   1. RRRobin Rombach4:35

      clip.

   2. AMAnjney Midha4:35

      Maybe we could, we could spend a couple minutes on a brief overview of diffusion models. That might be helpful. You know, how, how do diffusion models differ from other types of generative models and techniques like autoregressive models? If you could just give us a little bit of context before we dive in, that, that would help.

   3. ABAndreas Blattmann4:51

      Diffusion models are really the to-go models right now for visual media like images and videos. They're kind of different to autoregressive models because they don't represent data as, as a sequence of, of tokens or something, which we know from autoregressive models. Um, and in-- since images and videos are like composed as a pixel grid, um, this is really a, a, like, a, a good beneficial property. Also, they favor perceptually important details, which is like inherently baked into these models because their learning objective is just like tuned to favor these like important aspects of images a- as we perceive it as humans. And yeah, that is what we actually want, right? But they also have a k- kind of like some commonalities with, um, autoregressive models. They're iterative in their nature. They, they apply some kind of iterative refinement to the data. But as opposed to autoregressive models which iteratively generate token by token and, or word by word for language, these models gradually transform in small steps. So they gradually transform noise to data.

   4. RRRobin Rombach6:05

      Maybe one point to add to, um- Kind of the difference maybe to like the other very successful type, which is in language models or autoregressive models in general, you have to generate each token in a sequence, right? So you train your model, um, on a certain sequence length, and then you iterate over that in, while, while decoding a sentence. And in diffusion models, you also ... You, you train the model on... Like initially you usually use like 1,000 different noise levels between like data and like pure noise. Um, there exists different formulations. You can do this in a continuous space, but it's not too important. The interesting thing is that at sampling time you can actually use less steps. You can use like 50 steps. You can use... There exists like specialized, um, samplers for these models which use far lesser, uh, steps than recently. There's, uh, has been a lot of distillation work actually, which still reduces this further. We, we ourselves, we have published a distillation work, um, a week ago that actually shows that you can go, uh, as low as one sampling step.

   5. AMAnjney Midha7:13

      Mm-hmm.

   6. RRRobin Rombach7:13

      Which is, I would say, like a big, um, advantage of these diffusion

4. 7:17 – 10:55

   The benefits of single step sampling

   1. RRRobin Rombach7:17

      models.

   2. AMAnjney Midha7:17

      For folks who may not be familiar with why a single step sampling, uh, breakthrough is important, could you say a little bit about, you know, what benefits that leads to for creators or, or users of the model?

   3. RRRobin Rombach7:28

      Oh yeah, absolutely. I think like the most intuitive thing is that, um, you actually see what happens, uh, while you type in your text prompt. So think of like this text image, uh, model. Uh, you type in your prompt. Uh, one and a half years back you had, you had to wait for like a few seconds, maybe even up to a minute. Now you, now you see what happens and the quality is even better than what we had like with the first iteration of Stable Diffusion. So super, super exciting actually to see that kind of trajectory, these kind of developments. Um, like when I first sampled this model, I was actually shocked that it worked so well.

   4. AMAnjney Midha8:05

      To, to keep pulling on that thread for a bit, you know, if we, if we rewind, uh, the clock back to a year and a half ago, which is when you guys first put out Stable Diffusion. Um, between then and now, what has surprised you most about image models that you didn't expect?

   5. ABAndreas Blattmann8:26

      The pure im- improvements in performance, in text understanding of these models, in spatial compositionality of what these models can do. Just by typing in a single prompt you can describe a, a scene in really, really fine-grained, and it gives you a highly detailed like instantiation, visual instantiation of it. The developments has been huge. We, we published SDXL in June, and even then it was like, like a huge improvement in, in visual quality, in prompt following. Also other models which we see right now as, as like most recent, like DALL-E 3 is like a huge improvement still. But also like as Robin said, that there has been a lot of different samplers proposed to make these models faster and faster and faster, and right now we're getting really close to, to 50 steps performance in even one step. So this, this is like, this is a, a huge improvement. And I think a really important part of this is the fact that these models have been accessible to everyone.

   6. AMAnjney Midha9:26

      Mm.

   7. ABAndreas Blattmann9:26

      So open sourcing like a foundation model as Stable Diffusion initially, that led to a, a whole lot of research on these models, which was-

   8. AMAnjney Midha9:35

      Right

   9. ABAndreas Blattmann9:35

      ... yeah, it was in, in, in retrospect extremely important to, to do this. I think otherwise we wouldn't have seen the, the improvements we saw until now.

   10. RRRobin Rombach9:44

       Even before that like I was surprised that text-to-image with diffusion models worked so well, um, just before we published the model. Um, like when I first saw this myself, like we, we had like this latent diffusion approach, uh, that we developed at the university and then, um, I mean, we got a machine with like eight 80 gigabyte A100s, um, just after we put it on archive. And then, yeah, immediately started working on, hey, we want to have like this text-to-image model but not train it on one GPU. Let's use all of the... Like let's use our little cluster with eight 80 gigabyte A100s. Um, we trained this latent diffusion model on 256 by 256 pixels. Um, yeah, really had to sp- was the first time that we had to deal with like large scale data loading and these kind of things. Uh, but yeah, we made it work and it... And then using this model, combining it with, uh, classifier-free guidance, which is, uh, a sampling technique that further improves, uh, sample quality at basically no cost. I was like really surprised that we could do this on our own and achieve like pretty, a pretty good model I would say. And then like, uh, two days later, OpenAI published DALL-E 2 and all the hype was, was gone. But uh, it was a pretty nice

5. 10:55 – 13:10

   Why generative video?

   1. RRRobin Rombach10:55

      experience.

   2. AMAnjney Midha10:55

      You know, something you, you mentioned, Andreas, is that, um, the fact that you guys chose to release Stable Diffusion as an open source model resulted in this, this crazy ecosystem exploding around your research, right? Which is, um, which is just something that doesn't happen as quickly or as fast with models that aren't open source. And so in the last year and a half, one of the things that's been really fun, at least for me to watch, is all the really surprising things that, that developers and, and creators have done with the base model that you guys put out. You've, you've provided folks a set of Lego blocks that they can mix and match in different ways. Things like ControlNet that give people more controllability, allowing the community to build their own front end. Out of all of that, you know, I'm sure came a ton of requests for you guys. As you guys were prioritizing all those asks that came in from the world and the community, why was stable video the thing that you guys decided to prioritize above everything else as, as your sort of next major milestone?

   3. ABAndreas Blattmann12:01

      I think video is, is like an awesome kind of data because you, to, to solve that task, to solve video generation, a model needs to- Learn much about like physical properties of the world or like the physical foundations of the world. Th- there is, there is so much without knowing about, for instance, 3D scenes, you cannot generate a camera pan around an object or you cannot make an object move. If, if a person turns around, the model needs to hallucinate how it look, how this person looks from behind, right? So it's, it's like you, you, you need to, to know so much about the world by just like including that additional temporal dimension. Um, this is what fascinated me most on working on videos. Um, and yeah, I think it's, it's, it's also like computationally it's, it's really, it's, uh, it's ne- next level of, of computational demands because you have an additional dimensionality which is like ma- makes everything much, much, much, uh, harder I think. And yeah, I think we like challenges, that's why we probably focused on, on doing that.

6. 13:10 – 14:53

   Understanding physics through AI video

   1. AMAnjney Midha13:10

      Yeah. Something that's not known about you guys is, um, by background originally, I, I believe you're physicists.

   2. RRRobin Rombach13:16

      I'm, I'm a physicist, uh, but yeah.

   3. AMAnjney Midha13:18

      Um-

   4. RRRobin Rombach13:19

      I, I, I haven't done much physics in a while, unfortunately.

   5. ABAndreas Blattmann13:22

      Yeah. I'm, I'm originally a, a mechanical engineer, but that is also really related to physics and I was always inspired by physics and really fascinated by it.

   6. AMAnjney Midha13:32

      Well, both of your backgrounds academically, um, were spent studying the physical world and, uh, I, I just think it's poetic that, uh, you know, your primary interest in, in generative modeling came from, from trying to understand at some deeper level the physical world. And it seems like that seems to have motivated some of, at least some of the, the, the intuition and the research around your approach to stable video.

   7. RRRobin Rombach13:57

      Yeah, absolutely. Actually, I, I, I fully agree and I think like we're just like actually scratching the surface with, uh, the kind of video models that we have right now. Um, having something like we are seeing in language modeling, but trained on pixels, on videos will probably give like super interesting downstream behavior. Not, not only like generating videos, but also understanding of the world. Uh, like is it possible to derive something like a physical law from, from such a model? I don't know. Um, or such a model is alwa- also always, uh, predictive. So you can start with, uh, an image or with like a sequence of images and try to predict what happens next of course. Um, and then, and then I think like also coupling this with other modalities such as language, um, will maybe provide a way to ground like these models more in the physical world.

   8. AMAnjney Midha14:52

      Well, I,

7. 14:53 – 18:43

   The challenge of creating generative video

   1. AMAnjney Midha14:53

      I think that's a good segue into w- what is the main focus of, of today's conversation, right? Which is, uh, generative video. Um, at least to, to folks who are early users of stable video, of stable diffusion, um, stable video was a much awaited sort of natural progression from the original model. Um, just take us back a little bit to the original sort of conception of the project. How, how long have you guys been working on, on video modeling?

   2. RRRobin Rombach15:20

      I would say roughly half a year and like for this model that we just put out, I think the main challenge was that we actually, yeah, had to scale the dataset and the data loading. So if you train a video model on a lot of GPUs, you suddenly run into problems that you didn't really have, have before. Like, uh, loading high resolution videos, um, is just like a difficult task if you do it at scale.

   3. ABAndreas Blattmann15:46

      Also like only decoding videos is really hard. Like a data loader has to transform like the spikes it loads into a suitable representation for the model. And to do so you have to do a lot of computational, um, work from the video codec where the, the video was encoded to transform it, uh, into a suitable input sample for, um, the generative model. And this is computationally really expensive, um, and since we have so fast GPUs right now, it was really like the, the CPUs were just like in, in the beginning too slow. Building an efficient data pipeline for video was really a, a challenge.

   4. RRRobin Rombach16:27

      Yeah. And actually there's like interesting bugs that you can encounter during training. So we had one, um, where we would actually-- like if you train a diffusion model, you have your data and then you add noise to that data that the model tries-

   5. AMAnjney Midha16:41

      Right

   6. RRRobin Rombach16:41

      ... to remove, right? And if you do that on a video, you add noise to each frame of the video, and then we had like a, a bug where we added like different amounts of noise to different frames in the video, which just like, uh, complicates the learning task, uh, further or like unnecessarily. Things like this is just like one line of code, um, that can go wrong.

   7. AMAnjney Midha17:00

      What was the biggest difference between, uh, the image model research, you know, project and, and your video work? Because, you know, noise, um, noise sampling and no- noise reduction, these are sort of diffusion techniques that are shared across images and video. It, it would be helpful to understand what were unique sort of to the video challenge.

   8. ABAndreas Blattmann17:25

      First of all, the pure dimensionality of videos, I mentioned that before w- with, with, with this like additional dimension. This introduces of course, uh, lots of higher or like a, a higher, um, GPU cons- or memory consumption. Um, and, and this was really a challenge. So for diffusion models it's really important to have a high batch size because the gradients gets, uh, like y- you can approximate the gradient, uh, which drives the learning, um, much better if the batch size is, is higher. Um, and especially for diffusion models this is like really an important thing to have a really high batch size. But if you just like add this additional temporal dimension, if something breaks somewhere in, in, in one GPU It will just like throw down the entire training. And the more GPUs you add to your cluster, and on the more GPUs you train, the higher the probability will be that somewhere there's just like a, say, a hardware failure even, which also happens. So like this, this, this additional dimensionality just like introduces these new kind of scaling challenges which are, yeah, which w-which were really, really like interesting to, to come by,

8. 18:43 – 21:24

   Data set selection and training

   1. ABAndreas Blattmann18:43

      I would say.

   2. AMAnjney Midha18:44

      Well, that's very helpful. I, I think, you know, one of the, uh, most valuable things that your guys' research, uh, has done for the industry is that you often share in very excruciating detail some of the infrastructure challenges that came with training, you know, the lab's work. And I think since, uh, scaling models at the, at the magnitude that, you know, you guys are is, is r- is a relatively new infrastructure challenge, I think it's very, very helpful for other researchers to be able to hear, you know, the, the sort of nuts and bolts that you had to figure out, right, to get these models out. Then there's the whole other set of data-related challenges, um, that, that re- that aren't about the data pipeline per se, but it's about the representation of the data, the dataset, uh, curation, the dataset mixture. Could you guys just talk a little bit about how you approached, um, picking your dataset for this?

   3. RRRobin Rombach19:41

      Yeah, that's a good question. We actually, um, spend a lot of time talking about this in the paper that we just put out. Um, so, so roughly, um, what we, what we also define in this paper is that we can divide this training process into three stages. Um, and the first is that we actually train an image model. So for, for training video models, it's usually just helpful to reuse the structural spatial understanding from image models. Um, and there are like powerful image models that we can just build, that we should reuse, um, for then training the video model. And then there's next steps. So having a image model like Stable Diffusion, for example, um, you have to get this additional knowledge about like the temporal dimensionality and about motion, right? So, um, for that we train on like a large dataset, um, and I'm sure Andreas can talk about like how we filtered that in a second. But yeah, we-- like the next stage is training on like this, this really large dataset that we still have to curate a bit. So we don't want like, um, let's say optical characters like, I don't know. So yeah, we don't want like text in the, in the video. We want nice object motion. Um, we also want nice camera motion. So yeah, we have to filter for that. Um, and yeah, we, we do this in like two regimes. We train on a lot of videos in the first stage, and in the second stage, uh, we train on more curated, very high quality, smaller dataset to really refine the model. And it's s- similar to image models, uh, where you also train, pre-train on a large dataset and then refine on a high quality dataset. There was a paper recently that, uh, Meta put out that also describes just this process for image models in

9. 21:24 – 23:51

   Structural consistency and 3D objects

   1. RRRobin Rombach21:24

      detail.

   2. AMAnjney Midha21:24

      One of the largest, um, open questions in, in video generation for a while has been structural consistency, right, of 3D objects. When you-- when the camera is panning around, um, a person or a car or any, any, any subject to make sure that it stays and looks like the same s- subject from various angles has been a, um, has been a challenge for, for generative video. Um, how did you guys approach that? You know, you, you mentioned in the paper that, uh, 3D data and multi-view data was important.

   3. RRRobin Rombach21:55

      Yeah, actually like I think the main point we want to made in the-- make in the paper is, um, the one that we talked about earlier, like having a foundational video model actually learns much more than just, or actually gives us much more than just a model that can generate nice looking clips or videos, right? It learns a representation of the world. And one, um, one aspect of that is that we try to demonstrate in the paper that given a video model which has seen a lot of objects from different views, uh, lots of different camera movements, it should be much more easy to turn that into a multi-view model. And that's kind of the main message. So we, we take the pre-trained, uh, video model, which has seen a lot of different videos, a lot of different camera movements, and we fine-tune that on very specialized, um, multi-view orbits around, uh, around, um, 3D objects and kind of, yeah, turn the video model into a multi-view synthesis model and that works pretty well. So one of the like, uh, dominating approaches before that was that you would take like an image model like Stable Diffusion and turn that into a multi-view model. But yeah, we showed that it's actually like helpful to incorporate like this implicit 3D knowledge that, knowledge that is captured in all of the videos into the model and then the model can learn much quicker than if you start from the pure image model. So that's kind of the main message. But you're right, you can also, um, try to reduce this, uh, explicit multi-view data in the video training or maybe even something that we do in the paper, um, train LoRAs explicitly on like different, um, camera movements, um, and then put this LoRAs back into the video model so you get like a, a control over the camera for your very general video model, which is quite

10. 23:51 – 28:47

    Incorporating LoRAs

    1. RRRobin Rombach23:51

       cool.

    2. AMAnjney Midha23:52

       Yeah. So this I found was one of the coolest, you know, uh, pieces of the paper was, um, incorporating LoRAs for fine grain control in the creation process. Could you maybe give us a quick overview of what LoRAs even are conceptually, intuitively, and then why-- what was your-- what, what led you to the intuition that LoRAs would be an important part of the architecture?

    3. ABAndreas Blattmann24:11

       LoRAs are just really lightweight adapters which can be fine-tuned onto a existing base model which adapt the, the attention layers and by that you can just like on a smaller kind of subset or a small- It's really highly speci- specialized dataset. You can tune in a really, really lightweight way, um, different properties into the model. And in, in, in, in this case, we just like tuned it, this kind of, um, understanding of different kinds of camera motion into our video model. So if we use a small dataset which only contains like zooms or, or pannings to the left or to the right, we can actually, um, tune such a LoRA as a small adapter to, to the, to the, um, attention layers of our model to just like get exactly this behavior. And this is a really awesome way to just in a really, really lightweight way fine-tune these foundational models, and it has shown to be like really effective and it's, um, accordingly, it's, it's like really highly appreciated in the community, I would say, to get these kind of easy fine-tunes for these models.

    4. RRRobin Rombach25:25

       Yeah. And I think like for image models it's like extremely popular. Um, there's so many different LoRAs that people plug into these models. For video models, our goal was just to demonstrate that this is something that's possible. Yeah. It's just like at the beginning and, um, there's much more that it's possible, that should be possible, like very specialized kind of motions. I don't know, I think there's a lot of creative possibilities.

    5. AMAnjney Midha25:46

       That's actually, uh, I think worth exploring for a little bit. One of the, uh, one of the sort of windows that you guys have into the future is by understanding where the research is going, I think you get to see and tr- you get to live a little bit in, uh, to, in the future you get to time travel and kind of get a g- glimpse into the future of creativity. And so having seen how effective LoRAs are, at least at a few set of tasks like motion control, right? So you, in, in, in the, in the paper you propose using LoRAs for camera control, panning, zooming, et cetera. Um, the history of video creation has, has usually, uh, required creators to have a ton of different knobs and dials, uh, in their software that they use, right? Whether it's an Adobe After Effects or some other professional software, you literally have hundreds of do- dials and, and, and buttons that you can use to control and edit these videos. And conceptually, should people think about LoRAs as mapping to these controls? In the future, will a director or creator of videos basically be relying on hundreds of different LoRAs to express the control they want over the video? Um, or do you think fundamentally LoRAs will hit some scaling, um, uh, sort of limit? How, how should creators think about these new tools that you've given them?

    6. RRRobin Rombach27:09

       I, I mean, you actually, you actually said it right. Maintaining like a library of hundreds of LoRAs is maybe not like the most scalable approach. Um, actually, if you look at the model that we put out now, it's just like taking an image and animating that, right? Then we can do some stuff like with these LoRAs, um, but what you actually want, I think, is given the image and some text prompt, do exactly what I described in the text prompt. There is also some-- there's already some work that, uh, explores that, um, but yeah, giving like more control over what happens in the video, be it through LoRAs but maybe through a text prompt or through like, um, yeah, you know, like spatial, uh, motion guidance, like in Runway's Motion Brush. Um, there are different, uh, ways of doing that, but you definitely want like more control over this whole creation process. And then I think, uh, you're at the stage where you can really start to, uh, generate personalized individual content, um, individual short movies, I don't know. And maybe even for like, or especially probably for, for video creation, we want something like we just did with, um, with the image models, you want like very fast synthesis.

    7. AMAnjney Midha28:22

       Right.

    8. RRRobin Rombach28:23

       Because then this will become more like, I don't know, sometimes I think about this as like, like, like a video game, right? You, you type your prompt and you immediately see what happens, um, given your, your input. And I think this might be a super nice user experience actually. So we want this additional control, and we want, uh, fast rendering, fast sampling, uh, fast synthesis of this models, of these models.

11. 28:47 – 31:41

    How should creators think about these tools?

    1. AMAnjney Midha28:47

       You know, you said earlier that you're hoping that the community explores more things. Now that you've ac- actually put the model out there, you know, there, there, there are gonna be developers and creators who listen to this podcast, what would you like them to explore first and, and most intensely now?

    2. ABAndreas Blattmann29:01

       Well, I think just tr- tr- trying out the model, um, rendering some awesome stuff, of course. Also further exploring maybe the representation we built. We, we, in the paper we mentioned that we trained this model on a whole lot of data, and this has just seen really, really much motion. Be it on like low resolution-

    3. AMAnjney Midha29:22

       Right

    4. ABAndreas Blattmann29:22

       ... but this, this knowledge like just has seen all this and the representation is really fruitful. We showed that by our 3D fine-tuning. This was, by the way, this was completely surprising for me, seeing that model after thousand, 2000 iterations like, like already get, get- getting what is, uh, like 3D reasoning or like explicit 3D reasoning. This, this was really, really nice. Um, and so as we saw that, it will be extremely interesting to see other such, um, approaches. Model is open source. People can try it, and I, I think it will like give it a another couple of weeks, and then we will, we will see what happens. But I'm excited for it.

    5. AMAnjney Midha30:02

       You know, my personal favorite for what people did on day one was obviously animating memes. [laughs] I'm sure you guys have seen all that.

    6. RRRobin Rombach30:08

       That, that, that one, uh, that was really funny. Yeah.

    7. AMAnjney Midha30:11

       What are your guys' favorite creations so far that you've seen? Anything that jumps to mind?

    8. RRRobin Rombach30:15

       The one where the man is looking after another woman.

    9. AMAnjney Midha30:18

       Oh, it's the man looking behind. Right.

    10. RRRobin Rombach30:20

        The man looking behind. Yeah. That one. Um, I, I think it just like visualizes just like the additional experience that a video model can provide, right? So it's actually really nice to you, you- People are used to like 2D memes, but then you see, oh, I can, like, I can actually try to animate this and see what happens.

    11. ABAndreas Blattmann30:39

        Also, if you think about famous artworks or something, just bringing them to life is a really, really, uh, like n-nice property and it's, it's now enabled. Everyone can just like poke around a bit on Mona Lisa and see what sh- what she's looking from the side.

    12. AMAnjney Midha30:54

        Oh, that's cool. I haven't explored that one, but that, that's a... You're saying prompting the model with an image of a, of a notable art piece, um-

    13. ABAndreas Blattmann31:01

        Exactly.

    14. AMAnjney Midha31:02

        Ah, interesting. Okay.

    15. ABAndreas Blattmann31:03

        Just, just like ma-making Van Gogh's, uh, Starry Night, like, like make the stars shine and, and, and glimmer and, and stuff. So I think it's really cool. Yeah.

    16. AMAnjney Midha31:12

        The world is pretty lucky that you guys have gifted, you know, this, the model to the, the developer, um, and open source ecosystem. Um, it's, it's already such an incredible sort of step, leap forward, right, in what you can do with, um, with images and with, with video. Um, what do you think are the top, the, the two or three biggest sort of open challenges that you guys wanna prioritize next that are still limitations in video generation?

    17. ABAndreas Blattmann31:39

        I think a

12. 31:41 – 32:35

    Open challenges in video generation

    1. ABAndreas Blattmann31:41

       really important thing is to get these models to generate longer videos, to process longer videos in general. Not only generate them, also like see them and, and, and, and process them because I think eventually processing longer videos is, is like key to understanding what we talked about earlier, fundamental aspects of this world better. This is a really important part, um, to just like make these models or enable these models to generate longer content, more coherent content also with lots and other kinds of motion. Um, and what Robin already said, I think making them fast will just like-- It will just like unlock so much more exploration.

    2. RRRobin Rombach32:27

       Yeah. There are simple things like thinking about like multimodality, um, adding a-an audio track to your generated

13. 32:35 – 39:00

    Infrastructure challenges and future research

    1. RRRobin Rombach32:35

       video.

    2. AMAnjney Midha32:36

       So you were talking, Andreas, earlier about the infrastructure challenges here. If you had a magic wand, what infrastructure improvements do you wish the industry could solve for you?

    3. ABAndreas Blattmann32:46

       Oof. [laughs] A hard question. I mean, we could, we could, we could ask for more GPUs per, uh, more CPUs per GPU, and this would like solve, solve much of the data loading issues also for more like, like not only GPU memory is always good, but also CPU memory. But I think like hitting these limits is just a, some form of natural way. You, you always wanna try to improve your efficiency. You always wanna try to, to, to train faster, um, and, and at some point you will face a bottleneck, a limit, and you have to like come up with a, with a nice algorithmic way maybe or with another way of like, like overcoming this. For many years, data loading was not a, a, a big thing because the GPUs were too slow. But now we have extremely nice accelerators and, and, and like, like with the newest H100s, this is, it's insane how fast these, these GPUs are actually running, not how fast you can train models on those. Um, and then you will just like hit the next bottleneck. It's actually good to see this, that, that we hit limits. We have to overcome this, and then you improve, and this is how you learn and, and this is how you, you can, yeah, make things much more efficient in the end.

    4. RRRobin Rombach34:02

       If you only rely on like more compute, um, it's a bit boring. Um, I think like compute constraints can also drive innovation, right? So, uh, for example, the latent diffusion framework, we developed it at the university because, uh, we just like we had like single GPUs, uh-

    5. AMAnjney Midha34:21

       Right

    6. RRRobin Rombach34:21

       ... where we trained on. And, um, yeah, I mean, uh, that kind of naturally leads to some kind of innovation. And in this case, this is something that everyone uses right now. This is-- It's actually crazy to see DALL-E 3 uses a model that like the autoencoder that was trained on a single GPU.

    7. ABAndreas Blattmann34:39

       This, this is I think how intelligence also arises. If you have a constrained environment, you have to come up with a smarter way of, um, doing things, and this is how, how... Without, without, without any limitations, you, you, yeah, there wouldn't be as, as, like those nice solutions for many problems we have right now.

    8. AMAnjney Midha35:04

       Yeah. No constraints, no creativity, right? Um-

    9. ABAndreas Blattmann35:07

       Exactly.

    10. AMAnjney Midha35:08

        And I, and I, I do think one of the underappreciated parts of your guys' group ever since your university days has been just how compute efficient a lot of your research has been. Um, I, I certainly have talked to so many university-level, uh, researchers, you know, s- grad students, postdocs, um, who saw that research that you guys put out a year and a half ago with stable diffusion and felt really inspired. University and academic environments are somewhat compute constrained. Um, and so I, I think even though now you have access to tons of compute, um, the sort of self-imposed compute constraints, uh, leading to... It, it, it makes me very happy to hear that those, those are something, those constraints are something you guys think as a fe-- are a feature, not a bug. Right. Uh, and will probably keep the open source ecosystem pretty vibrant. But in addition, you're often sort of racing and, and responding to other labs as well in the field. Um, some of these labs are much better funded than you, are bigger than you. Um, so how do you think about prioritizing your research pipelines and your timelines? Um, and how would you say that's different than labs that are largely academic?

    11. RRRobin Rombach36:15

        Yeah. That's a, that's a very good point. I think like, um, actually this whole competition, it also drives the field of AI. It's probably very important to not get distracted by this too much. But of course, like [chuckles] since one and a half years, everyone is doing something with, uh, diffusion. It can actually be quite fun to, yeah, uh- Work in this competitive environment, um, I think, yeah, everyone here enjoys doing that. It's, it's quite fun that we have like this, this lab here in, in Germany actually. Um, we compete with, I don't know, OpenAI, um, Google, other research labs a- across the world. It's, it's quite fun. Um, it's intense definitely, but it's, it's a lot of fun.

    12. ABAndreas Blattmann36:54

        I think in the, in the end we're not a big lab, but I think we're, we're really like all having kind of the same spirit and we're feeling like that, that we're working on something which makes sense and which in the end gives not only us something for us it's also really cool, but also we can give something back to the community, to other researchers which might have not the resources we have.

    13. AMAnjney Midha37:21

        Well, what I, what I love about, um, the lab and the group you guys have put together, the, the philosophy of, of the rising tide lifts all boats, right? Because you guys publish your research for the world to use. Um, and I thought one of the coolest things about the DALL-E 3 paper was, uh, the citations list which included, uh, your work, right? And so they were saying, they were sort of thanking you, um, for the work that the, the Stable Diffusion group had put toget- put out. Um, I think your work ends up benefiting all kinds of labs across the industry. And so while the competition can be intense, um, it's, it's also one of the best, more s- most inspiring examples of industries-- of an industry helping each other out. And it, it comes up all the time in conversations with researchers at many of the labs we just talked about that they're very grateful for the research you guys do. So I, I hope you'll keep doing that.

    14. RRRobin Rombach38:14

        Yeah, just shows that it is super important to have this kind of, um, contribution to open and accessible models and everyone, um, in our team is super motivated to contribute to that. So, um, couldn't, couldn't imagine doing anything else right now.

    15. SPSpeaker38:30

        Thank you so much for listening to the a16z podcast. If you've made it this far, don't forget to subscribe so that you are the first to get our exclusive video content. Or you can check out this video that we've hand-selected for you. [outro music]

Episode duration: 39:06