By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Latent SpaceWhy FLUX 3 Trains on Images, Video and Sound — Andreas Blattmann & Dustin Podell, Black Forest Labs
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
Great conversations run long. Their best ideas don't have to.
uListen is free on iPhone, iPad, Android, Chrome, and Claude.
See howTranscript
115 min read · 22,926 words- 0:00 – 3:39
Host introduction
- SWswyx
Happy to bring to you our episode with Black Forest Labs. Our guests today are Andreas Blattmann, he also goes by Andi, who is co-inventor of latent diffusion, the technique that eventually got scaled up by Stable Diffusion and Stability AI. And they eventually split with a bunch of, uh, the core Stability AI image gen folks to form Black Forest Labs. And Dustin Podell, who I also met with Ansh, uh, at one of the stories that we tell, and, uh, who worked his way up from just being a regular engineer, a freelance, like, web developer person, into a model and research lead at Stability AI and now, uh, Black Forest Labs, creator of SDXL, Control LoRA, post-training lead on SD3, Flux Ultra, Flux.1, and Flux Kontext. And we go into all of these details. Um, as you may know, sometimes we're very honored to have an extended history with some of our guests, and this is one of them. You might have heard in our Ansh Mittal episode where we talked about introducing each other at one of the NeurIPSs and meeting Black Forest Labs for the first time. And, uh, this was right after Stable Diffusion. But what you may not have heard, and we only tell this story in this episode, is how Latent Space itself was started because of Stable Diffusion directly, uh, crediting the launch for me basically quitting my job to go all in on AI based on what we saw. And today's guest, Dustin, also [chuckles] had that experience, and so did Vibhu. And so we have a very, very strong tie with the team that basically led up to Black Forest Labs, and now we're also interested in what's next for them. Like, they have created some of the best open and closed weights models for image generation and video generation. I think especially Flux Kontext, if you're not aware of image editing models, I do think that this is in the ex- in the limit one of the things to fully replace Canva and, um, and Photoshop and a bunch of other workflows and editors that you might use in a normal creative workflow. It is not going to replace the top end of the market, but for regular, normal, everyday edits, why would you not want anything else other than to query a model and ask it to do, uh, image editing for you? Uh, we do this obviously with this kind of video editing. It's all Clanker done, and I do review it, and they're also moving into robotics. And so this ties in with a mini-series of model labs that have been pursuing this direction, building world models from image gen like with Runway, uh, as well as, um, video models and now robotics models. And so I think a lot of things rhyme in a lot of our recent episodes, like even the artificial analysis episode that we're gonna release soon, they're also going into robotics. And, like, it seems pretty s- obvious that when you see a whole bunch of smart people who are at the frontier all leaning one direction, that you're probably six to 12 months out from some kind of really big breakthrough. This has been a consistent theme over my last four years of podcasting. It's not gonna change now. So look out for it. Um, that's the most obvious call-out I can make, but there's also a lot of interesting lore for people interested in the general diffusion space. Uh, we're gonna d-do a little bit more over time. I, I'm thinking about an AI-- AIE LA where we talk about the generative media field and the creative arts and how, uh, the complicated relationship is evolving, um, and I think maturing, to be honest. And finally, for those longtime listeners, uh, we are starting to open up to advertisers, people that we really admire and respect and wanna showcase what they're working on. Um, the first one was with Superbase, which I think a lot of you enjoyed, and the second one today is brought to you by Deepgram, and you'll hear more from us at the midpoint of the episode.
- 3:39 – 4:23
Cold open
- SWswyx
[upbeat music]
- ABAndreas Blattmann
You should be showing a robot an action one time, and then it should be able to do this, and this would be truly a superhuman intelligence like-
- DPDustin Podell
At Stability that we couldn't essentially leverage the value of what the models could be as well. The mistakes of not being able to actually capture the value of the models themselves.
- ABAndreas Blattmann
We raised all in all $450 million-
- SWswyx
Oh my God
- ABAndreas Blattmann
... of funding so far.
- SWswyx
Yeah.
- ABAndreas Blattmann
Currently, the last, uh, that was published in, uh, December was $100 million in revenue. BFL was not founded to be an image generation lab. Video modeling is actually the central means to understanding how our world basically works.
- DPDustin Podell
It was maybe like six hours of training, four hours of training. It was, it was a very, very short training run. And all of a sudden, like, I saw what felt like a clear generality coming out of it because it was [upbeat music]
- 4:23 – 9:23
Black Forest roots
- SWswyx
Okay. Finally, our BFL pod. Welcome, guys, Andreas and Dustin. [chuckles]
- ABAndreas Blattmann
Great to be here. Thank you so much.
- DPDustin Podell
Thank you. Yeah, definitely thank you for having us. I know we've chatted about doing this for a long time.
- SWswyx
Yes. So we've had Ansh on a couple times, and that's where I met you, at Vienna. I think one of the ICMLs or-
- DPDustin Podell
Was this like one of the, like, was, was the like rooftop, uh, event, or, or what, what was it?
- SWswyx
There was a year where there were two Viennas back to back.
- DPDustin Podell
Yeah. Yeah, yeah, yeah.
- ABAndreas Blattmann
That was first ICLR, and then it was ICML.
- SWswyx
I don't know how they do this thing. [laughs]
- DPDustin Podell
Yeah. [laughs]
- SWswyx
Anyway. Anyway, uh, and at the time, you guys had-- were like just about to, uh, launch BFL. Like, we had a dinner, and like, you were like, oh, and thinking about what's next after Stability and, uh, all that. So, uh, anyway, congrats. How do you like-- guys like to introduce yourself? Maybe we'll sort of get a crisp, uh, self-intro up front.
- ABAndreas Blattmann
So I'm Andi. I'm a co-founder of Black Forest Labs. Been developing diffusion models for, oh wow, I think seven years now. Uh, met Robin and Patrick way, way back in the day, I think it's, yeah, seven years now since we met at Hei- in Heidelberg. Uh, I'm from Germany o- originally. And there we started looking into, yeah, just diffusion modeling for images. Uh, we're all having a kind of a physics background, um, in some sense, and this, this paradigm of just like mapping this general physical phenomena of diffusion to machine learning and generative modeling had a, a kind of a philosophical kind of beauty to us. That's why we got into this. And then, yeah, developed a lot of the, I guess, fundamental algorithms that are powering diffusion models today, or image generation, video generation, and also action prediction today. I guess we'll talk about this later
- SWswyx
We'll talk about it.
- ABAndreas Blattmann
But, um-
- SWswyx
Yeah, the list of, of a kind related to... It's not immediately obvious to people, image generation to action prediction.
- ABAndreas Blattmann
Yeah.
- SWswyx
But we can talk about that. Uh, and then one, one thing I did wanna comment is a lot of people don't know the relationship. Black Forest is a forest in Germany?
- ABAndreas Blattmann
It's a mountain range, actually.
- SWswyx
Ah.
- ABAndreas Blattmann
But, um, yeah, it's a, it's a region in Germany, basically.
- SWswyx
And you named it after that.
- ABAndreas Blattmann
And funnily, Robin and me, we f- we first met during our PhD, and then we figured out that we were actually born, I don't know, 10 miles away from each other. Um, and that was, that was pretty fun. And then eventually that led us also, uh, to come back to Germany.
- SWswyx
Beautiful. [laughs]
- ABAndreas Blattmann
[laughs] Yeah.
- SWswyx
All of Europe looks like this. Yeah.
- DPDustin Podell
You can see the lab right in the middle there. [laughs]
- SWswyx
Yeah.
- ABAndreas Blattmann
Yeah. Yeah. Um, so-
- SWswyx
So you named it because you're from here.
- ABAndreas Blattmann
Exactly. We named it because actually, um, we founded this research lab, and back in the day we could basically pick where we wanted to do it. No one said, "That's okay, you have to come to, say, SF or, or whatever place." So we decided, okay, we wanna be actually starting this at the place where we were born in Freiburg, which is like at the foot of the Black Forest basically, or a city in the Black Forest. And that's how it came, and that's how, uh, then the company also started there.
- 9:23 – 17:04
Dustin’s path
- SWswyx
[laughs]
- DPDustin Podell
Nice.
- SWswyx
Dustin, like, uh, yeah, what was your journey into diffusion and AI in general?
- DPDustin Podell
Yeah, yeah. So I, I came from a little bit of a different background than, than Andi and Robin and team so I'm, I'm not from the German side but I've still had a fantastic time working with these guys. And, uh, honestly it's funny you mentioned that you quit your job when Stable Diffusion came out because, uh, the original Stable Diffusion I was not working with these guys yet. And it was actually a cause for me to quit my job at the time so I, um-
- SWswyx
You caused mass unemployment. [laughs]
- DPDustin Podell
[laughs] I'm actually curious like the numbers of people that just com- like changed their life direction when Stable Diffusion dropped. I, I would actually be very curious to know kind of the branching effects from that.
- SWswyx
Nice.
- DPDustin Podell
But I don't know, we can, we can theorize on it. [laughs]
- ABAndreas Blattmann
That's actually super interesting because here in SF, in the Bay it had this huge impact whereas in Germany where we stayed back in the day, that day people didn't really notice it because like-
- DPDustin Podell
Oh
- ABAndreas Blattmann
... the people were so, so far away from tech.
- DPDustin Podell
Mm-hmm.
- ABAndreas Blattmann
Like there are a couple of people obviously that are, that have been involved in the open source, uh, community also in Germany but the majority of people didn't really notice, uh, that drop in summer 2022.
- DPDustin Podell
Mm-hmm.
- ABAndreas Blattmann
Which was really interesting and then you see what happens here like h- Yeah. That, that was-- to, to us that was really kind of super interesting.
- DPDustin Podell
Yeah. But, but anyways so I, I guess to give a little background on me so I, I, uh, at BFL I, uh, am one of the founding researchers that joined in with the, uh, we had a, a bit of a larger team that I'm sure we'll get into the history of kind of how that all came to be. But my background is not actually traditionally in you know the ML space whatsoever. I was traditionally a software resea- or, or a software developer for you know around 10 years or so. Early in 20- you know around the 2020s I became very, very interested in just like open source AI. I saw like the kind of the early work in GANs and other you know in generative media and I was tracking with it, I was experimenting with it a little on the side. It got me quite excited. I mean, and I remember following originally the, the latent diffusion work 'cause actually I was, I was following that and quite excited about it and then, uh, the one you know one of the things that kinda tipped me off to go okay maybe this is something a little bit you know a little bit more serious is actually a shout out to, uh, Midjourney V1.
- SWswyx
Wow.
- DPDustin Podell
Um, back when that came out and I, I remember thinking like okay this, this is actually going in, in a direction I, I can see this being quite impactful. And then the one big moment, the big drop that really changed everything for me as we talked about was the Stable Diffusion release and at the time I was working at a software studio down in Los Angeles. Stable Diffusion came out and basically that day I turned in my resignation [laughs] and said, "Okay-"
- SWswyx
But what was, what was the plan like with it? [laughs]
- DPDustin Podell
The, the plan? So, so the way, the way that I saw it, the way that I like envisioned it at the time I saw this thing that felt like-- it, it literally felt like a magical thing that came out, and I viewed, okay, the people that are working on this must be like arcane wizards, and I need to do whatever I can to understand like what magic or what runes or what things that they did to imbue in this to make this happen and let this go out to the world and have this, have this impact. It wasn't like I was just totally disconnected. At the time, I was in the Stable Diffusion Discord. I actually had friends in there. I was, I was kind of interacting with the community. So when I, when I-
- SWswyx
So was, was, was it viewed as basically open source Midjourney, or like, like do people not make the connection?
- DPDustin Podell
There was-- I mean, I, I was in both. I remember there were a lot of kind of, uh, discussions around this, like, oh, Stable Diffusion was-- it's having such a-- You know, there, there was a lot of discussion, and I remember the Midjourney Discord and the Stable Diffusion Discord around that time around the, the impacts it was having. I mean, it- it's been a long time, so I can't remember the exact-
- SWswyx
Yeah
- DPDustin Podell
... nature of it. I don't want to speak on the, you know-
- SWswyx
Midjourney's doing fine. Like they're doing, they're doing great [laughing] .
- DPDustin Podell
They're doing-- I mean, they're, they're, they're doing great. Um, but for me it was more just like, I was very interested in the fundamental technology, and then the fact that I could now run this at home. And then really the thing I saw was then just this community of developers actually like taking it home, developing on it. I was hacking away at it. I was meeting a bunch of other really, really smart, great people, some of which have-- are actually at BFL now that have, have come along for the journey. That moment of just community interaction and, and learning and developing and building together was s- really an impactful moment for me. Luckily, luckily enough for me, I was able to find a kind of a job opportunity to get myself into Stability at the time, where, uh, you know, this, this, this was starting. And it, it-- I didn't actually start working with these guys right away. It took me a while to kind of, to kind of work my way up to that, and, and we can go into that history later if we want. But-
- SWswyx
'Cause Stability had a bunch of different, uh, teams, right?
- DPDustin Podell
It was, uh-
- SWswyx
Stable LM, Stable Music. There was-
- DPDustin Podell
Yeah, Harmoni, yeah, the audio team.
- 17:04 – 27:38
Visual intelligence
- AFAlessio Fanelli
You did latent diffusion. We'll get through all the history, but for people that haven't super followed the space, where are we today? Like, what's the, what's the state of things? I know FLUX.3 video is out, uh, FLUX.3 action, we're now like 7B doing robotics. What's the latest in this space with you guys?
- DPDustin Podell
We started as an image generation lab-
- AFAlessio Fanelli
[laughs]
- DPDustin Podell
... when we started Black Forest Labs, right? As a kind of legacy of Stable Diffusion and the likes. Um, and that has been, I think, our, our thing for the first year. But then we realized, okay, you actually-- image generation is nice, but this technology can unlock so much more.
- AFAlessio Fanelli
Mm-hmm.
- DPDustin Podell
And that's when we slowly went into what we call visual intelligence, which is basically trying to get our intuitive intelligence that we people have when we-- say, when we catch a ball and just understand physics without actually knowing how to mathematically describe it or something, or when we grab a bottle and just know exactly how to do this because we've done it 100,000-- maybe not 100,000 times, I don't know, but a lot of times, uh, in our, in our childhood, right? We wanna be transforming that
- ABAndreas Blattmann
Knowledge and that f- form of intelligence, which is very different from language model intelligence, into models. And learning based on pixels, which means not only images, but also videos and maybe also audio, say not pixels, but natural signals is a, maybe a better term-
- SWswyx
[laughs]
- ABAndreas Blattmann
... that really unlocks this. And that is what we realized, and we realized, hey, we have the algorithm to actually do this, which is diffusion or flow matching. And that is when we just said, "Okay, we need to extend from image generation to basically all the natural, uh, representations," which led to FLUX 3 basically, which is one model that learns multimodal from images, videos, audios, also actions. We can talk about how this comes together later. To really get this, to, to keep it with Daniel Kahneman, I don't know if you read-
- SWswyx
Thinking, fast and slow
- ABAndreas Blattmann
... Thinking, Fast and Slow. It's a ki- system one intelligence. It's not system two intelligence.
- SWswyx
Mm.
- ABAndreas Blattmann
And you can maybe say-
- SWswyx
Oh, my God
- ABAndreas Blattmann
... may- maybe, maybe-
- SWswyx
Everyone is system one models now. [laughs]
- ABAndreas Blattmann
May- may- maybe, maybe-
- SWswyx
Visual, yeah
- ABAndreas Blattmann
... language modeling is system two, but we are focused for now on system one, which is extremely important for everything that is intuitive and that requires real-world interaction. That's why at, on a high level, it's also so good for robotics. So that's been the journey, and now we're really focusing on that. And in the, in, in the end obviously, um, it's about combining both of these systems into this broad, very, very broad way of intelligence that humans also have.
- SWswyx
I wanna go into sort of, uh, history and all, all these things. Uh, but just to also recap for the, for people at the front, you're already making a s- significant money as a generative media company, right?
- ABAndreas Blattmann
Exactly.
- SWswyx
Um, I don't know what the, the numbers are and, and maybe you can sort of recap there. I, I think you have some-
- DPDustin Podell
You raised, like, you raised $300 mil-
- SWswyx
You raised a ridiculous amount of money
- DPDustin Podell
... in multi. Yeah.
- ABAndreas Blattmann
Yeah.
- SWswyx
So-
- ABAndreas Blattmann
We, we raised I think-
- DPDustin Podell
Any re-
- ABAndreas Blattmann
... uh, all, all in all, $450 million-
- 27:38 – 33:59
Inside SDXL
- AFAlessio Fanelli
What are the key things people should know? You know, I'll leave it, I'll leave it pretty blank.
- SWswyx
I mean, first would be SDXL maybe-
- AFAlessio Fanelli
Yeah, man
- SWswyx
... if we're talking about SDXL a bit.
- DPDustin Podell
I was gonna say, what were the, what were the things we did in SDXL? [laughs]
- SWswyx
[laughs]
- AFAlessio Fanelli
The refiner.
- DPDustin Podell
The... Oh, no, please.
- SWswyx
[laughs]
- DPDustin Podell
Not the refiner.
- SWswyx
Wait, why, why that reaction?
- DPDustin Podell
So SDXL was a big piece of work in trying to make the models feel good to use for the first time, if I was to kinda like lay anything out. Not to say like Stable Diffusion 1 and 2 weren't, you know... I mean, they were very exciting, they were very fun to use, but, uh, to anyone that was like in the space at the time between the period of SD, you know, SD1 to SDXL, uh, we saw a ton of work out in the, in the open source community working on how do we fine-tune these models, how do we make these models, you know, more aesthetic, how do we make these models into like, you know, uh, multi, you know, uh... I, if I remember, it's like the original Stable Diffusions were only able to do square images. And so SDXL was the first work of trying to r- actually release like a found, like the, the base foundation model, kind of with a lot of these general improvements implicit in the model. So, uh, SDXL involved, you know, the first time we did large scale, uh, multi-aspect training. There was a, an, you know, at the time, uh, this was before, uh, packing. There was like an efficient way we could do essentially bucket style training, so training, you know, on all sorts of different resolutions of, of images and, and, you know, at a large scale. This was the first time we did multiple actual training stages, so taking it from pre-training, actually developing like post-training sets. So the, there was a lot of data work that was done in SDXL. I'm trying to think like all the other aspects. And then, uh, I'll, I'll give you kinda like a little bit of a backstory on the refiner and maybe, maybe some things that, uh, didn't quite make it in. So here, here was the whole process of the final release model is essentially SDXL fundamentally works similar to the original Stable Diffusions. It was a latent diffusion model, took in text. It had, you know, you had a noisy latent, took the context of the text, actually generated the, you know, generated the imagery in latent space, and then a VAE decoded it. And then from there, the refiner was actually another full diffusion model that the, uh, the whole point of it was to take essentially what was not a rough image, but SDXL wasn't the best at what we would call high frequency detail, which is this like the fine detail, like skin texture, texture of grass-
- SWswyx
Yeah, upscaling.
- DPDustin Podell
Tex- texturing. Up- upscaling.
- SWswyx
Yeah.
- DPDustin Podell
It wasn't an upscaler, 'cause it didn't actually change the resolution, but it, it, y- yeah, upscaled in terms of-
- SWswyx
It's a refiner
- DPDustin Podell
... in terms of the, in terms of... It refined it. The reason I thought it was janky is 'cause I worked, I... Not just me. The, the whole team that worked on this was, was fantastic. I don't wanna take all the credit at all, 'cause there were, there were a lot of fantastic people that, that worked on this. We did a lot of work to try to push the detail quality of the model while trying to keep this like generality And we just didn't quite get there. And so then there was this like side track of work of like, "Okay, what are other methods to do this?" And this is where the refiner came into being with actually-- The refiner was actually a model that was, it was what was supposed to-- So th-this is a little bit of the secret lore, is there was a version of SDXL that was never released. Uh, it was a different architecture. It didn't quite work as well. But it was trained pretty far along, even to like the post-training stages. Uh, the refiner was that model. It is a full diffusion model.
- ABAndreas Blattmann
I actually completely forgot about that.
- DPDustin Podell
Yeah, yeah. [laughs]
- ABAndreas Blattmann
[laughs]
- DPDustin Podell
It- it-- Like, the refiner was a full diffusion model. And the one thing about that model is it had a little bit of better performance on these high-frequency details. It just had very poor structure. And so this whole idea came of like, what if we essentially like-
- ABAndreas Blattmann
Different model, different job
- DPDustin Podell
... different model, different job. We, we took it. We did a lot of work to really push the, you know, the work on how well it could actually refine this high-frequency detail. And then we made this, this, essentially this pipeline to-
- ABAndreas Blattmann
Yeah, which, which is the XL part
- DPDustin Podell
... which, which then became, okay, this together is like a more performative model, you know, or a more performative system altogether. And then that was what we ended up releasing here.
- ABAndreas Blattmann
Which is actually interesting. It was one of the early pioneering works of like actually-- that looked into, okay, what kind of properties learned in which part of the trajectory of this diffusion kind of, uh, trajectory, right? Because you have these structural things that are learned closer to the noise, if you-
- DPDustin Podell
Mm-hmm
- ABAndreas Blattmann
... if you come closer to the noise. And then you have the fine high resolution, uh, high frequency stuff that comes further down and, and is closer to a kind of, um, the clean image, basically. And like today, there's a bunch of different works that actually analyze this. But you do actually, you, you kind of, you regenerate from low frequency to high frequency. If you go from like noise to clean image, right, along this trajectory. But that was not really known. Like, it was informally maybe a bit known by then.
- DPDustin Podell
Yeah.
- 33:59 – 39:57
Learning motion
- AFAlessio Fanelli
see the differences right there.
- DPDustin Podell
Yeah, yeah. Uh, so video.
- ABAndreas Blattmann
I've been working on video for quite a-- Actually already before Stable Diffusion, I was working on, uh, video generation, uh, for NVIDIA. Um, had a paper there, Align Your Latents, where we basically mapped the-- It was actually in parallel to Stable Diffusion. We j- basically mapped this latent diffusion algorithm to video, which was the, the na- natural next thing to, to, to do.
- DPDustin Podell
Mm-hmm.
- ABAndreas Blattmann
It was very exciting to kind of then this-- at, at once see things moving. And you can imagine the kind of compression, um, reasoning behind why Stable Diffusion was so efficient compared to other models in the, in the time. Also holds for video where you actually have this temporal dimension that you can also dis- uh, compress, right? So that made a ton of sense. Uh, worked on this for, first for NVIDIA, and then when we, when we basically joined Stability, we, um, basically after, I think after SDXL, um-
- DPDustin Podell
Mm-hmm
- ABAndreas Blattmann
... we worked on this, this, um, Stable Video Diffusion, which basically first applied this, this latent diffusion, um, approach to video asset. But importantly, this, this work was a lot about investigating the effects of data filtering, which as we felt, or as I personally felt back in the day, were like in research, and they still are very much underappreciated. Like data [laughs] is just like, when, when, when we, when we think about what these models are doing, effectively, we're estimating a probability distribution, and the data set is what gives rise to this probability distribution, right? And if you have the perfect algorithm, the best algorithm in the world that, that just learns anything you want, but your data is very bad, and you have like on average blurry videos or images and like, like not a lot of motion in the videos, your model will learn that no matter how good your algorithm is. So what comes out of the model will actually be very bad and like, like-
- DPDustin Podell
Mm-hmm, yeah
- ABAndreas Blattmann
... not nice. So the data is one of the most important levers that you have as one who, who wants to train these models. And in the entire research field, there was never much talked about data because a lot of people from the research community, they just worked on benchmarks and like, like tweaked the algorithms. But if you talk about frontier models, obviously we know that by now that data is extremely important, right? And, and that is what we wanted to be really like sharing a bit, like sharing more context on. So we investigated how much data, like... And then you have this, this trade-off, I wanna be training on a, a lot of data.
- DPDustin Podell
Quantity versus quality, basically.
- ABAndreas Blattmann
Exactly. Where is the-
- DPDustin Podell
Mm-hmm
- ABAndreas Blattmann
... where is the sweet spot? That was, uh, what we investigated there. And then also a very interesting chart-
- DPDustin Podell
Is there, is there a chart we should pull up? Uh-
- ABAndreas Blattmann
... co-co-coming, coming.
- DPDustin Podell
Ah.
- ABAndreas Blattmann
We, we can actually t-take it. The, the, the most exciting experiment that we did was not the data one, but actually if you go down a bit, um- We then-- And it comes back to what we discussed earlier. We then showed that just pre-training on plain 2D videos allows you to fine-tune a explicit 3D model extremely fast from it. So if you go to the, to the last paragraph here of the, of the paper. Here we go. These, these are these 3D, uh, 3D, um, views here of, of the model. So what we actually did is we took this video model that we trained, and then we had, I think, 1K fine-tuning steps on 3D data, and we got a 3D model out of it.
- AFAlessio Fanelli
Mm.
- ABAndreas Blattmann
And, and that, that comes back to the question, okay, what, what do you actually need to, to kind of understand the world in 3D, right? Only 1K-
- AFAlessio Fanelli
[laughs]
- ABAndreas Blattmann
... steps of fine-tuning that shows that this video model, although only trained on-
- AFAlessio Fanelli
It was learning something
- ABAndreas Blattmann
... on a plain video-
- AFAlessio Fanelli
Mm-hmm
- ABAndreas Blattmann
... actually had this insane amount of 3D understanding, and you just took a bit of a, a, a, a focus to get a 3D model out of it.
- AFAlessio Fanelli
Mm.
- ABAndreas Blattmann
And that, that is really like-- That was one of the early experiments where we saw, okay, this is actually kind of-- You can get world understanding through-
- AFAlessio Fanelli
Got it
- ABAndreas Blattmann
... training on plain video. And like, from a research perspective, this was, like, I think the most interesting experiment, um, in this paper.
- AFAlessio Fanelli
Also pretty short paper. There's like a section here where you even explain the pipeline, right? There's like three, four stages, pre-training, some post-training. Uh, yeah, stage one, image pre-training, video post-tr- uh, video pre-training-
- 39:57 – 46:57
Flow matching
- SWswyx
Forest Labs. Uh, anything else in the stable world?
- AFAlessio Fanelli
Uh, S... I mean, there's SD 3.
- SWswyx
SD 3.
- AFAlessio Fanelli
Yeah, SD 3 was kind of-
- SWswyx
And that was the last paper that you did before you-
- AFAlessio Fanelli
Yeah, yeah, SD 3 was a, a, I felt like a pretty big paper. [laughs]
- ABAndreas Blattmann
That, that was a-
- AFAlessio Fanelli
Yeah
- ABAndreas Blattmann
... that was a, I think one, a culmination of our work of, and stability, I would say, like-
- AFAlessio Fanelli
Yeah, absolutely
- ABAndreas Blattmann
... we got-
- SWswyx
It's your graduation thesis. [laughs]
- AFAlessio Fanelli
[laughs]
- ABAndreas Blattmann
If you want, yes, I mean...
- SWswyx
[laughs]
- ABAndreas Blattmann
No, this, this paper was really about understanding more in which part of this trajectory from data to noise does the model actually learn what properties and where do we have to spend most of our computation that we put in-
- AFAlessio Fanelli
Yeah
- ABAndreas Blattmann
... training to maximize the performance of a model.
- AFAlessio Fanelli
Yeah.
- ABAndreas Blattmann
So b- before-- A- and obviously that was also the transition from classic diffusion to flow matching with a, a slightly different, uh, uh, paradigm, basically.
- SWswyx
We call it rectified flow.
- ABAndreas Blattmann
Exactly. Um-
- AFAlessio Fanelli
This was also the first time we used like diffusion transformers, right, with the, the, the-
- SWswyx
Yeah.
- AFAlessio Fanelli
And then, and then, and then we took it a bit further, yeah.
- ABAndreas Blattmann
So lot, lot, lots of changes like going from a convolutional architecture to a transformer architecture-
- AFAlessio Fanelli
Yeah
- ABAndreas Blattmann
... changing the probabilistic model from, from a diffusion, um, to basically flow matching-
- SWswyx
Mm-hmm
- ABAndreas Blattmann
... which is way simpler mathematically, which is super, super nice.
- 46:57 – 50:36
Starting BFL
- SWswyx
anything that you would see of this quality was either old Stable Diffusion or pretty much-
- DPDustin Podell
Yeah
- SWswyx
... a closed, closed, closed door.
- DPDustin Podell
Mm-hmm.
- SWswyx
Yeah. Then you had to big-- make the big break to Black Forest. Uh, what was that whole period like, right? Like basically you had to leave and start a lab from scratch.
- ABAndreas Blattmann
I think it was probably one of the most exciting times, uh, we all had because when you're in stealth, you're just, like... You're a small team. We were 13, 13 people I think-
- SWswyx
Yeah
- ABAndreas Blattmann
... when, when, when we started. 14, yeah.
- SWswyx
We first went to France. Had a, had a, had a, a fun little week in, in, in the, in some-
- ABAndreas Blattmann
I remember that. It was fun, yeah
- SWswyx
... house in France in a, in a, in a chalet. Spent the entire week just hacking, hanging out. Dustin figured out some really cool, uh, um, [laughs] uh, stuff to make our models way more consistent. I, I, I, I remember that. That was so fun. That was a magical week. That really was our kind of founding offsite. That is when we started, um, training FLUX.1, which three months later, um, resulted in this nice company launch basically with a model that, that I think blew pretty much everyone away. Uh, at least blew us, us also away. [laughs]
- DPDustin Podell
Oh, yeah.
- ABAndreas Blattmann
Like commercially what, what, what this directly unlocked, like only, only having, having kind of incorporated the company for three months and then releasing this model and it was like-- Yeah, just like was received very, very well, right? That was, that was amazing.
- SWswyx
Yeah.
- ABAndreas Blattmann
Uh, so that was-
- SWswyx
But like-
- ABAndreas Blattmann
... a lot of fun
- SWswyx
... did, did you know that when you started this, it-- I mean, you had to get all the, the, uh, you know, the clusters together, like all the data that you had to start from scratch, scratch.
- ABAndreas Blattmann
We had to start from scratch-
- SWswyx
Mm-hmm
- ABAndreas Blattmann
... yes, but we knew what to do basically.
- SWswyx
Yeah. Obviously.
- ABAndreas Blattmann
We, we had a lot of experience, um, and-
- SWswyx
But like, do you-- Did you have a h- You just did SD3. Did you have a hypothesis like that you were exploring of like-
- ABAndreas Blattmann
I mean-
- SWswyx
... what you would do differently?
- ABAndreas Blattmann
We, we, we've obviously gathered a lot of, lot of, uh-
- SWswyx
Yeah
- ABAndreas Blattmann
... knowledge, like in these entire times, even, even before, uh, Stability as well, right? And that all culminated, and then we also had a team that, that just like was in one place basically- That was very focused, but which, which had never been the case before-
- DPDustin Podell
Yeah
- 50:36 – 51:43
Deepgram sponsor break
- SWswyx
Uh, we've been talking about how we can generate images and video and improve the perceptual understanding of models. But what about voice? And that's the subject of today's ad. Uh, we've been working with Deepgram for a while for on the AI side, and they recently acquired Pipecat, which has also been a big sponsor of AI. So Deepgram is becoming one of the critical pieces of voice infrastructure for all agents. So this episode is sponsored by Deepgram. If you're building a voice agent, you need it to keep up with how people actually talk. That means understanding words and context and keeping latency low enough for a conversation to flow. That's what Deepgram's speech models and real-time infrastructure are built for. One in 5,000 organizations uses speech-to-text, text-to-speech, and voice agent APIs in production. The focus is accuracy, low latency, and cost efficiency, so you can build and run voice experiences at scale. You can use the cloud APIs or deploy it yourself, including on premises. And there's an offer on Flux text-to-speech. Spend a dollar, get a dollar in credits, up to $500. To sign up for free or find out more, go to deepgram.com/contextiseverything. Deepgram, context is everything. Thanks to Deepgram for bringing us this episode, and now we'll go
- 51:43 – 54:58
Early use cases for FLUX
- SWswyx
back to the interview.
- DPDustin Podell
Well, I, I-- and like what use cases were you seeing, right? Because I think a lot of people went straight to like Hollywood, like we-- you know, that's our goal. Or like, uh, Imad used to say, "We will make Game of Thrones Season 8 again." [laughs]
- ABAndreas Blattmann
Yeah.
- DPDustin Podell
I mean, I, I'm trying to remember this. I kind of-- I don't know if I'll be able to remember the name of this company, but th-this, this might have even been before SDXL, um, but it was a company that for like a week or two, it went viral where anyone could take like a picture of themselves, and then it-
- ABAndreas Blattmann
Yeah
- DPDustin Podell
... like created a-
- ABAndreas Blattmann
Uh, the older version of SDXL, right?
- DPDustin Podell
The older version, and it used-
- ABAndreas Blattmann
Yeah
- DPDustin Podell
... like the old Dream Booth style fine-tuning, which was extremely inefficient. And then, uh, it-- but it w- it like blew up. Like everyone-
- ABAndreas Blattmann
Yes
- DPDustin Podell
... was using this. Everyone was excited about it. The models could make so many things, and we saw that there was a lot of potential in how everyone wanted to use it. And, you know, we, we saw where the models were going to... You know, at the time they couldn't do typography. We knew obviously, okay, typography is gonna be important. And we, we eventually got there with SD3 and, and then so on. So there wasn't the-- just one use case. It was we saw that this was like a general technology that had a lot of potential and that was going to continue to grow and have potential in the future, which now is expanding into, you know, the, the physical world a lot more.
- ABAndreas Blattmann
And-
- DPDustin Podell
But, but yeah
- ABAndreas Blattmann
... th-this was actually already the potential we saw that like-
- DPDustin Podell
Yeah. Yeah, absolutely.
- ABAndreas Blattmann
BFL was not founded to be an image generation lab. It was more of a kind of a matter of the, the, the Flux model being so good that we then capitalized on it as well-
- DPDustin Podell
Yeah
- ABAndreas Blattmann
... because it gave us a kickstart. But when we pitched the company to investors for the C series, it was clear we wanna be working on video because we see that as a-
- DPDustin Podell
Yeah
- ABAndreas Blattmann
... I think we shared this on a podcast actually back in 2023 already, when we were still at Stability, we were saying, okay, this-- like video modeling is actually the central means to understanding how our world basically works. Because, again, what I said earlier, right, essentially we humans are also first observing the world, then starting to interact with it, and by that we, we learn so much. So that was always the thesis. And then obviously we had to kind of take this detour, which was very successful, I would say, to, through image generation. And eventually-- and, and then-
- DPDustin Podell
Mm-hmm
- ABAndreas Blattmann
... you always like face the, the, the kind of the issue of, okay, where do I spend my time on? That-
- DPDustin Podell
Yeah
- ABAndreas Blattmann
... that's unfortunately a, a thing we, we, we, we cannot get across, that the day only has 24 hours, unfortunately. But eventually I think Flux.3 is really like where we saw this going for a long time and what-
- DPDustin Podell
Mm-hmm
- ABAndreas Blattmann
... we wanted to be doing for a long time. And that's why we're so excited about like having the model out there, being able to create videos, being able to create images, being able to create audio, being able to act as a robot brain to actually kind of work in real manufacturing workflows. And there's, there's a lot of stuff coming this year still that we, we have-
- DPDustin Podell
Mm
- ABAndreas Blattmann
... to announce on, on, on, on the physical AI side.
- DPDustin Podell
Mm-hmm.
- 54:58 – 1:03:15
Editing with Kontext
- AFAlessio Fanelli
Um-
- DPDustin Podell
FLUX reaction?
- AFAlessio Fanelli
FLUX reaction. Yeah
- DPDustin Podell
Yeah, before we get there-
- AFAlessio Fanelli
Okay
- DPDustin Podell
... there's one more I want to cover-
- AFAlessio Fanelli
Yeah, yeah. Go for it
- DPDustin Podell
... which is image editing in context. Sure.
- AFAlessio Fanelli
Yeah, yeah.
- DPDustin Podell
Uh, we've okay. Uh-
- AFAlessio Fanelli
Also, really nice, uh, history right here on the website.
- DPDustin Podell
Yeah. I, I, I was actually- Like that is- It's, it's weird we're now at the stage of company that like things just happen, and I d- I'm like, "Oh, we have this." Um, but I was, I was kind of sad FLUX 1 wasn't on there to view, but it's like the original one.
- AFAlessio Fanelli
Right. Yeah. [laughs]
- DPDustin Podell
Yeah. Yeah, but it's, it's fine. It's, uh- We know it's pro. Pro is good
- AFAlessio Fanelli
No, but I, I think like even just the visuals here, like maybe someone will clip it of original Stable Diffusion to SDXL to all the papers-
- DPDustin Podell
You see the progression, right?
- AFAlessio Fanelli
You see the progression.
- DPDustin Podell
Yeah, yeah.
- AFAlessio Fanelli
But one thing that I do want to cover at some point, I no longer personally see the progression, right? You show me state-of-the-art image video models. A lot of Chinese labs are putting them out. It's hard to eval. I know people care. I've talked to Sumit at BFL about, okay, no, actually, people in the industry aggressively A/B test industry. There's, there's a lot that people care about on evals, but there's also a level of, it all looks very realistic to me. Um-
- DPDustin Podell
[laughs] The rest is prompting skill issue.
- AFAlessio Fanelli
I think there's a-- Yeah. [laughs] I think there's a lot of little nitpicks. Like there were hands, there are still issues in text sometimes. So-
- DPDustin Podell
There's still hands issues?
- AFAlessio Fanelli
No, there were hands issues.
- DPDustin Podell
Yeah.
- AFAlessio Fanelli
There, there's issues in perception and layout, like I'm redesigning my living room. It's horrible.
- DPDustin Podell
Controllability is a huge thing.
- AFAlessio Fanelli
Controllability.
- DPDustin Podell
I think controllability is the, the thing we need to solve, like large scale. There's so many ways you can control these models, and maybe that, that is a good seg into, into, uh- Kontext ... FLUX Kontext as well-
- AFAlessio Fanelli
Yeah
- DPDustin Podell
... because that, that was a pain point that we saw for a lot of time. People were s- training so many LoRAs for-
- 1:03:15 – 1:07:32
Training data
- ABAndreas Blattmann
to, to, to understand basically the world better, right? This, this is, I think what, what's, uh, as said, what has always been at the core of what we w- we're doing.
- SWswyx
Yeah. Okay.
- ABAndreas Blattmann
Yeah.
- SWswyx
Well, before we leave the- the sort of data discu- or IP discussion-
- DPDustin Podell
Yes
- SWswyx
... is there a particular kind of data that is, uh, everyone in an industry knows is, like, valuable? Uh, if people, if- if I have a data vendor listening, how do they sell to you? What, what... How do they approach you?
- ABAndreas Blattmann
It depends heavily on what you, what type of problem you want to be solving. That's just super important. If I, if I train a, uh, a robot, I obviously need a lot of eccentric data. I think that's not a secret, right?
- SWswyx
Yeah.
- ABAndreas Blattmann
Because that, that is just how this thing will be, will be basically-
- SWswyx
E- egocentric or...
- ABAndreas Blattmann
Yeah.
- DPDustin Podell
I mean, there's also a benefit for you guys, right? On the language model side, there's a lot of people doing RL environments. Is there a parallel that people should come to you-
- SWswyx
I'm trying to get you vendors
- DPDustin Podell
... do that? Yeah.
- SWswyx
Yeah. [laughs] Just like get you the-
- ABAndreas Blattmann
Oh, we have so many data vendors actually that are, that approach-
- SWswyx
Too many?
- ABAndreas Blattmann
Yeah, yeah.
- DPDustin Podell
Okay.
- ABAndreas Blattmann
It's, it's a lot of them.
- SWswyx
Oh, okay. What, what, what is good versus bad?
- ABAndreas Blattmann
It, it, again, depends on which stage of the training you are-
- SWswyx
Yeah
- ABAndreas Blattmann
... um, and what your effective goal is. I think there- there's a, there's a trade-off between we had this already, right, scale of the data versus, um, versus the quality.
- SWswyx
Yeah.
- ABAndreas Blattmann
And that is what you have to, for every training run, like large scale pre-training run you do again, you have to kind of find, find that sweet spot out, because depending on the architecture you, you, you might be using, uh, that might look very different again, and that is one of the central pre-training experiments like or pre-scaling experiments that you need to do. You need to do a ladder basically, um, to understand at scale how much da- or to, to predict at scale how much data do I actually need and h- how should the quality look like. So where is the sweet spot between like these two dimensions of data quantity versus data quality?
- SWswyx
Mm-hmm.
- DPDustin Podell
Any other major jumps that we wanna call out? So Flux.1 Kontext was a big one. Flux.2, Flux Tools, Action we talked a bit about. Any other-
- SWswyx
Yeah
- DPDustin Podell
... major-
- 1:07:32 – 1:13:12
Testing an edit
- ABAndreas Blattmann
can actually-
- AFAlessio Fanelli
[laughs]
- ABAndreas Blattmann
... kind of process in, in, in a two minutes clip or something.
- DPDustin Podell
This also gets way... Like, coming back to the... I know you mentioned evals earlier. Building evals for larger, like, contextual models-
- ABAndreas Blattmann
Yeah
- DPDustin Podell
... becomes a, a much bigger process.
- ABAndreas Blattmann
Yeah.
- DPDustin Podell
Like, even for-- Like, we, we ended up releasing a, uh, an open eval set for, um... I think it was Context Bench, which was built off the internal bench that we use to evaluate context. And then as you scale up how many references-
- AFAlessio Fanelli
Oh, I haven't seen this. Okay.
- DPDustin Podell
Yeah, yeah. Please, please-
- AFAlessio Fanelli
I'll pull it up
- DPDustin Podell
... have a look. Con- This was a lot of work, but it all, kind of a l- it was actually quite fun to put this together.
- AFAlessio Fanelli
Make the cat very fat.
- DPDustin Podell
Yeah. I mean, it was... We lit- I remember sitting in the back room of our SF office with, like, a few of the other guys. Like, me and Tim were l- uh, were laughing. Tim Dawkhorn is another one of the great, great guys I work with.
- AFAlessio Fanelli
I feel like, I feel like, do you just, like, take some edibles and just go like...
- DPDustin Podell
[laughs]
- ABAndreas Blattmann
[laughs]
- DPDustin Podell
D- I mean, we just-
- AFAlessio Fanelli
[laughs]
- DPDustin Podell
We're sit there late-
- ABAndreas Blattmann
What's the process?
- DPDustin Podell
We're sitting there late at night literally just typing in the s- like, we're just pulling images and typing in the stupidest things.
- AFAlessio Fanelli
[laughs] I mean, it works, right?
- DPDustin Podell
Yeah, that, that like... It... But it's, like, real. Like, I don't know. So because, because-
- ABAndreas Blattmann
And in the end you always, you need to kind of make one step to make it a kind of-
- DPDustin Podell
And wait, wait
- ABAndreas Blattmann
... enjoy the comedy. Yeah
- DPDustin Podell
... what is, what is ground truth? You, you, you need to make sure that you-
- ABAndreas Blattmann
We have a ground truth somewhere?
- DPDustin Podell
Yeah.
- 1:13:12 – 1:20:50
Autoregression
- ABAndreas Blattmann
One more part of the text tree, and then we want to go into robotics in action.
- DPDustin Podell
Mm-hmm.
- ABAndreas Blattmann
Um, autoregressive-
- AFAlessio Fanelli
Yes
- ABAndreas Blattmann
... image gen. It seems like a big trend with both Gemini and ChatGPT doing it.
- AFAlessio Fanelli
Yeah.
- ABAndreas Blattmann
Uh, like-
- DPDustin Podell
Is that as big an impact in, in your mind as it is in ours? You know what I mean? Like, I-I, it seemed like a very big change because then you could conversationally edit. It is the, like the next obvious step from conte- uh, I-FLUX Kontext, where instead of editing with a image and a, and, and text prompt, you're conversationally working with a, with a model.
- ABAndreas Blattmann
That I would say is not a, a feature of autoregressive models-
- DPDustin Podell
Okay. Mm-hmm
- ABAndreas Blattmann
... uh, only. It's just based on the data you train on it. And like you, you can get language capabilities into a diffusion model as well, right?
- DPDustin Podell
Yes.
- ABAndreas Blattmann
Which we, which we know is, is, uh, working. And, um, I said in, in long term, this is where we believe this will, this will also go into and where we're working towards to also get more modalities in, including this language model, right?
- DPDustin Podell
Mm-hmm.
- ABAndreas Blattmann
So it's not, it's not, it's not a question of the, of the, of the algorithm there. Um, it's just a question of like, what do you focus on, basically. I think both approaches are extremely exciting and have a lot of strengths and a lot of like shortcomings. I-I would say autoregressive models are way more data efficient. You can train them way better in data because every... Like you have the-- for, for diffusion models, you have this trajectory from data to noise, right? And every training example you're training on is effectively, because you have this continuous path from data to noise in between, it's giving you ri-rise to infinitely many training examples. Because you can pick every possible spot to denoise from, from this trajectory between data and noise for one example. Obviously, the model would overfit if you would only train on, on, on this one, uh, example. Theoretically, it's just like one training example is giving or one data point is giving rise to infinitely many training examples. Whereas autoregressive models can just train on all the tokens in parallel, basically, and get the maximum gradient from each training example. That's the training side. So there I would say autoregressive models are more data efficient. However, at the sampling, at sampling time, autoregressive models have to-- don't get across that. They have to autoregressively create one token after the other-
- DPDustin Podell
Yes
- ABAndreas Blattmann
... because they're iterative in this na- like, like data direction, right? And diffusion models, you can just go down and distill fifty steps down to four, and by that you can speed that up so much, which is on the autoregressive side just not possible because you have to autoregressively generate token after token after token. So these are just two, two, um-
- DPDustin Podell
Well, you, you make it sound like it's just worse then.
- ABAndreas Blattmann
So, so-
- AFAlessio Fanelli
Yeah
- ABAndreas Blattmann
... autoregressive generation is in that respect, it has a limitation, but on the other side, diffusion has in, in the terms of data efficiency or training, has also a limitation. So we, we actually, I think it's-- I, I like both approaches. We've also been working on autoregressive models like-
- DPDustin Podell
Mm-hmm
- ABAndreas Blattmann
... long, long, long back, and both are super exciting, I would say.
- AFAlessio Fanelli
So on, uh, data efficiency, autoregressive wins out though?
- ABAndreas Blattmann
Yeah.
- AFAlessio Fanelli
Okay. So scale. Um, I guess one of the other big things that you see, having talked to people that worked on both like ChatGPT and Imagen, is how you can add in the loop RL like agentically. So models can do a-agentically, you know, generate something, reason on it, edit the image, and continue that workflow.
- ABAndreas Blattmann
Yeah, they can do that too. Yeah.
- AFAlessio Fanelli
And this is, but this is separate than pipelining in diffusion generation, right? It's all the one same model doing it, where as you train that in on a data level, you know, you can scale that. Um, I guess, is there a counter to this in diffusion, or?
- DPDustin Podell
There's no inherent limitation to, that you couldn't have a model that is diffusion-based-
- AFAlessio Fanelli
Yeah
- 1:20:50 – 1:30:08
World action models
- ABAndreas Blattmann
yeah.
- SWswyx
So Flux Action, this is coming to very, very recent launches and, and work that you're doing. How do you approach this? What is your-- what is the feedback and impact so far?
- ABAndreas Blattmann
Yeah, it's actually like how, how we approach it. Basically you, you pre-train on, again, videos, images, audio, everything together, and then you have this very general model that understands the world and then it actually-- you can, you can take a bit of action prediction data. Basically you add kind of predictions in the state space of this robot, which is basically a kind of a parameterization of here is this robot arm, um, which every joint will be described by a, a translation and a rotation, six degrees of freedom basically. And then you predict that together with a video. And since the model can already reason about the video and what it needs to create or to, to fulfill a certain task in video space, it's actually very simple to then add this new dimensionality or this, this, this new dimension, this state space how we call it, and then basically predict that as well given a chunk of earlier video, condition it on that and an instruction. Basically the model creates the next chunk of actions to basically fulfill-
- SWswyx
Mm
- ABAndreas Blattmann
... um, this action and this, this works extremely good. It's-- We, we can do it in the real world like for real world robots. We can do it in a simulation like computer use is also kind of a mapping of that to the digital world and we can like we-
- SWswyx
Ooh
- ABAndreas Blattmann
... can make the model play games right and stuff like that.
- SWswyx
Mm-hmm.
- ABAndreas Blattmann
So this is all enabled by this, this single Flux 3 model and then you see also that it's extremely capable because we have this very general pre-training which is not only limited to say classic robot data-
- SWswyx
Mm
- ABAndreas Blattmann
... but actually all kinds of data that gives a model just this inherent world understanding that you also see then in performance. And being a diffusion model is also pretty speedy as we see here, um, with a, with a-- with these distilled, uh, versions basically, right?
- AFAlessio Fanelli
Would you call this a world model or world action? Do you have a term you like to use? [laughs]
- ABAndreas Blattmann
It's a world-- This, this one is a world action model. I think the, the word, the term world model I don't like to use. It's so overloaded.
- DPDustin Podell
I'm in the same, I'm in the same boat with you. It feels like a hominy to me at this point.
- AFAlessio Fanelli
Hominy.
- ABAndreas Blattmann
Let's, let's call this a visually intelligent model because it can do all these, these basic things, right?
- AFAlessio Fanelli
I guess the, the distinction that I see in these models is you're not just predicting video or world, you're also prediction action states, right?
- ABAndreas Blattmann
Mm-hmm. Yeah.
- AFAlessio Fanelli
So one is what is happening in the world, what is the state, but also what is the next action to predict. So it's like-
- ABAndreas Blattmann
Exactly
- AFAlessio Fanelli
... sort of two outputs, right?
- ABAndreas Blattmann
Mm-hmm.
- AFAlessio Fanelli
So.
- ABAndreas Blattmann
Yeah.
- SWswyx
Is there a target frame rate, uh, that you target that is industry standard?
- ABAndreas Blattmann
So the-
- SWswyx
Because you don't have to predict out that far-
- AFAlessio Fanelli
Mm-hmm
- SWswyx
... is, is my, is my idea.
- ABAndreas Blattmann
You mean in terms of the real-time factor like how, how, how-
- 1:30:08 – 1:32:16
Robotics in 2027
- ABAndreas Blattmann
of robotics.
- AFAlessio Fanelli
Oh.
- DPDustin Podell
Oh, absolutely.
- AFAlessio Fanelli
Okay. Say more on predictions. Where, where do we go from this in 2027? How, how fast do things pick up?
- DPDustin Podell
Single shot learning.
- ABAndreas Blattmann
Yeah. In context learning-
- AFAlessio Fanelli
Is it?
- ABAndreas Blattmann
... is a, is a big, is a big thing we'll see first.
- AFAlessio Fanelli
Is this a-
- ABAndreas Blattmann
We'll see first commercial applications of these things in real manufactories. That's, that's why it's actually good that we have a, this German base because there's a ton of, uh, industry data-
- AFAlessio Fanelli
Mm
- ABAndreas Blattmann
... and, um, obviously lots of potential for partnerships. People are really hungry for robotics, I would also say.
- AFAlessio Fanelli
Yep.
- ABAndreas Blattmann
So there's a, there's a good bunch of, um, of synergies that we wanna be using to actually make 2027 the year of robotics.
- DPDustin Podell
I, I'm very excited about it. The research, the research on this is also moving very quickly, so I-
- ABAndreas Blattmann
Yeah
- DPDustin Podell
... I suspect that robotics will improve very quickly. Yeah.
- AFAlessio Fanelli
What about the side of, like, robotics is very physical, manufacturing constraints. Like, you can scale up tech, you can scale up models, but that's-- It's pretty hard to affect, right? So-
- ABAndreas Blattmann
Mm.
- AFAlessio Fanelli
Predictions 2027, where, where, where, where are we realistically on robotics? Can I buy something off the shelf that can single shot learn? Like...
- DPDustin Podell
I suspect, like, later 2027, like, people will be able to get some, some quite interesting robotics in their house. I mean, I... Whether it's, you know-- Is it gonna be able to do, like, fine motor tasks and all this? I, I don't know. But I suspect, like-- And again, this is just entirely predictive. I've, I have no baseline for this. But I, I suspect next year There might be some, some interesting like, you know, at least niche for like the, the tech side of people like us. Not like, you know, uh, the average person would be picking this up, but some interesting potentially useful robotics in your house.
- ABAndreas Blattmann
And for manufacturing-
- DPDustin Podell
Mm
- ABAndreas Blattmann
... I think 2027 will be, uh, a year-- Because manufacturing-
- DPDustin Podell
Yeah, absolutely. Yeah
- ABAndreas Blattmann
... is, like, more restricted, right, than, than kind of... It, it, it isn't that long tail less like, say, autonomous driving and stuff like that, right? Or, or like consumer robotics where, where you can just like do any- You would ideally like to do this thing to do anything you want. But for manufacturing, I think there's a very good chance that we will see like a lot of applications actually being unlocked by Robotics in 2027.
- DPDustin Podell
Very much so.
- ABAndreas Blattmann
And that's what we are also working towards, basically.
- DPDustin Podell
Very much so.
- AFAlessio Fanelli
Awesome. I mean, exciting. It seems like very fun place. You guys have robot labs [laughs] image video models.
- 1:32:16 – 1:38:23
Who uses FLUX?
- ABAndreas Blattmann
future.
- DPDustin Podell
You're also-
- AFAlessio Fanelli
You-
- ABAndreas Blattmann
You're just missing the flying car.
- AFAlessio Fanelli
You mentioned Germany a lot-
- ABAndreas Blattmann
That is true
- AFAlessio Fanelli
... but you also have a pretty big SF office. I actually didn't know there was a big European side. There's a substantial SF presence.
- ABAndreas Blattmann
Yeah, it's called Black Forest Labs.
- DPDustin Podell
Yeah. [laughs]
- AFAlessio Fanelli
And this-
- ABAndreas Blattmann
And so I like inviting everyone to come over to our place. Uh-
- AFAlessio Fanelli
Yeah
- ABAndreas Blattmann
... 410 Townsend Street. Uh-
- DPDustin Podell
[laughs]
- ABAndreas Blattmann
Super, super nice.
- DPDustin Podell
Yeah, come, come by.
- ABAndreas Blattmann
And we, we, we, we're not a European company, not in-- Like-
- AFAlessio Fanelli
Yeah
- ABAndreas Blattmann
... we're a company with roots both in Europe and the US.
- DPDustin Podell
Absolutely.
- ABAndreas Blattmann
And we are acting globally, basically.
- AFAlessio Fanelli
That's fun. We got you on because you're visiting in town, and we're like, "Okay, we should-
- ABAndreas Blattmann
Yes
- AFAlessio Fanelli
... have you on."
- DPDustin Podell
Yeah.
- AFAlessio Fanelli
How can people help you?
- ABAndreas Blattmann
Yeah.
- AFAlessio Fanelli
You hiring?
- DPDustin Podell
Yeah.
- AFAlessio Fanelli
Looking for-
Episode duration: 1:38:23
In the uListen app, the transcript lights up word by word as you listen. Get uListen
Transcript of episode 8cGfJOU3AlQ