Runway’s Bet Beyond Video: World Models, Robotics, and the Neural OS — Anastasis Germanidis

1. 0:00 – 1:44

   Hook

   1. AGAnastasis Germanidis0:00

      Effectively, the, the end game of something like Interface World Models is you have a, a fully neural operating system. So I think, uh, uh, Andrej Karpathy has written about that qui-quite a while back. But it's, uh-- You know, I think to me, it's, it's a bit, um, it's a bit odd that, you know, we have, for example, with an interaction with, with, with an LM of today, you have this LM that can basically talk, talk to you about anything. It can-- You can take the conversation any direction. You can, you can-- It's very general, so it can solve all those different tasks, but you interact with it through a very rigid interface. And so to me, it's just a matter of time before the interface itself becomes learnable and becomes, you know, part of the, the, the whole loop of like you're not just delivering, you're delivering an application end-to-end, and that means you're delivering the language model, but you're also delivering the, the render and the pixels and, and, and that's also a learnable component.

   2. SWswyx0:59

      Before we get into today's episode, I just have a small message for listeners. Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content. We've been approached by sponsors on an almost daily basis, but fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way. But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring The In Space to you each and every week. If you do it, I promise you we'll never stop working to make this show even better. Now let's get

2. 1:44 – 7:01

   Introduction

   1. SWswyx1:44

      into it. [on-hold music] Okay, we're here with, uh, Anastasis from Runway, with, uh, me and Vibhu in the studio. Welcome. [laughs]

   2. AGAnastasis Germanidis1:52

      Good to be here.

   3. SWswyx1:53

      Congrats on all your su-success and progress with Runway. You're opening offices all over the world. Did you envision this when you first started out?

   4. AGAnastasis Germanidis2:00

      Uh, not quite. I think, I think even when we started, we thought-- we had this idea that, you know, it was more a matter of when, not if. Uh, we were seeing the early generative models of twenty sixteen, twenty seventeen, and just extrapolating, assuming, you know, we-- resolution quality increases predictably over time. There's gonna be a point where most of content will be generated, um, and that was maybe the initial, the initial thesis of Runway was we will need, as a result of those generative models, rethink how creative tools are made. Um, and as we built out the research behind, um, our generative models, it then became clear that they were useful far beyond that as well.

   5. SWswyx2:41

      And it is more obvious now with like the real-world stuff and the world models that we'll talk about later. I-I'm just kinda curious how you go from a background in like ZocDoc and [laughs] you know, computer vision into Runway. Like take us back to the early conversations with Chris and, you know, whoever else is on your founding team.

   6. AGAnastasis Germanidis2:58

      Uh, I was always split into those two worlds. One was the-- You know, I had my own art practice. I was making a lot of interactive art, um, I think for a long time. Uh, and, and then on the other side, um, I was working in startups, and I was working as an ML engineer, as a backend engineer at different, different companies. I've always been interested in, uh, coding and computation and especially interested in simulation and brought it back into my early artwork as well. And at the same time, I was interested in-

   7. SWswyx3:27

      His-- The personal site has, has a few, right?

   8. AGAnastasis Germanidis3:29

      Um, yeah.

   9. SWswyx3:29

      Is there, is there one that we should pull up? Uh, just, just in case there's something that's like... I just like to go down memory lane.

   10. AGAnastasis Germanidis3:35

       Yeah, sure.

   11. SWswyx3:35

       Okay, what is this?

   12. AGAnastasis Germanidis3:36

       So this was, uh, a project that I made I think back in twenty fifteen where I, I, I built this software that would give, uh, voice instructions to people in a gallery space. So it would basically coordinate interactions between people. And so it will first give you an identity, like you're an, uh, architect, you're thirty years old, and, uh, you like sports. Um, and then it would match you with another person, and you have this completely generated interaction. Obviously, language models were not quite there at the time. Um, and so it was ma-- it was a mix of some templates and some, like, some Markov chain generated text. Uh, and it would just completely simulate these small talk conversations between, uh, everyone in the, in the gallery space. Um, so was always very fascinated on the, on the one end with, uh, generative models and like the early machine learning work that was being out at the time, but at the same time, there was a separate thread of simulation and what it means-- like, what can we learn about humans by creating those very simple models of their interactions and their behavior?

   13. VIVibhu4:41

       Did you generate the prompts or, you know, the thirty-year-old, whatever? Was it you generating them? How'd you, how'd you do it?

   14. AGAnastasis Germanidis4:47

       Exactly. So, so the program would just generate-

   15. VIVibhu4:49

       Oh, okay

   16. AGAnastasis Germanidis4:49

       ... those, uh, from... Yeah, a l- a lot of it would be kind of mad lib style of just-

   17. VIVibhu4:54

       Yes

   18. AGAnastasis Germanidis4:54

       ... you know, you have lists of different professions, lists of different, uh-

   19. VIVibhu4:58

       Hobbies [laughs]

   20. AGAnastasis Germanidis4:59

       ... you know, personality types, lists of different, uh, ages, things like that, and then it would just combine those things together. And then maybe the next project we go is, uh, Uncanny Valley, Uncanny Road, which was-

   21. VIVibhu5:11

       Gants

   22. AGAnastasis Germanidis5:11

       ... one of the first projects that, uh, we built with, uh, with, uh, one of my two co-founders, Chris. This was taking, um, pix2pix HD, which was one of the early image-to-image models that NVIDIA released back in twenty sixteen or twenty sev-seventeen. Uh, and it was a model that would take a semantic map of a scene and then generate a photorealistic, let's call it, uh, output. Obviously, very, very early days, so it was not very high-fidelity outputs, but it w-- I think it was the first image generation model that would generate at, uh, at one K resolution. And it was all trained on self-driving Data sets. So the semantic categories it would support were only, you know, things you would encounter on the road. So it would be pedestrians, traffic signs-

   23. VIVibhu6:01

       Stoplights

   24. AGAnastasis Germanidis6:01

       ... you know, like, uh, bikes, stoplights. And so that was, that was actually one of our first indications that we built this and people were making all this, like, very surreal imagery of, yeah, uh, a million pas- a, um, a million pedestrians or a million traffic signs or like gigantic humans. And it was a indication that you could take a model that was trained on this very boring data set essentially of like not that many interesting things happen when you're on the road, and then you can repurpose it and go very out of distribution and make something that was artistically compelling. And that was -- it's a summary of the thesis of Runway in some ways, that you can take the same generative models and if you look at them from another direction, if you build interesting tools around them and you give them to artists, they're gonna do things that you don't expect.

   25. VIVibhu6:47

       Very cool. I like the, you know, UX of it basically. You're just given an empty canvas, drag whatever, do whatever. And then the other one, like you see everyone with wired headphones, you know? Like that's, that's a sign that it's, it's very easy-

   26. AGAnastasis Germanidis7:00

       The Apple-

3. 7:01 – 14:07

   Runway’s Origins and the Bet on Generative Video

   1. VIVibhu7:01

      The Apple earbuds

   2. AGAnastasis Germanidis7:01

      ... original ads. Yeah.

   3. VIVibhu7:03

      Yeah, yeah, yeah. Take us to today. You've been doing this for seven years at Runway. How have we got to this? You know, like [laughs] how did we go from driving simulator data to all this? And you, you kind of cover the whole stack of generative media, you know?

   4. AGAnastasis Germanidis7:18

      I mean, interestingly, we're almost back in, uh, you know, we're, we're full circle where [laughs] we're now applying our models and kind of beyond creative tools into real world scenarios. But it was a, it was a long journey. It was very early on we realized the, yeah, the first version of Runway was a way to easily use the, all the open source models that- of the day, things like pix to pix to, and give them to artists. That was, that was the initial idea, is those models are too difficult to use if you're not a machine learning engineer. Like what happens when you give them to artists? Very quickly, we realized we needed to build a research org, uh, inside of Runway, and that happened maybe on year one. And, uh, a lot of the mandate there was the image generation models of the time, the video generation models of the time, or there were barely any video generations model at the time, but they were not quite there where they could be productionized and brought into tools that would be part of creative workflows. Um, so we need to push the frontier of the research. And so maybe the, the first four years of Runway research was almost happening on the background until there was a moment in 2022, uh, with latent diffusion, with, uh, DALL-E 2, where, you know, there was that step function change and the, you guys maybe remember [laughs] around that time.

   5. VIVibhu8:33

      I started in, in space because of basically latent diffusion and Stable Diffusion.

   6. AGAnastasis Germanidis8:38

      Yeah.

   7. VIVibhu8:39

      Uh, because I was like, "Wow, this is not only like feasible, it is actually doable on consumer hardware."

   8. AGAnastasis Germanidis8:45

      Exactly. Yeah. [laughs]

   9. VIVibhu8:46

      I think the delta is also huge. Like I learned pix to pix, like this was intro to ML.

   10. AGAnastasis Germanidis8:51

       [laughs]

   11. VIVibhu8:51

       The TensorFlow, like Jupyter, Google Colab notebooks are like this, and then you have a sudden step function change, you know, with diffusion and whatnot. Any other one sense that, like there were clear examples of what early diffusion were to get to here. Any other changes in key technology, uh, research?

   12. AGAnastasis Germanidis9:10

       Uh, between, uh, 2018 when we started in 2022.

   13. VIVibhu9:13

       Yeah.

   14. AGAnastasis Germanidis9:14

       Um, so one of the early work that we did in Runway was solving segmentation. You mentioned video segmentation. It was a very important problem because most VFX involves essentially separating-

   15. VIVibhu9:26

       Rotoscoping

   16. AGAnastasis Germanidis9:26

       ... subjects. Yeah, rotoscoping. Extremely manual process. Nobody, nobody enjoys doing that. Uh, and so a lot of the early days of Runway was building this tool that was called Green Screen, and it was for a long time the main thing that people were using Runway for. It ended up being used in, uh, Everything Everywhere All at Once and a bunch of other kind of high visibility films and series. But that was essentially Runway for a long time was a post-production tool until latent diffusion, generative Gen-1, Gen-2, um, happened.

   17. VIVibhu9:56

       Cool. I mean, let's, let's go past that moment. You've come a long way. Then you started releasing your own models. Maybe describe that journey as well.

   18. AGAnastasis Germanidis10:05

       Yeah. So we go to at the point, yeah, in kind of mid-2022 when it became clear that we're doing research at a fairly, uh, fairly small scale of, of compute, and it became clear that like scaling laws would apply to, um, image and video gen in the same way that we're applying to language generation. So we made a big bet, and I, I think at-- So at the time we, we signed this deal to build a cluster of 1,000 A100s, which at the time we were a Series B startup. That was a almost, you know, slightly irrational decision maybe. But we really believed that if we trained a video model at the large scale, um, we would, we would get like a, a great model at the end. And at the time, the goal, you know, the goal or we set the goal around fall of 2022 of what is, what is the latent diffusion, Stable Diffusion moment look like for video? And at the time, the, the best model of the time was called CogVideo. Uh, it was one of the early kind of video models, was very 256 by 256 resolution, very, not, not very high quality. Uh, and so we decided we're gonna build out this cluster and we're gonna just invest in like, uh, in, in building out our, uh, our own video model. Uh, it became clear as, as we're training Gen-1 that it was, it was difficult to get to fully... Uh, we wanted to build text to video, um, but it, it, it became clear to us that an easier starting point would be to, to start from video to video because when you have a stronger conditioning, it's, it's a basically an easier problem to restylize an existing video versus generate a video from scratch. And so we released Gen-1 first back in, you know, it was January of, uh, 2023. Yeah.

   19. VIVibhu11:48

       It's just a fun visual podcast honestly. Like you, we can see February 2023 was the state of stuff.

   20. AGAnastasis Germanidis11:54

       I mean, it, it's so interesting 'cause at the time when you see those results, you think this is, this is so incredible and this is [chuckles] like it's almost like image generation or video generation is solved, and then you look back a few years after and it's like it's obviously... [chuckles] It's just like you get used to results very quickly, uh, with those models. But at, at the time when we started seeing those results, it was, it was, you know, it, it, it felt quite, quite incredible, uh, and the level of like quality you could get. And, um, so the Gen-1 was a depth-conditioned video model, so it would turn-- It would, it would take an input video, uh, it would predict, uh, it would, it would con- We would first convert it into the depth map, and then we would generate, uh, pixels with a, with a light and diffusion model.

   21. SWswyx12:45

       Yeah. Very effective.

   22. VIVibhu12:46

       Yeah. I didn't realize how distracting the blog post would be. Sorry. [laughs]

   23. AGAnastasis Germanidis12:49

       Yeah. But, uh, one of my favorite examples actually of the... on, on those, on, on Gen-1 was both, uh, if you, if you go up to Mode 3 or Mode 2, there was this storyboard use case where people would make-- would basically, um-

   24. VIVibhu13:04

       You can mess around with the-

   25. AGAnastasis Germanidis13:05

       May make a city out of books or out of boxes, and then they would, they would kind of shoot a video with their phone and then translate it into a-- the photo-photorealistic output. There was all these ways in which those models were starting to be used for storyboarding and also for really, uh... And then for if, if you go to Mode 4, like, of taking kind of untextured 3D scenes and then turning them into photorealistic output. So we saw a lot of use cases early on where people that were familiar, you know, were power VFX editors would just take a, a blender, uh, a render, and then they would get translated in with Gen-1, or create a scene in Unity and then take a-- capture a video of it and then, and then translate into, you know, restylize it. So I still, I still think video-to-video is powerful. I think we, we had a, a recent video-to-video model as well, and it's one of my favorite ways of using, using those models, is essentially using them to, to u-use ground truth video as like the initial inspiration and then translate into, into different styles or different

4. 14:07 – 20:28

   The Stable Diffusion Story

   1. AGAnastasis Germanidis14:07

      outputs.

   2. SWswyx14:08

      But I think we're gonna go into like the rest of Runway and catch people up to speed today. I did wanna cover the, let's call it the s- the Stable Diffusion controversy or, you know, what, what happened with Stability AI, what-whatever. You know, I think there was like sort of two sides of the story. I think there's part of that is a normal thing of like people, uh, you know, join and leave companies. But what is the, you know, retrospective now that, you know, there's been some years behind it?

   3. AGAnastasis Germanidis14:33

      Yeah, it's a very, it's a very long story- [laughter] ... to, to go, to go into. I think-

   4. SWswyx14:38

      Which I remember you actually wrote a really long post about.

   5. AGAnastasis Germanidis14:40

      We would, we would probably cover the whole hour to- [laughter] ... to, to, to, to go into it in, uh, in more detail. But, uh, essentially, you know, there was the latent diffusion paper that came in, um, I think that was at the, at the end of, uh, twenty twenty-one. And then Patrick Esser, who was one, one of the researchers behind, uh, latent diffusion, and he worked at Runway at the time. He built latent diffusion collaboration with Robin Rumbach and a few other folks back, uh, in the co-- in, at Convys, uh, which was, uh, uh, a lab determined-

   6. SWswyx15:10

      Like a research group, yeah.

   7. AGAnastasis Germanidis15:12

      And, uh, after releasing the early latent diffusion model, they, um, e-essentially there were, you know-- The goal was to keep working on versions of the model, kind of scale it up, uh, incorporate new data, incorporate new tasks. And Stable Diffusion was basically the same model but trained on more compute and then with a few more tricks like a classifier-free guidance paper came at some point, I think at the early twenty twenty-two. And that-

   8. SWswyx15:36

      Which like was a big prompting improvement.

   9. AGAnastasis Germanidis15:39

      Yeah.

   10. SWswyx15:40

       You know.

   11. AGAnastasis Germanidis15:40

       That improved results, uh, was trained on better data, so like the aesthetic subset of, uh, of LAION, but it was effectively, you know, the same, uh, the same underlying architecture. And there was that big training run, uh, that, uh, st- happened on Stability's cluster. Uh, Stability kind of financed, uh, financed that run. And looking back at that story, I think it was the work to build and train that model was, was done. It was a, it was a research project. It was done as part of like continuation of the latent diffusion work. It then-- I think the model became very successful and it, um, I think there were the-- And, and I think as, as a result of its, uh, its success, other companies tried to, uh, figure out the commercialization path for it. But for us, it, it was very important that we try to, you know, we, we, we make sure that we-- It was meant to be an open source research project, and so the-- we decided that we should continue releasing versions of it, uh, since that was kind of the original, the original goal of Stable Diffusion, and that led to releasing Stable Diffusion one point five. There was maybe a day of, uh, a bit of, uh-

   12. SWswyx16:50

       [chuckles]

   13. AGAnastasis Germanidis16:50

       ... miscommunication there, but ultimately that was resolved very quickly within, within hours. Um, so yeah-

   14. SWswyx16:56

       Okay

   15. AGAnastasis Germanidis16:56

       ... there was-

   16. SWswyx16:56

       Not that-

   17. AGAnastasis Germanidis16:57

       A nice-

   18. SWswyx16:57

       I just wanted to, you know-

   19. AGAnastasis Germanidis16:59

       Yeah

   20. SWswyx16:59

       ... you have the-- You're actually one of the main players in, in that sort of journey, and so it's nice to hear from the source of like what happened. Yeah. [laughs]

   21. AGAnastasis Germanidis17:07

       Yeah, I mean, uh, I, I, I think it, it's all, it's all in the past now-

   22. SWswyx17:10

       Yeah

   23. AGAnastasis Germanidis17:10

       ... I would say. Um, and like both companies, you know, Stability took its own path, uh, Runway took its own path.

   24. SWswyx17:16

       Yeah. There's still-- I mean, James Cameron is backing the new Stability, whatever they're doing with the, the Hollywood studios.

   25. AGAnastasis Germanidis17:22

       Right.

   26. SWswyx17:22

       I don't know what they are doing. [laughs] I think one thing that, that impresses me, and I'm happy to move on, is that back in the-- that time, let's say like twenty twenty-one, twenty twenty-two, there was this community of people that you were involved in that was researching all this stuff, right? And like from everyone I talked to who was active then, it seemed like it was fairly obvious that somebody would do the hero training run that would produce Stable Diffusion. So I, I guess the question is, you know, like you had the-- You had-- You were-- You had made investments, you, you, you were-- you had the foresight. Is it accurate to say like that is reflective of like what people were thinking at the time? Or was it still very much like, "Well, we'll use it as like a post-production tool or something. I don't know." You know? [laughs] Like, like where, where, where in the sentiment were we that maybe you can s-sort of think back to, like what, what the community was like back then?

   27. AGAnastasis Germanidis18:12

       I reminisce and I, um, I, I think of very fondly those early years, uh, from like 2018 to 2022 'cause it was a very small community that, as you said, were very convinced that this was gonna be a big thing. And at the time, you know, anyone who-- because it was such a small circle and e-everyone who would like be part of that circle and like make projects with it would, you know, i-immediately kind of get, you know, go viral. [chuckles] Uh, so like I remember that there-

   28. VIVibhu18:39

       And you don't even know who they are, right? They're just some name on a, you know, a GitHub or Hu-Hugging Face somewhere.

   29. AGAnastasis Germanidis18:44

       Exactly.

   30. VIVibhu18:44

       Yeah.
5. 20:28 – 24:46

   Gen-2, Controllability, and the Weekend Hack

   1. VIVibhu20:28

      there to share?

   2. AGAnastasis Germanidis20:28

      We had to build the entire model serving infrastructure for video diffusion models. There was no- [chuckles] n-nothing else, uh, already, like 'cause we had Gen-2 was the first text-to-video model, I think, out, out in the, in the market. So many things that we learned over time. I think the, the-- I think the biggest one was like we, you know, it was very clear early on that text-to-video was not gonna be the, the answer. Like you-- like people wanted a lot more control than that. And so we invested in like controllability on top of those models very quickly. You know, how, how do you use the camera trajectory as control? How do you use an i-initial input frame as control? So, so that was a very early learning for us. With text-to-video was, um, like Gen-2 was an amazing, you know, step function improvement in the quality of video models, but it was u-used much more in an exploratory way because there was nothing to ground it to. There was no reference that you could bring into it. There was no-- You couldn't really control the, the camera motion. You couldn't control the object motion. And so the first year, I think 2023, was really all about what are all the interesting ways in which we can condition those models? And it was a lot of just post-training rounds on top of the base model to figure out like what, uh, how do people actually wanna control them. And so there was like this quick succession of the-- we, uh-- it was called Motion Brush, which was you could like, you could basically draw arrows and dictate where things should move in the scene.

   3. VIVibhu21:53

      That's so cool.

   4. AGAnastasis Germanidis21:53

      There was camera control. That was you could just describe like how you want the camera to move in the scene. And because we work with filmmakers from kind of the most of the history of Runway, we immediately got this feedback and got this, you know, uh, decided that this was worth investing in. And so controllability became a big theme, I think, very early on as we were building, as we were building those models. Something fun that I haven't really talked about too much was just how Gen-2 came to be out of Gen-1. So, so it was a bit strange because we announced Gen-2 two months after Gen-1 and, um-

   5. VIVibhu22:28

      We're accelerating. [laughs]

   6. AGAnastasis Germanidis22:29

      It was, it was before Gen-1 was even generally available. But Gen-Gen-1 was a depth-to-video model, so it would take a depth map and it would convert it into RGB. Um, and we couldn't get, you know, text or image to video to work directly, and that's why we started from depth to video. Um, but, uh, you know, and we had discussions of like, okay, we need to spend the next six months actually investing in text to video, maybe increasing the compute scale or the model scale, like training a larger model. And I, I had this weekend project idea, which was, what if I take a model that, uh, starts from text input and converts to depth maps, and then, and then use Gen-1 to convert the depth maps-

   7. VIVibhu23:12

      [laughs]

   8. AGAnastasis Germanidis23:12

      ...into RGB? Uh, so-

   9. VIVibhu23:14

      It'll probably work

   10. AGAnastasis Germanidis23:14

       ...and so Gen-2 was basically that. And-

   11. VIVibhu23:16

       Oh. [laughs]

   12. AGAnastasis Germanidis23:18

       Um-

   13. VIVibhu23:18

       The hackathon pipeline. The weekend hackathon pipeline.

   14. AGAnastasis Germanidis23:20

       Yeah. [laughs] Um, so-

   15. VIVibhu23:21

       But it looks good. [laughs]

   16. AGAnastasis Germanidis23:23

       And it worked pretty well. I mean, there, there were, you know, if you-- with the knowledge that it has this like two-stage pipeline, you can tell in some cases that the structure of the video looks a bit off because you had to generate the depth first before you go into, into the output video. But it worked, [laughs] and it allowed us to bring, bring this to, to our-- to users very quickly. But it, it, it's actually in-- Now it's interesting 'cause like people are coming back to this almost two-stage approach. Like if you look at the, the ReeVe text-to-image model that came a few months ago, it had this, um, this planar model that would generate bounding boxes before it fed that into a diffusion transformer.

   17. VIVibhu24:04

       Yeah, Ideogram also the same day.

   18. AGAnastasis Germanidis24:06

       Yeah.

   19. VIVibhu24:06

       I, I remember that was very strange that both of them came out the same day with the same exact innovation.

   20. AGAnastasis Germanidis24:11

       It's a small community, I think. [laughs]

   21. VIVibhu24:12

       I'm like, this is like-

   22. AGAnastasis Germanidis24:13

       People

   23. VIVibhu24:13

       ...this is completely coincidental, right? [laughs]

   24. AGAnastasis Germanidis24:16

       Um, pe-people talk. Um-

   25. VIVibhu24:17

       [laughs]

   26. AGAnastasis Germanidis24:19

       So, so yeah, there is, there's definitely something into this approach. And o-obviously now, uh, like every single vid-like video generation model in production uses a, a complex prompt completion pipeline under the hood. I think that's no secret that there is the community-

   27. SWswyx24:32

       Humans are terrible at prompting. [laughs]

   28. AGAnastasis Germanidis24:35

       Um-

   29. VIVibhu24:35

       I think across the board.

   30. AGAnastasis Germanidis24:36

       Yes.
6. 24:46 – 29:47

   From Video Generation to World Models

   1. VIVibhu24:46

      what you would want.

   2. AGAnastasis Germanidis24:47

      Exactly. I, um-- And there was the DALL-E 3 paper beforehand that, uh-

   3. VIVibhu24:51

      Mm-hmm

   4. AGAnastasis Germanidis24:51

      ... was the kind of the, the first public, uh, description of the fact that synthetic captions and really detailed captions work really well. And then Sora kind of built, uh, bu-built on that. Yeah, so it was 2023, we were kind of releasing all these updates to Gen-2, like the camera control, motion brush. And there was actually something very interesting about camera control 'cause it, it was the first time that you felt that instead of like you were creating video, you were creating a short video, you were actually navigating inside the world. And I think camera control was maybe the seed of some of the ideas that we had around world models and really opening up that research direction. We realized, you know, it was this era and this series of, you know, Gen-1 and Gen-2 models really proved to ourselves, yeah, this is, this is the-

   5. SWswyx25:41

      Cool. [laughs]

   6. AGAnastasis Germanidis25:41

      Uh, so this is not the original camera control. This was the updated camera control on top of Gen-3. But yeah, I think it made those models usable to, to filmmakers, uh, I would say. The-- So camera, camera control was very, uh, was, was very popular. And so we realized, you know, there is one way of seeing those models, which is, you know, you're just as a, as content creation machines, and there is the other way, which is you're-- as you're predicting video, in order to predict video well, you need to simulate the world in an increasing and increasing capacity. And if scaling laws apply on video just like they apply on language models, then as we scale the computer we put into those models, then they're gonna be able to simulate physics. They're gonna be able to simulate human actions and dynamics increasingly well and predictably well. That was the, the thesis about around our efforts on world models, and we spin up this research group to just focus on, on world models and how, how do we turn the video generation models that we're building into something broader and something that would be useful beyond, uh, also content creation as well.

   7. SWswyx26:49

      And that was roughly when?

   8. AGAnastasis Germanidis26:50

      Yeah. So that was in-

   9. SWswyx26:51

      Oh [laughs]

   10. AGAnastasis Germanidis26:51

       ... in late 2023.

   11. VIVibhu26:53

       Interesting. You know, like I think, uh, a lot of people have been saying a lot of video gen model companies have all pivoted to world models these days, but like, you know, 2023 you're posting it. Uh, one-

   12. SWswyx27:05

       I mean, it, it's, it's debatable whether it's a pivot.

   13. VIVibhu27:08

       Yeah.

   14. SWswyx27:08

       Like arguably-

   15. VIVibhu27:08

       Yeah

   16. SWswyx27:08

       ... that's what you always had to do anyway, right?

   17. AGAnastasis Germanidis27:10

       Yeah. It's in a, in a way an expansion-

   18. VIVibhu27:12

       Yeah

   19. AGAnastasis Germanidis27:12

       ... of the applications-

   20. VIVibhu27:13

       Yeah

   21. AGAnastasis Germanidis27:13

       ... of the models as they, as they become more capable.

   22. VIVibhu27:16

       The early signs, it seems like the original models you guy had-- guys had, people would say it's very not bitter or lesson pilled, right? You're adding rewriting prompts. You're having all these one-off things, but that's just the state of the tech as it was versus the future of, as you said, you can scale it up is, you know, we can scale up to world models.

   23. AGAnastasis Germanidis27:35

       Yeah. So it, it just became... And, and, and if you looked at the outputs of Gen-2-

   24. VIVibhu27:40

       Yeah

   25. AGAnastasis Germanidis27:40

       ... it was not-- I think it was not obvious to people that this would scale to become a general simulator of the world. Like you had very limited movement. You had, you know, very low fidelity or low resolution, like obvious mistakes in human anatomy, like all, all kinds of limitations. But it was just, you know, uh, the idea was that's just GPT-2 and GPT-2, you know, it can barely generate like coherent sentences. Similar, Gen-2 can barely create coherent video, but if you scale it up, you're gonna-- there is no reason why it shouldn't work in a way. It's, uh... And I think that was-- that's, that's always the mindset of kind of, of Runway is like this extrapolation of like if, you know, like even when we started in 2018 and you looked at the results of the day, you need to look more at the trend of like where we were in 2018 versus where we were at the, you know, when the first GAN came out in 2020, uh, for 2014 or 2015. And, uh, you started from like thirty-two by thirty-two images of faces and then by the time in 2018, you could generate, you know, uh, street images at a 1K resolution. And it was kind of the same with world models, very early signs of something much bigger.

   26. SWswyx28:52

       Yeah. I was almost thinking like it's kind of diffusing into focus. Like if you look at our visible output from year to year to year, it looks like a diffusion process itself.

   27. AGAnastasis Germanidis29:02

       Yeah.

   28. VIVibhu29:02

       Especially watching the early like, uh, old, old blog posts, you can really see the choppiness, the-

   29. AGAnastasis Germanidis29:08

       Yeah

   30. VIVibhu29:08

       ... details.
7. 29:47 – 36:48

   Learning From the World, Not Just Language

   1. AGAnastasis Germanidis29:47

      Exactly, yeah. So the, the, the way we see it is, uh, there is that, um, that, uh, initial mission statement of DeepMind, which is, uh, uh, solve intelligence and then use it to solve everything else. But I, I think it's starting from everything else [laughs] uh, could be valuable of like starting from, you know, there is just so much complexity, uh, and detail in the world that in order to That it's, it's hard to learn directly from just human descriptions of the world. Like, we're assuming that, you know, like language models learn from everything that humans have written about the world, uh, like our own understanding as of, you know, the 2020s and there is just so much that we don't know and so much that's not captured by existing text, uh, about both the, you know, the low-level dynamics of the world, like we're not describing in detail. You know, if, if I tell you to describe like how you tie your shoes, that's a very difficult thing to, to describe in words, but it's very obvious thing to demonstrate. And so I think there's been-- and there's, you know, more of X paradox, like we're constantly underestimating all the complexity that goes into very like things that we do subconsciously as humans, and we don't even necessarily always have the words to describe them. And so in my mind, the simulating the world and simulating, um, physics, simulating the dynamics of the world has always been kind of underestimated, uh, uh, compared to, uh, we place too much emphasis on the things that are easy to talk about. Um, but there is just all this complexity and kind of richness of the world that if we just try and train directly on that observational data instead of training on how people describe the world, we would learn something new that we wouldn't otherwise know.

   2. SWswyx31:39

      You think that the present architectural paradigm is fine, you don't need like another layer like a Jeppa, you know, like another famous, uh, New York AI leader would say?

   3. AGAnastasis Germanidis31:49

      You know, we-we're a very pragmatic research lab. If, uh, we have evidence that an approach works better than the approach that we're taking, then we have no qualms to taking it. We just have seen no indication that video prediction itself doesn't scale. And even if you look now, you know, not just our work but the work of others, you're seeing in robotics some of the most promising work, um, starts from video prediction models and then you adapt them to also be action models, for example. Um, so there is very little evidence that you need something else and that your time is better spent on a novel architectural change compared to improving data and improving the-- and scaling the current, the current approach. And so we don't have any indication that, you know, the, the, the, the, there is that counterargument that, uh, I think there was a, a tweet by Yann LeCun a few days ago about, you know, uh, understanding the dynamics of the world is very different than, uh, generating, uh, cute videos. Uh-

   4. SWswyx32:49

      And your answer is no, they're the same thing. [laughs]

   5. AGAnastasis Germanidis32:51

      Yeah, they're, they're the same thing.

   6. SWswyx32:52

      My cat videos are the same as [laughs] understanding physics.

   7. AGAnastasis Germanidis32:56

      Right. 'Cause if you wanna generate, you know, obviously video models can cheat and like they could-- you could give like successive dif-- uh, shots of the scene in a way that doesn't require you to actually simulate difficult physics. There is alw- like all these different ways in which you can hide the deficiencies of the model, and it's important not to be kind of too tricked by the performance of the current video models. It's easy to, you know, cherry-pick examples and, and think that video models are further advanced than they actually are. So there is a lot, a lot more work that we need to do to improve those models. But in my mind, very similar to language and like we've, you know, you go from barely coherent sentences to something that, you know, could hold a conversation with a human to something that could-- can operate autonomously for, for a day and like create entire code bases. And the main difference, there is obviously some architecture improvements along the way, but the main thing is scale. And so the-- it's the same bet for video, and we have no indications that this is out trading. Like we have benchmarks that we use for measuring the physics of those models, and we see those predictably improve as we scale those models. So there is, uh, if, if you want to pull up, uh, Physics IQ, uh, is one of those benchmarks that measures how well does the model perform at solid mechanics or fluid dynamics or optics. Um-

   8. VIVibhu34:16

      I'm curious if you've seen any emergence, any scaling law around this.

   9. SWswyx34:21

      Yeah, he's saying there is a scaling law, right?

   10. AGAnastasis Germanidis34:23

       Exactly.

   11. VIVibhu34:24

       Yeah, I mean-

   12. AGAnastasis Germanidis34:24

       So, so the way those, those, those models, the, those benchmarks work is you, you know, the researchers have gone and like captured, uh, a few videos that are representative of different physical phenomena, and then you can take the first frame and then pass it through an image to video model and then generate, generate kind of a rollout that shows what should happen next. So you have, uh, a ball hanging from the ceiling, and then you use that as input, uh, and then you, you gen-- the model predicts how the mo- the, the ball should fall on the ground. Uh, and this measures, you know, we have an intuitive understanding of physics. I know, you know, you can imagine what will happen next if I drop this, uh, this bottle. So it's measuring that same intuitive physics understanding of those models, and we've measured that at different model scales, and we see, uh, and compute scales, and we see that the score and physics IQ predictably improves. There's other, you know, tricks and techniques that you can make to improve the score even further, but even scale alone helps, uh, in, in the, in the model learning better physics.

   13. SWswyx35:24

       My main sympathy with Yann LeCun is the, uh, Plato's cave aleg-allegory, right? Like you're, you're like learning on the output of a thing, not the internal process of a thing, and it's very, very noisy and, you know, if only you could observe the internals of a thing. It's hard to observe the internals of a human mind, but you can very much observe or at least we have a, a whole branch of science and physics that we're ignoring on how to model, uh, physics and, and movement and, uh, you know, gravity and, you know, other interactions. Uh, and we're just like throwing away all that and just saying, "Just, just scale data," which is very much the lesson of unsupervised learning, but it feels wrong. [laughs] It is basically the main idea.

   14. AGAnastasis Germanidis36:06

       I think the history of machine learning is, uh, at large it feels wrong.

   15. SWswyx36:09

       Yeah. It's a bit of lesson right now. It's, it's, it's the simple answer to that. I guess how much can you scale? So like even on, let's say, the video generation side, like there's one side of video understanding. Video generation, are we still gonna have tools where it's like, "I wanna generate two hours, 20 hours"? Um, there's a infra way to do it in batches and stitch it together, but like do we just keep scaling? Do we just continue long generation consistency, all that with scale? And like tying it into where we're at now from we looked at Runway 2 to 4.5-

   16. AGAnastasis Germanidis36:42

       Yeah

   17. SWswyx36:42

       ... like technically, what, what advancements have we made to today, and then where do you see things still going?

8. 36:48 – 41:23

   Sora, Runway’s Existential Crisis, and Gen-3

   1. AGAnastasis Germanidis36:48

      So part of the answer is definitely scale. Um, and that was... We, we learned that lesson in a, in a big way for-- with Gen-3. So Gen-3 was the model we released, uh, the year after, um, like in 2024. That was a, a few months after Sora was released. Um, so yeah, there is an, an interesting story of that, that, that, that came to be as well. Gen-3 for us was, you know, the, the first time that we need-- really needed to build... Basically, we had to learn all the lessons that the language model world learned in it to-- in, in three years, in the span of a few months. Uh, one of the biggest changes of Sora was using diffusion transformers instead of ConvNets. So a lot of the early, you know, latent diffusion models were all, um, uh, ConvNets for the diffusion model part. And the diffusion transformer paper came at some point in 2023, and it basically showed scaling laws for image, uh, diffusion transformers. And we, we realized at that point that we needed to invest in infrastructure for model parallelism, for really scaling, scaling training to larger than, you know, a few billion parameter models. And we spent maybe the, you know, most of the fall of 2023 building out our infrastructure for distributed training. And we had a lot of false starts and a lot of failure in trying to scale, um, image and video diffusion transformers. And at that point, uh, you know, February 2024, Sora comes out and the results are very much superior to what Gen-2 could produce. There were a lot of, you know, a lot of chatter on Twitter about Runway. [chuckles] “Runway’s done.” Uh, like there is, uh, there is no way Runway will catch up. And if you remember also OpenAI in the early 2024, it felt very, uh, like it's a-

   2. SWswyx38:41

      To the moon

   3. AGAnastasis Germanidis38:41

      ... it's a formidable opponent now, but at that point, it, you know, they were on the top of their game. You know, nobody could even get close to them. There was maybe Gemini, was just the first version of Gemini had just released. So when OpenAI came with Sora and it was such a big jump of like quality, it gave me... There was like an existential crisis for, for a few hours. But that-- I think the, the amazing thing about Runway and like I think the, you know, we've been around eight years now, which is almost we're a dinosaur in AI, and we had to, uh, like there was a lot of those moments we had to learn, adapt very quickly and build out skill set in the team that we didn't have. And so, you know, if you ask anyone what is their favorite time at Runway that was there during that time, it was that, that push in like three months to get to a model better than, than Sora. Uh, and it-- you know, we scaled 10X the model scale, the, the, the model size and the, you know, compute that we were training on. Uh, we figured out model parallelism. We had zero expertise in that. And then we came out with Gen-3 during, during that summer. So, um, so that was a big turning point, I think, for the company, where the, the research org grew very quickly and we really started pursuing this vision of the general world model, uh, in earnest, I think after, after Gen-3 was out.

   4. SWswyx39:56

      Yeah. I mean, that's the, the amazing thing about building when you're building. There's no stack to-- You have to invent everything yourself. You have to be completely full stack. You know, now, now I, now I think like there are inference specialists like Fal or whatever that can help with like, uh, model serving, and I, I think you guys work with them as well. Uh, but yeah, like it's, uh, you know... But at the time it was just-- It's very interesting to think about what you do when s- when Sora comes out and people are questioning whether your company should still exist. [chuckles]

   5. AGAnastasis Germanidis40:25

      Yeah. And yeah, there was no, you know, v- there was no VLAM of diffusion models. They-- We had to build the whole model serving infrastructure and make things efficient. And a few months after we released Gen-3, we released the turbo version, which I think was the first step distilled model in production at the time.

   6. SWswyx40:41

      And that was the whole trend that we covered as well. Yeah.

   7. AGAnastasis Germanidis40:43

      So that allowed us, uh, actually to serve, uh, to serve the, those models at the, at the larger scale, 'cause I think the f- the first version of Gen-3 was, was quite, uh, expensive to serve.

   8. SWswyx40:53

      You know, I think the, the whole like trend in like consistency models, lightning and, uh, turbo and all these things somehow didn't really stick around. I don't know if you have any reflections on this. Because at the time I, I was like, well, obviously everything should start with a distilled model first, and then you can upscale, right? That it, it-- Basically, your, your bigger models just turn into fancy upscalers. But like you should always draft with a, a smaller model and faster model, right? Because you can get it so quickly, like near real-time.

9. 41:23 – 44:50

   Why Real-Time Video Is Inevitable

   1. AGAnastasis Germanidis41:23

      Yeah. I, I would not be so sure to, [chuckles] to say that didn't stick around. I think that, uh, it's, it's likely to, uh... I mean, there, there is a lot of step distill models that are actively used in production. Uh, there is still obviously a gap in quality compared to the, you know, the, uh, non-distill model. Uh, but i- in my mind, we're still-- You know, there is a two to three-year offset from language models. So the things that, um... So it, it's just a matter of time b- before there is better distillation techniques. Um, you know, we, we use-- Right now we have a real-time model, Core Character, that I think is the largest deployment of real-time video models. Um, that's a, a step distill model, and it's actively being used. It's a very specific use case compared to a general video model. So this is a-

   2. SWswyx42:11

      Very cool, by the way.

   3. AGAnastasis Germanidis42:12

      This is avatar stuff, right?

   4. SWswyx42:12

      Consistency, character.

   5. AGAnastasis Germanidis42:14

      Yeah. So this is a, a talking avatar, uh, model, uh, we were able to You know, we, we, we optimized the, the hell out of it, uh, and-

   6. VIVibhu42:23

      [chuckles]

   7. AGAnastasis Germanidis42:24

      ... and it, uh, it generates at twenty-four FPS, and it's a, it's a step-distilled autoregressive video model. So if we look at our world model direction, uh, a big component of it is starting from the bidirectional diffusion that basically generates an entire video at once and making autoregressive shows. So you generate one frame or a few frames at a time. Um, so there's a lot that goes into that pipeline of getting to a real-time model. It's first you need to make it into a causal autoregressive model, and then you need to turn it into-- You need to do some additional step distillation to get it to actually be real-time. Um, and I think that part is actually just, just starting. Uh, I will be very surprised if we're, you know, two years from now, we don't primarily use real-time models. To me, real-time video generation is just inevitable that, you know, it has much better user experience, it's much cheaper to serve, and, you know, the quality gap between the base model and the real-time model is only gonna close as we, uh, as we figure out better, uh, distillation techniques. And we made a lot of progress there internally on maintaining the quality of the base model when we, when we distill them.

   8. VIVibhu43:33

      How much of this is transferable? So is it the same base model? Like if you're doing diffusion across the whole sequence and you're converting it to step autoregressive distillation, is this like distillation where you still need to train both, you can use the same base and converter? What's that process like to go from regular model to something that's real-time on a technical level?

   9. AGAnastasis Germanidis43:56

      So the nice thing about diffusion models is you have, uh, two axes of distillation. So there is the... You can distill to a smaller model, which resembles what you do in the LLMs, or you can distill in, in terms of taking less steps, uh, less diffusion steps. So you could take a model that generates in fifty steps and, and, and generates in four steps and get to, um, you have some performance, uh, degradation, but very often you get comparable, comparable outputs. So you can even take the, the, uh, you know, the large frontier model and distill it with step distillation and get to real-time performance, and that's what we've seen. So, um, depending on the use case, in some cases, we might also start with a smaller model, but in a lot of use cases, we actually just use the-

   10. VIVibhu44:40

       Step distillation

   11. AGAnastasis Germanidis44:41

       ... the frontier model, and we're able to make it work in real time.

   12. VIVibhu44:44

       I think this might be a good time to cut over to his laptop to show off some of the real-time stuff that you're doing.

10. 44:50 – 52:09

    Interface World Models: Software Without Code

    1. AGAnastasis Germanidis44:50

       This is one of the research updates that we, uh, we did recently. Um, so we've been working in f- uh, in getting our general world models to, uh, different applications. Uh, one of them that we think is very, uh, is very compelling is using general world models as essentially, uh, an interface, a universal interface to software. This is a, a, a version of our, uh, world model that's called an interface world model. Uh, and the idea is that it, uh, it essentially, uh, replaces, uh, uh, the f- front end of a software application. It renders the pixels directly of an interface, and it's trained to predict what happens next as a result of a, a click or another interaction you have with the interface. So this is all pixels. Uh, it's-- there is no HTML, CSS, React that's powering this interface. This is directly at the output of our real-time, uh, video generation model, and it takes clicks directly as input.

    2. VIVibhu45:53

       And drags. Click and drag.

    3. AGAnastasis Germanidis45:56

       Right. So it supports-

    4. VIVibhu45:57

       Ooh.

    5. AGAnastasis Germanidis45:57

       Uh, yeah, clicks, it supports drags, uh, it also supports scrolling. Um, and the, the, the amazing thing about this is that you can effectively describe in the prompt how you want different elements. Like, what do you want the behavior of different elements to be? So it's almost you're-- you can turn, uh, an interface from a, you know, markup language description of like an HTML interface, and instead you can just describe the interface. You know, if I press this button, I expect this to happen. If I press this button, this should happen. And it's useful, we believe, both for prototyping, for like just testing like what different interactions would feel like. Um, you can also add audio to it, so it's a, a video/audio generation model. So you get-- essentially, you can describe both what the visual outcome should be of your click and also what the-- if, if it-- there is a sound effect that comes out of it. So we, we believe that's gonna be a much more flexible way of building software. Just render. It just... You know, uh, why, why generate the code that generates the pixels? Just generate the pixels directly.

    6. VIVibhu47:02

       [chuckles]

    7. AGAnastasis Germanidis47:03

       Um, uh, it's the end-to-end philosophy applying-- applied to, uh, to front ends. So we think there is a few interesting use case, so you can build creative tools on top of it. Um, we think that, you know, for any kind of use case that involves a lot of exploration or like a educational use case where you wanna learn about a new concept, and you want some kind of visualization and ex-ex, uh, and kind of open-ended exploration, we think those, you know, this is a very powerful, uh, approach. You know, you can imagine new, new forms of, uh, design in kind of industrial design software that could emerge as a result of, uh, of, of those models. And this is all, uh, generated in, in kind of in, in, in real time as well. So, um, you, you can, you know, you can build a lot of interesting kind of camera transitions and kind of forms of interaction that are very difficult to, to build otherwise. And one way in which we evaluate this is what if you try to generate the same interface with, uh, with Claude by just, you know, prompting Claude, "Here's an image reference of my interface that I made in Figma or that I created some- somewhere else. Uh, create this particular interaction," which in this case it's, you know, drag that object, uh, upwards. Um- And beyond it being slower, it's also very difficult to capture some interactions by just fully, uh, with, with just LLMs. So we think that this, this is likely to be the way that a lot of the fut- like software in the future will be created. Uh, and one of the a- additional benefits is personalization might be a lot easier done with, with those models. Like you can essentially try out different prompts based on who is, who's visiting the interface. You can, uh, more easily, you know, prompt engineer the, the interface to have larger size, um, text for more accessibility reasons, or you can make this... Or, or like if you have a part- particular aesthetic preferences. So, so we're very excited about this approach. It's obviously early days, and I think we'll need to, uh, make it more cost-effective as well to serve those models. 'Cause, you know, running a real-time video model versus just purely rendering HTML, there's obviously v- the computational needs are much higher. Uh, but we do see a lot of potential in this approach to building kind of front-end interfaces.

    8. SWswyx49:31

       So the-- we covered this similar thing with Flipbook before with our, uh, Ethan Ha episode with Groq, uh, video. And I, I think it's very engaging visually. I think it's maybe very good for education, but it's-- it does sound expensive. I think there's an upper bound to how expensive it will be though, right? Like, uh, you know, the inference costs will go down over time. You'll figure out ways to optimize it. Effectively, when it pauses, you don't-- you're not receiving human input. You don't have to generate anything, right? So [laughs]

    9. AGAnastasis Germanidis49:59

       Yeah. I mean, you could also... Like in this case, you have, uh, ambient motion, so there is parts of the, of the screen that might... You know, if, if you're-- let's say you wanna, um, visit Paris, and then you, you get this interface that you-

    10. SWswyx50:13

        People walking

    11. AGAnastasis Germanidis50:13

        ... allows you to explore.

    12. SWswyx50:14

        Yeah.

    13. AGAnastasis Germanidis50:14

        You have people walking or like things happening. But, um, it's, it's a n- yeah, it obviously makes things more expensive because you need to run the model all the time. Maybe you have some looping mechanism, so you don't need to do that. But all those things I think is stuff we'll need to figure out.

    14. SWswyx50:28

        Yeah.

    15. AGAnastasis Germanidis50:29

        I think our first consideration is let's make this clearly find some use cases where it's clearly a much more compelling interaction compared to traditional interfaces. And then it's a matter of time b- before it becomes more cost-effective to serve.

    16. SWswyx50:42

        Yeah. When, when it comes to the people walking, you know, I think the approach that makes the most sense to me is basically Moon Lake.

    17. AGAnastasis Germanidis50:48

        Lake.

    18. SWswyx50:49

        Ugh, I keep, I keep messing up their name. With Chris Manning and Fan Yunsun, I don't know if you've come across them, where they basically map to some kind of game engine, I think it's Unity or something, or Godo- Godot. And they-- you can obviously, obviously script some NPC behavior behind that and train on that. Whereas here you can really imagine whatever you want. Like that is a UI, right? Like, and, and it feels like more tractable, I guess, to, uh, create a world model of software that is interactable because we have many of examples of that, and you can, you know, do your fancy RL environment stuff on that than it is scaling up to embodied in real-world physical use cases. But this is a nice first step.

    19. VIVibhu51:27

        Or, you know, there's the opposite of you have like 1B models, 350 million parameter language models. It just gets so small that they're just predicting like, you know, fishes moving-

    20. AGAnastasis Germanidis51:37

        I mean, small models are now 120B, so [laughs]

    21. VIVibhu51:41

        Ultra mini on device.

    22. AGAnastasis Germanidis51:42

        [laughs]

    23. VIVibhu51:43

        But, uh, no, I think it like, it puts it into pers-perspective, at least the car one for me, like the applications, right? The amount of work to do that, sure, you only make one model your car per year, but applying this, it's also a cost saving to have to manually make all this, right? So it opens up a lot of possibilities too. I'm curious if you extend this out two, three years. So where do you see things going even further,

11. 52:09 – 56:55

    The Fully Neural Operating System

    1. VIVibhu52:09

       you know?

    2. AGAnastasis Germanidis52:10

       Effectively, the, the end game of something like interface world models is you have, uh, a fully neural operating system. So I think, uh, uh, Andrej Karpathy has written about that quite, quite a, quite a while back. But it's, uh... You know, I think to me it's, it's a bit, um, it's a bit odd that, you know, we have, for example, with an interaction with, with an LLM of today, you have this LLM that can basically talk, talk to you about anything. It can-- you can take the conversation any direction. You can, you can-- it's very general, so it can solve all those different tasks, but you interact with it through a very rigid interface. And so to me, it's just a matter of time before the interface itself becomes learnable and becomes, you know, part of the, the, the whole loop of like, you're not just delivering... You're delivering an application end-to-end, and that means you're delivering the language model, but you're also delivering the, the render and the pixels and, and, and that's also a, a learnable component. And the concept of applications might not necessarily-- I think we'll need to figure out new abstractions for software. Um, the, the concept of ap- application comes from this idea that you need, you know, separate kind of code bases to describe to, uh, for, to power each individual, um, tool and each individual application. But you might, uh, you might think of something a lot more unified if you're, if you have a, a video model that's actually generating the interface as you go. Um, so it can take context from an LLM and allow you to combine kind of different, uh, different functionalities that traditionally would live in different, uh, applications. So, so it's a, it's a way to solve, you know, software end-to-end, uh, effectively. We also see this as a powerful way to train computer use agents as well. Um, so this is, you know, one way to see this as... And in general with world models, there is those two directions. One is world models for humans and world models for, for-

    3. SWswyx54:08

       Agents

    4. AGAnastasis Germanidis54:08

       ... to train agents.

    5. SWswyx54:10

       Yeah.

    6. AGAnastasis Germanidis54:10

       And so for every new work of, uh, world models that we do, we have this, these both uses become possible. So this is a powerful synthetic data generator for training computer use models. It could become, uh, a live, uh, RL environment that you could use to do online RL with a, with a computer use agent. Uh, and you can get wide diversity of different interactions, kinds of interfaces, uh, just generated on the fly that, um, to improve the how robust the-- your, your, your agent, uh, becomes. So that's the same also with the world models that we're working on for a robotics use case as well.

    7. SWswyx54:46

       Is there a research breakthrough that you're waiting for that would unlock the next set of use cases that you really wanna

    8. AGAnastasis Germanidis54:53

       Pursue. Long context is a very important one, so being able to maintain consistency for long periods of time, and that depends on the use case. So for our characters model, for example, or for the interface world model, it's easier to maintain long sessions of interaction. If you go into more open-ended worlds that you navigate and you take arbitrary actions in, we-- like, there is m-more the context at which you can, and duration which you can generate becomes limited much more quickly.

    9. SWswyx55:24

       Yeah.

    10. AGAnastasis Germanidis55:24

        So we see more degradation and error accumulation happening. Um, so the biggest challenge with autoregressive models is error accumulation, is basically you're fitting generative frames back into the model to generate the next-

    11. SWswyx55:36

        [chuckles]

    12. AGAnastasis Germanidis55:36

        ... the next frames, and if there is any small errors, they accumulate over time. That's not a new problem. It's a problem that LLMs also have, and we've seen the, uh, you know, the ability to generate now really, really long outputs. So it's a solvable problem, but it's definitely still, still a challenge.

    13. SWswyx55:52

        Yeah. And what is the state-of-the-art? Uh, so for, for Grok it would be, like, ten to twenty seconds of context going in there for video.

    14. AGAnastasis Germanidis56:00

        With our, our characters models, we're able to generate up to thirty minutes of, uh, video autoregressively.

    15. SWswyx56:06

        Yeah.

    16. AGAnastasis Germanidis56:06

        Um-

    17. SWswyx56:06

        But, but that's just for the avatars.

    18. AGAnastasis Germanidis56:08

        Yeah. So if we look at the GWM worlds, which is more our open-ended world exploration model, um, it's, it's on the order of a few minutes, which is-

    19. SWswyx56:18

        Hmm. [chuckles]

    20. AGAnastasis Germanidis56:19

        Yeah, um, still-

    21. SWswyx56:20

        It's probably enough for people because you have to cut to the next scene anyway, right?

    22. AGAnastasis Germanidis56:24

        Yeah, it's not, it's not the ideal game experience if you have to [chuckles] restart every few minutes. So I think, but, uh, I think it's, yeah, i-i-- for certain kinds of game experiences, you can, you can work around it. Uh, ideally, you are able to just generate forever, and it doesn't, it doesn't degrade, and I think that's a matter of time be-before we get there.

    23. SWswyx56:43

        Yeah. Genie has, like, one, max one minute, you know?

    24. AGAnastasis Germanidis56:45

        Right. Yeah.

    25. SWswyx56:46

        Sure.

    26. VIVibhu56:46

        This was your-- You did a study on robotics. I think I also have just-

    27. AGAnastasis Germanidis56:52

        Yeah

    28. VIVibhu56:52

        ... your Runway robotics page, though. Is this better?

12. 56:55 – 1:04:16

    World Models for Robotics

    1. AGAnastasis Germanidis56:55

       So la-last year we released Gen-4.5, so that was our, our latest v-base model, and we've been, uh, as I mentioned, we've been doing all this work in world models and, uh, which essentially a lot of our approach to world models is how do you take a bidirectional diffusion model and make it autoregressive and make it accept actions? So instead of being a video you watch, it becomes a, a simulation that you step in and you can, you know, control it every step of the way. You can explore counterfactuals, like what happens if I take this action versus if I take this action. And GWM one was the, it's the world model that we built on top of Gen-4.5. So we did all this autoregressive and, like, uh, uh, distillation, uh, auto-autoregressive and then step distillation on top of Gen-4.5. And one of the biggest use case that we saw for GWM one was in robotics. One thing, thing we like to say is we, as we scaled video models, we accidentally, uh, created one, a state-of-the-art model for robotics, uh, by just scaling-

    2. VIVibhu57:56

       [chuckles]

    3. AGAnastasis Germanidis57:56

       ... video models. So we realized at some point, um, mid last year that robotics labs that are coming up to us and kind of asking to use video models for synthetic data, kind of asking us to post train our v-video models to work really well for robotics, uh, so that they can, they can use that to basically generate variations. That was the first use case that we saw. And then i-increasingly, it became clear that the models will be useful beyond just creating synthetic data to train robotic policies. They would also be very useful as simulators. So that means that you can use, uh, a video model online to test how your robotic action model performs. So you can take an action, roll, and then get the outcome of the action inside the world model and then, and then con-continue that loop like this closed loop simulation. And you can use that to evaluate how well your robotics model works. Um, and the biggest thing that I think you need to solve if you wanna build a simulator is establishing real-world correlation, that if you take an action inside the world model, if you take the same action in the, in the, in the real world, you get a similar outcome. So that was, that was the goal of some work that we did earlier this year. So if you go to the first link, so that was, uh, uh, essentially we wanted to establish that, that, you know, uh, real, real to sim correlation for our world model, so that if you do a series of actions inside the world model and if you do the same actions in the real world, you get similar outcomes. And we took our GWM one model, and we, we used some benchmark data that there is this RoboArena, uh, benchmark that's very commonly used to evaluate how well do different action models perform. And we use the same scenarios and settings and embodiments inside our world model, and we measure the correlation of how well did the action model perform inside the world model versus in the real world. And we saw that we could get very good correlation between, uh, between our world model and reality. And that means that if you wanna evaluate how well your robotic policies perform, you can scale that much f-faster inside simulation instead of sc- instead of having to do that with actual physical hardware. And so that was a first indication that the, our models could be, uh, quite useful in robotics, and we saw as, as we're working with robotics labs, that that became, like, the, the first use case where they could use video models in a way that fit into their, their, their kind of training pipeline.

    4. VIVibhu1:00:24

       Can I ask what-

    5. AGAnastasis Germanidis1:00:25

       Yeah

    6. VIVibhu1:00:25

       ... the difference was from 4.5 to solving that? So the sim-to-real gap has always been the issue, right? You train a robotics model on video data. It doesn't generalize to real world, and the simulation had an issue. So seems like you solved it, but how? [chuckles]

    7. AGAnastasis Germanidis1:00:40

       Yeah. So a, a big problem with simulators is, you know, if, if you're trying to simulate rigid objects, like it works quite well if you know, if you, if you can describe the physics of objects very accurately, um, uh, then you're able to use, um, Isaac Sim or, or MuJoCo or one of the traditional simulators. But for more complex interactions with, with cloth, for example, or, um, like slippery surfaces, um, you know, with the, all the complexity that you, you wanna be able to solve with a manipulation, with an action model that solves manipulation tasks, it's very- Difficult and so time-consuming to build, you know, for each of those environments and each of those tasks build the simulated version of that, uh, the digital twin of that, of that environment. Whereas with a world model, you, you just need to provide the first frame, and then you just can roll out the policy inside the first frame. So whereas, you know, we compared it to methods that required like 3D scanning an environment and then 3D scanning each i-individual object before you can now, uh, you know, you can bring that to simulation. Um, whereas with a world model, you just take a picture of the, of the environment, and then, and then you're able to test how, how your policy performs. Our general thesis on robotics is, you know, there is companies that are leveraging a lot of teleoperation data to, to train robotics action models. There is now companies are using, um, UMI data, which is, uh, essentially human, uh, e-egocentric video where humans use kind of, uh, robotic grippers to perform different manipulation tasks. And then there's companies that are focusing on egocentric data, which is, you know, you strap a GoPro on, on someone's head, and then you kind of capture them performing a task. We think that, you know-- And all those are great source of data for training robotics models, but the most plentiful source of video data is third-person video data. It's-- And if-- How, how do we as humans learn how to perform different tasks? A lot of it is by observing others perform those tasks. We don't learn from first person. We obviously do some trial and error and like, uh, to learn different things. But ultimately, a lot of what we learn how to do in the world, we learn by watching other people do it. And that's how when you're pre-training a video model, you're essentially doing that. It's a lot of third-person video, uh, footage of people performing different tasks in the world, people doing sports, people doing household tasks. And the, our main thesis is that video pre-training, once you do that, you can then adapt a model to be useful in robotics use cases with lay-way fewer hours of actual robotic data. So you require way less teleoperation data, which is very difficult to scale. Uh, and even if you look at egocentric data, which is a bit more easy to scale compared to teleoperation data, which requires actual hardware, it's still three orders of magnitude less of that that exists in the world compared to third-person video data out there. And so our thesis is, and generally, like the most plentiful source of data will ultimately, ultimately wins. Third-person video data pre-training is the right starting point for, uh, for models that, you know, you want them to generalize and be able to deal with new environments, new tasks, things that you haven't seen during training. That's the motivation for why we think our, our models are especially useful in robotics, uh, settings, and we've seen that to be the case, um,

13. 1:04:16 – 1:09:31

    Robot Policies and World Action Models

    1. AGAnastasis Germanidis1:04:16

       as well.

    2. SWswyx1:04:17

       You said pre-training. So maybe it's like third-person pre-training, first-person SFT. Is that like a curriculum that you can sort of introduce?

    3. AGAnastasis Germanidis1:04:25

       Exactly. So if we look at GWM worlds, uh, so GW- so GWM Robotics. So the GWM Robotics, it starts from Gen-4.5.

    4. VIVibhu1:04:34

       It's the same video diffusion backbone, right?

    5. AGAnastasis Germanidis1:04:37

       Exactly.

    6. VIVibhu1:04:37

       Yeah.

    7. AGAnastasis Germanidis1:04:37

       Yeah. So you start from the base video model, the one you're using to generate, uh, cats and dogs and-

    8. VIVibhu1:04:42

       [laughs]

    9. AGAnastasis Germanidis1:04:43

       ... other interesting stuff, and then you, uh, fine-tune on a very small number of hours of robotic data. So it's something on the order of kind of hundreds of hours compared to if you were to pre-train a robotics model, the current pre-trainings go up to, you know, uh, hundred thousands or like millions of hours of, of data. And you're able to get quite, quite good performance, uh, quickly, uh, because the model leverages all the things that it has learned about the world, physics and human dynamics and the tasks that people care about from pre-training. And ultimately, you, you want those models to generalize. You don't want to just be able to perform the tasks that it has been during training. And the diversity of actions and environments that y- that you have with a pre-training video data set is much larger than, uh, what you can kind of realistically capture manually.

    10. VIVibhu1:05:38

        How is the scale looking like for the post-training? Like, do you still wanna do, is it like roughly ninety percent of the compute in regular video diffusion model and then scale up a lot or do it like we want different robotic models for different tasks or just the one base really good world model can also apply to robotics?

    11. AGAnastasis Germanidis1:05:58

        So currently, uh, we are post-training our models for specific, uh, embodiments that we, uh, for as particular partners. So if they have a particular kinda single-arm robot or a bi-manual robot or a humanoid robot, we would post-train our GWM robotics model on their particular data set. Over time, we see the different variants of GWM unifying. Like I would expect, you know, if a, a year from now or two years from now, you have a single world model that can, uh, that can simulate manipulation tasks, it can simulate navigation, which is a lot of the gaming world models are navigational world models. You're moving around the space, and it will s- also simulate kinda human behavior, so that's the character models. So instead of having three different models, you, you have a single model that's able to... You know, i-ideally, you're able to simulate what it's like to be in the world. You're moving around an environment, you're maybe performing different tasks, uh, you're talking to other people, and that happens with, uh, the same, uh, a single kinda real-time video model that's generating that.

    12. VIVibhu1:07:01

        Do you think you can solve self-driving? So if you are learning to drive a car in a simulator, you have a world model, your robot is basically, you know, car can manipulate so many axes. How far off are you from something like that?

    13. AGAnastasis Germanidis1:07:15

        So world models-

    14. VIVibhu1:07:16

        Or a really good ADAS system.

    15. AGAnastasis Germanidis1:07:18

        World models are definitely being applied to, uh, self-driving, uh, research right now, mainly for evaluation use cases, but our focus has been more on robotic manipulation. We've done some work on AV, uh, world models as well. Uh, but- Yeah, I-- we do think that world models are-- and video models are the best starting point for both simulators and also policy and the action models. So that's, that's the other side to this, is that once you have a great world model, then you can just add an action head, and it can predict actions as well. One way to think about it is if you take the starting frame of a scene with a robotic arm and you ask, you know, you prompt the model, "Generate the arm picking up an object," it would-- And if it generates an accurate enough video, then it, it should also be able to generate the exact poses, uh, in 3D that the arm should take to, um, to perform the same action. So this is the direction that's now the, the popular term for it is world action models, which is you're starting from a video model, and then you're adding an acti-an action head to predict the actions, and it becomes a policy essentially.

    16. SWswyx1:08:28

        One thing I'm also impressed by is how much data you actually need to train these kinds of models. You probably can't say exactly how much, but like, uh, you know, like the, the original, uh, diffusion models and from what I know, even of the open source Chinese models, it's not that much data. Isn't it surprising?

    17. AGAnastasis Germanidis1:08:46

        What do you define as much data?

    18. SWswyx1:08:47

        Um, yeah, this comes close to... Is the token count still relevant?

    19. AGAnastasis Germanidis1:08:53

        So it's a bit more complicated and, uh-

    20. SWswyx1:08:55

        What is-- I mean, just gigabytes, right? [laughs]

    21. AGAnastasis Germanidis1:08:58

        Yeah. Hours of video, right?

    22. SWswyx1:08:59

        Yeah.

    23. AGAnastasis Germanidis1:08:59

        Yeah.

    24. SWswyx1:09:00

        Yeah, yeah. I feel like something that's interesting is it seems like the-- let's call it tokens to param counts in language models has really, you know, maybe they're three years ahead or whatever, seems to be a lot higher than, uh, video models still, even though technically video has more information, you know, per, per bit. I don't know if it seems intuitive or maybe there's just a lot of, uh, the-- like the, the variability between a pixel to the next pixel is not that high. So like maybe there's just a lot of information that is

14. 1:09:31 – 1:13:25

    The Lucid Dream Test

    1. SWswyx1:09:31

       repeated.

    2. AGAnastasis Germanidis1:09:32

       My answer would be it's still very early. Like-

    3. SWswyx1:09:34

       [laughs]

    4. AGAnastasis Germanidis1:09:34

       [laughs] Like the training video models will scale way further than it currently is.

    5. SWswyx1:09:40

       Yeah.

    6. AGAnastasis Germanidis1:09:40

       And, and you'll have capabilities that go much further than the current models can do. So one thought experiment that, uh, I, I like to use, it's, it's almost like the Turing test of video models or like the Turing test of world models. Uh, go, um, uh-- I call it the Lucid Dream Test, is you have a-

    7. SWswyx1:09:59

       You mean the, the actual person lucid, lucid dream?

    8. AGAnastasis Germanidis1:10:01

       It comes from this idea-

    9. SWswyx1:10:02

       Lucid, lucid rains, right?

    10. VIVibhu1:10:03

        Lucid dreams is telling you're dreaming while you're in a-

    11. AGAnastasis Germanidis1:10:07

        Yeah, exactly. So lucid dreaming is when you realize you're-

    12. SWswyx1:10:10

        In a dream

    13. AGAnastasis Germanidis1:10:10

        ... inside a dream, and then you basically-

    14. VIVibhu1:10:12

        You play around

    15. AGAnastasis Germanidis1:10:12

        ... be able to control what happens in the dream.

    16. SWswyx1:10:14

        No, there's also an inference guy called Lucidrains. Yeah. Or quantization guy.

    17. AGAnastasis Germanidis1:10:18

        He's very prolific, uh-

    18. SWswyx1:10:19

        Yeah

    19. AGAnastasis Germanidis1:10:19

        ... person. Yeah. So let's say you have a VR headset and, uh, you-- and, uh, you're in a room with a-- and, and you're wearing a VR headset, and that VR headset, you know, most, uh, of today's VR headsets have a passthrough mode, so you can see directly what's in front of you in the world, or you can obviously render kind of something inside, inside the VR headset. And there's gonna be a point where those interactive real-time video models become good enough where you wear the headset and you're in the same room and you're walking around and you're kinda-- and you're interacting with objects. You're able to kinda move freely in that room and do, um-- and interact with any object. And at the end, someone asks you, "Did you-- Were you using passthrough mode or were you actually-- or was this, uh, uh, rendered or generated, uh, footage?" And if you cannot tell for sure if that was what you were seeing as you were interacting with and moving around the world was generated or it was, um, kind of passthrough mode and was just what was happening in front of you, that's an indication that the models have become good enough. And we're not, we're not close to that yet. And a lot of it is just this idea of really simulating dynamics and counterfactuals well. Like, you know, if you ask a video model to generate a person scoring a goal versus a person failing to score a goal, it would do a better job at scoring the goal because there is a bias from the training distribution. There is a lot more videos of the person succeeding at scoring the goal. But if you have an interactive model, you want it to be able to generate counterfactuals, like if I take this action versus this action, you want it to generate equally realistic outcomes. Uh, so that's I think the big gap between video models and, and, and world models is that idea of the counterfactual generation. And if you want a great model for robotics, you wanna simulate failure very well, 'cause whether you're using it for evaluation or you're using it as a-- in an online RL loop in the future, you wanna be able to have the model kinda try and fail to do things and, and improve. Um, and so in order to do that, you need to be able to simulate things failing.

    20. SWswyx1:12:27

        This is the only domain where you have too many successful exa-examples and not enough bad examples. [laughs]

    21. AGAnastasis Germanidis1:12:33

        Yeah.

    22. SWswyx1:12:33

        Should be easy to generate failure. [laughs]

    23. VIVibhu1:12:36

        Oddly enough, I think like early image video models weren't good at being human realistic, right? Like you see a lot of the high-res 4K like professional photography, but not just everyday life like normal picture, right? Everything looks like it's professionally generated, like professional pictures, but not just like normal, like, you know, messy cables on a desk.

    24. SWswyx1:12:57

        Okay. So there's th- there's this stuff, uh, you know, one thing we also covered that you guys have, um, video agents that you launch. I guess how does the traditional, let's call it frontier L-- like, you know, autoregressive LMS like feed in, you know, to, to all this? They're driving ro-- uh, your robotics models or they're driving other-- your video agents, uh, production, um, any-anything where you, you see the overlap of autoregressive and diffusion, let's call it.

15. 1:13:25 – 1:24:56

    Video Agents and Omni Models

    1. AGAnastasis Germanidis1:13:25

       Yeah. So harnesses are really important across all those different use cases. So we have this video agent, which is essentially an LM that is very effective at tool use of different, uh, image models, video models, and kinda helps you through creating a, a project end-to-end. So, you know, very often in like a traditional kind of Advertising flow, you have a brief, you start from a-- and then you generate some-- a storyboard, and then you generate the video. A v-video agent and, or Runway agent kind of helps you through that whole process, and it helps you also analyze performance data, for example, how well did this ad perform versus this ad? And then generate me more of the-- based on those learnings, figure out what, what, what to generate. We think the, the, the harness is a very important piece of the, the pipeline. Uh, as I mentioned, all the video production, uh, all the, all the production video models use some prompt completion that happens, and we expect, you know, that to become more and more complex and more, you know, you generate longer and, and more detailed descriptions before you use the, the, the diffusion transformer. I do think eventually, you know, there is increasingly this unification into omni models where you have the-- you're training the models end-to-end to both do autoregressive text prediction and also, uh, diffusion as well. So you're predicting the next token, uh, of like you're, you're maybe using some reasoning and planning of the scene, and then you're passing it into the diffusion head that's actually generating, uh, generating the pixels.

    2. SWswyx1:14:54

       Yeah. I, I think currently maybe only Gemini and Qwen do it. Uh, I'm, I'm not sure who-- which of the Chinese models are omni, but yeah, it's, it's not, it's not a very well, um, popularized modality, I guess.

    3. VIVibhu1:15:10

       It's an interesting use case when you think about it, right? Because not only do you have to end at like language model reason, diffusion head generate, you don't have the output there. You can go back in to feed that output to the same model, reason again on improvements, and it can do a lot of loops just in its own. Uh, I guess the question is like, do we need that, or can we just do agent scaffold, like do it outside the model? Is there a big benefit to doing it in?

    4. AGAnastasis Germanidis1:15:38

       I think there's generally the trend of something is first done by a harness-

    5. VIVibhu1:15:43

       Yeah

    6. AGAnastasis Germanidis1:15:43

       ... and then it becomes part of the model, right? So you had a chain of thought prompting where you had to do this super detailed system prompts to, to get things to output.

    7. VIVibhu1:15:52

       Yeah, let's think step by step. [laughs]

    8. AGAnastasis Germanidis1:15:52

       And now the, the model basically generates the reasoning trace by itself before it gives you an answer. And in the-- in video models, similarly, a lot of the video models of kind of the, the early days were single shot video models, and you had to use some kind of orchestrator to turn-- you know, generate multiple shots in parallel, um, and then turn it into an actual video. Um, and-

    9. SWswyx1:16:12

       Well, I mean, ComfyUI just all over the, you know-

    10. AGAnastasis Germanidis1:16:15

        Yeah, like a-

    11. SWswyx1:16:15

        ... all these nodes

    12. AGAnastasis Germanidis1:16:15

        ... spaghetti- [laughing]

    13. VIVibhu1:16:17

        [laughs]

    14. AGAnastasis Germanidis1:16:17

        ... uh, workflow. Um, and now you have multi-shot video generation where you have the-- you directly generate multiple shots. And there is a benefit to that because then the video model learns some-- You know, to generate a single shot well, you need obviously to figure out a lot of stuff about the world. Uh, to generate multi-shot video well, you also need to basically get some like video editing instincts. Like you need to figure out what is the right pacing of shots. And also, LLMs are not that good at it. Like they're not that great video editors. If you asked a LLM to take some videos and then kinda auto-create a edited video out of that, uh, it would, it would feel uncanny. So I don't think LLMs are actually that good yet at, at being video editors. And I think there's benefit to learning that end-to-end. Um, so I would expect, you know, the training generally is the things that, you know, you need the harness for e-eventually get kind of injected into the, into the model itself, and you learn that end-to-end.

    15. SWswyx1:17:17

        Do you find that you need to hire engineers who can-- or, or researchers who are also artists to infuse that taste, or do you have kind of artists in residence to distill them? [laughs]

    16. AGAnastasis Germanidis1:17:29

        We have, we have a, a, a re-- a, a large creative team that's very actively involved in the-- in training those models, um, like on the, you know, in every part of the way. And like how do you caption video as well, so that you capture the stuff that you need for like the cinematography, the aesthetics, the camera direction in as detailed ways as possible so that you're able, at inference time, to actually, uh, elicit that through the model. You know, we have our creative team also does a lot of evaluation of like, you know, what constitutes a usable video out of those models. And so they're very involved kind of through every part of the, the process. And I think that's one of the special things at Runway is just that, that mix between like kind of creatives and researchers kind of sitting by, by-- side by side and kind of working together to, to build the next generation of our models. I think that's been a really, really important piece to, to, um, you know, how we've operated as a company.

    17. SWswyx1:18:21

        Yeah. In some senses, well, you can only do this in New York.

    18. VIVibhu1:18:24

        I mean, it's calling-

    19. SWswyx1:18:25

        Maybe-- I mean, you have other offices, but like, you know, I'm trying to find some poetic, uh, significance in the fact that you are a big New York company.

    20. AGAnastasis Germanidis1:18:33

        As there's a few parts to being in New York, obviously there is that intersection of all those different industries and like media, kind of advertising, uh, like huge-

    21. SWswyx1:18:42

        Yeah, this is very advertising hub

    22. AGAnastasis Germanidis1:18:43

        ... the art, like the, the art scene is New York. Not, not to, not to say anything bad about, uh, San Francisco, but, uh, it's-- [laughs] there is more, more going on. There is that component, and there's also... I think we benefit from being outsiders and thinking of things a bit differently, like not being in the same like hive mind of, uh-

    23. SWswyx1:19:03

        B2B SaaS?

    24. AGAnastasis Germanidis1:19:04

        ... ASI, uh-

    25. SWswyx1:19:05

        [laughs]

    26. AGAnastasis Germanidis1:19:05

        ... of, uh, of Bay Area and like taking-- and also taking our time to, you know, to get where, where we are today. Like building the-- growing the team intentionally and bringing people who are, yeah, both on the creative side and also on the engineering research side. There's obviously huge talent pool of amazing people in New York, so that, that hasn't really been, been a problem.

    27. SWswyx1:19:25

        I mean, congrats on everything. Uh, what are you hiring for? You know, what should people look forward to, uh, for the future of Runway?

    28. AGAnastasis Germanidis1:19:33

        We're hiring across the board. Uh, I think this is probably the most open roles we've ever had in the history of Runway. Uh, we're growing our research team quite significantly. So if you're, um, if you're excited about video models, if you're excited about world models, if you're excited especially about robotics, the robotics team, we're hiring roles in the robotics across, uh, kind of software, hardware, and research. Um, so definitely, definitely reach out. Um-

    29. SWswyx1:19:57

        And, and, you know, a lot of people don't have direct robotics background, but what should they have, you know, if they want to be useful in robotics? [laughs]

    30. AGAnastasis Germanidis1:20:06

        So ideally some experience with, uh, learned policies, uh-
16. 1:24:56 – 1:28:58

    Artists, AI, and Creative Workflows

    1. VIVibhu1:24:56

       Amazing.

    2. SWswyx1:24:57

       We're gonna end with the AI, Runway AI Summit. The, the last sort of soc-societal issue, I guess, I don't know if, if this is a, a thing, is the, uh, you are at the tension between sort of artists, uh, creatives and AI. A lot of people in that, that community hate AI. Obviously, the people that are in the Runway community don't mind using tools. Uh, it's just another brush. But, um, how have you seen the sentiment change?

    3. AGAnastasis Germanidis1:25:22

       I mean, our perspective, yes, it's just another brand, uh, br- uh, brush. It's just another camera. It's, you know, it's the latest of a, a long generation of, of tools-

    4. SWswyx1:25:30

       Technology in art

    5. AGAnastasis Germanidis1:25:31

       ... uh, technology.

    6. SWswyx1:25:32

       Yeah.

    7. AGAnastasis Germanidis1:25:32

       And art and technology have kind of evolved together. I think there's been a pretty significant shift over the past few months, and it came, uh, some of it you can see with a lot of public figures speaking out in favor of, of AI and being, you know, like i-in Cannes you saw a few, a few directors speaking in favor of AI. We had Ron Howard in our film festival. There is, uh, Mark Suchesa Also adopting AI models. So you have more of those stories coming out every day of, like, a, a well-known figure, um, kind of speaking in favor of AI. And it's just a matter of, in, in my mind, it's those models are becoming more and more demystified. I would say I have, I have also a, a bit of a hot take that one of the things that made the, uh, the initial response to those models maybe a bit more heated than it needed to be was this idea of text-to-video of, you know, you have a single text description, and you get back a, a, a full-

    8. SWswyx1:26:31

       Gamut of-

    9. AGAnastasis Germanidis1:26:31

       ... a full video.

    10. SWswyx1:26:31

        [laughs]

    11. AGAnastasis Germanidis1:26:32

        Um, yeah. There, there was a misconception, obviously, you can generate it to our feature-line film. But the models of today now take a lot of references. They take... They are very controllable, and I think when people see a, a tool that allows, affords many degrees of freedom and control, they respond to it differently, and it matters less that it's a generative model than, um, than the fact that you can actually steer it to the direction that you want. Um, and so I think when people look at, you know, complex workflow on top, on top of those models, when they look at, you know, all the ways in which you can steer them and you can provide now with some of the latest models up to 50 references, like, the conversation becomes a bit different because it feels much more like a-

    12. SWswyx1:27:18

        Storyboard

    13. AGAnastasis Germanidis1:27:19

        ... a tool-

    14. SWswyx1:27:20

        Yeah

    15. AGAnastasis Germanidis1:27:20

        ... versus, like, something that a magical kind of entity that figures out, like, the-- your entire film for you.

    16. SWswyx1:27:27

        Any notes on, like, workflows changing for people in the field? Like, I think engineering at least has had a lot of people where their, like, expectations have changed. I'm, you know, 10X, 100X more productive, and you can get a lot more done. Uh, same thing as, you know, you're making dev tools for creatives. Um, any notes there? Like, there's some people that don't wanna adopt, some that do. Like, anything?

    17. AGAnastasis Germanidis1:27:52

        Yeah. So I, I think, uh, in terms of, like, what people care about, uh, I see that we have gone through a few stages. So we started from a stage where the main thing that people were looking for was quality. Like, uh, as, you know, we scale those models, then pr- the quality improved dramatically. That's something that people still care about, but it's, it's now in addition to controllability, like being able to steer those models with references, with, um, you know, different kinds of inputs, with storyboards. And now my sense is increasingly people are gonna care about latency more and more. As those models become better, the ability to iterate very quickly becomes more important. And, like, if you can, you know, with a single prompt generate 10 different, uh, outputs, like, almost instantly, you can explore way faster than before, and you get some of the magic that characterized the, you know, the creative tools of the past, like Photoshop was instant. Um, and we, we lost some of that with generative models. You, you're waiting for two minutes to get back a video, and I think we're gonna bring, uh, some of that back now with, uh, with the, the-

    18. SWswyx1:28:52

        Real-time

    19. AGAnastasis Germanidis1:28:52

        ... those real-time models.

    20. SWswyx1:28:53

        Yeah. Exciting. And-

    21. AGAnastasis Germanidis1:28:55

        Exciting.

    22. SWswyx1:28:56

        Uh, the last thing we'll plug is this one, uh-

17. 1:28:58 – 1:38:05

    Physical AI and the Future of World Models

    1. AGAnastasis Germanidis1:28:58

       Runway AI Summit

    2. SWswyx1:28:58

       ... Runway AI Summit. You're finally doing this in SF.

    3. AGAnastasis Germanidis1:29:01

       [laughs] Yeah. So, uh, we're very excited about this. So this is, uh, in late September, September 30th. We're doing a summit on, uh, primarily focused on physical AI and real-time video generation. We have panelists from NVIDIA, Physical Intelligence, the Botco, DeepMind. Yeah, it's gonna be, I think, a very interesting series of conversations. We'll, we'll try to make the, the panels really technical and, uh, elicit actual kind of substantive discussion and hopefully some, some interesting kind of disagreements and interesting debates on, on things. And, uh, yeah, the-- there's tickets available. Hope, hope people can join.

    4. SWswyx1:29:40

       Since you mentioned it, uh, what kind of disagreements and debates should people think about, or do you expect?

    5. AGAnastasis Germanidis1:29:48

       So it's things like, you know, the-there is, uh, one debate right now in the robotics world is, uh, VLAs versus-

    6. SWswyx1:29:54

       Yes

    7. AGAnastasis Germanidis1:29:54

       ... world action models.

    8. SWswyx1:29:55

       Okay.

    9. AGAnastasis Germanidis1:29:56

       Um, so there is labs that are really, really betting on one of those two directions. Um, there is, like, what is the best source of data to train robotics models? Um-

    10. SWswyx1:30:05

        There's just the third party, first party thing that we talked about.

    11. AGAnastasis Germanidis1:30:08

        Yeah. There, there is, you know, the people who really believe in kind of fu-further scaling teleop data versus leveraging more large-scale video data. So th-those are kind of some of the... And then there is, you know, the, the world models debates of predict pixels directly versus something like JEPA versus a more 3D-based, uh, 3D-based approach. Um, so I think we're at a nice time in world models because there is still that kind of active debate happening of, like, what is the best long-term direction. I feel very strongly that this [chuckles] video predict pixels directly and, and scaling video generation models is the right approach. But it's-- I think there is a lot of interesting, uh, debate happening, uh, by, by researchers on, like, what is the best, best, best, uh, kind of path to take.

    12. SWswyx1:30:54

        It's interesting that it's a-all on, like, sort of, let's call it the policy layer and the data model layer. Is the, the physical side is completely solved? Like, all the sensors, all the actuators, all these things are-- We, we have everything that we need?

    13. AGAnastasis Germanidis1:31:07

        I don't think that's, uh, solved either. [laughs]

    14. SWswyx1:31:09

        [laughs]

    15. AGAnastasis Germanidis1:31:09

        I think there's definitely, um-

    16. SWswyx1:31:11

        Different problems. You know, like, it's like-

    17. AGAnastasis Germanidis1:31:13

        Yeah

    18. SWswyx1:31:13

        ... I wanna dream about all these things, and then I get, uh, you know, I, I, I buy a robot, or I buy, I buy-- I try to assemble my own, and I can't even get the-

    19. AGAnastasis Germanidis1:31:21

        [chuckles]

    20. SWswyx1:31:21

        ... you know, the, the, the motors to, like, work right, right? Like, and it's you're dealing with very sensitive, uh, equipment that has, uh, voltage and power and, like, heat and [laughs] all these things which y- you know, abstracted away, we're sitting here, we're talking about software and talking about models. But, like, really you have to deal with those kinds of things too.

    21. AGAnastasis Germanidis1:31:41

        Yeah. And, uh, I think I'm, I'm, I'm generally also not the Not opposed to incorporating other modalities into our models like we've seen.

    22. SWswyx1:31:48

        Yes.

    23. AGAnastasis Germanidis1:31:49

        The simplest case is they can generate video and audio at the same time, so they can generate RGB and they can also generate, uh, they, they can generate sound and audio. But my-- You know, I, I've written about this as like what i-- what does the maximalist version of a world model look like is you're incorporating more and more modalities from the universe-

    24. SWswyx1:32:09

        Mm.

    25. AGAnastasis Germanidis1:32:09

        -and you're training a model on different scales of obs-observation as well.

    26. SWswyx1:32:13

        X-rays.

    27. AGAnastasis Germanidis1:32:13

        So yeah. Uh-

    28. SWswyx1:32:14

        You've got a good, uh, good essay that people should read on-

    29. AGAnastasis Germanidis1:32:17

        Yeah. Yeah

    30. SWswyx1:32:17

        ... real-world simulation. No, uh, Meta released a model that was like six modalities in one, right?

Episode duration: 1:38:05