No Priors Ep. 2 | With Runway ML’s Cristobal Valenzuela

1. 0:00 – 1:50

   Introduction

   1. SGSarah Guo0:00

      (music plays) Chris, welcome to the podcast.

   2. CVCristóbal Valenzuela0:07

      Thank you for having me here. I'm super excited to chat with you.

   3. SGSarah Guo0:10

      So, can we start all the way back? I think you are the only person I know with degrees in economics, business, design, and then also went to art school. How did, how'd that happen, and then how'd you stick an interest in ML in there that became very real at some point?

   4. CVCristóbal Valenzuela0:23

      Yeah, that's an interesting question. I've always been very curious about just things in general, and so I've been trying to, like, uh, find ways of channeling that curiosity. I'm originally from Chile, and, um, I studied in Chile a combination of, like, business and econ, and then went into design, and it was a very particular design kind of like program. I spent a lot of time with physical computing, which is like working with hardware, with, like, um, electronics, mostly applied to design and, like, art. And while I was doing that, I was also consulting, so for a moment I ha- I thought I had, like, two lives. Uh, I was, like, doing art, uh, on the one end, with, like, Arduinos and electronics, and on the other side I was, like, consulting for these banks, which was very, like, different. But I love it. I think it's, it's, it's perspectives and worldviews that are very opposite at the same time you gain from being at both. And around that time, I just spent, like, three, four years doing that in Chile and started, like, teaching myself software engineering, like programming. That was just driven by curiosity. Uh, again, just, like, exploring or experimenting, which I think has... I would consider a constant, like, characteristic of how we think about what I wanna do and what I wanna, like, learn. And it also has been combating in the company itself, in, in Runway. Long story short, I, I kind of, like, fell in love or was experimenting with, um, early computer vision models in 2016, '15, and then went into a rabbit hole. S- apply and got a scholarship at NYU, and then spent,

2. 1:50 – 6:46

   Cris’s background and how he doesn’t see barriers between art and machine learning

   1. CVCristóbal Valenzuela1:50

      like, two years in, in art school. ITP, uh, that's the name of the program. It's a very unique program. Happy to go deeper into (laughs) , into that particular program b- because for me it was very fundamental kind of, like, piece in, uh, in my career of, like, understanding how to bridge business, design, art, and, and, and technology in a cohesive way. But yeah, I, I think just curiosity, I think, drives me a lot.

   2. SGSarah Guo2:10

      Amazing. And, uh, now you have one life that combines those.

   3. CVCristóbal Valenzuela2:14

      (laughs)

   4. SGSarah Guo2:14

      But in the art side, how should I picture Arduino electronic art?

   5. CVCristóbal Valenzuela2:18

      Media arts probably is the best, uh, way of, uh, describing it. (laughs) I think I was just interested in the wild y- the, the kind of, like, experimental side of technologies. How do you take, like, this, this hardware, these systems, these networks, and, and build interactive experiences? So I had the chance of presenting at a few festivals, uh, with, with, actually with one of my co-founders. There we were at, at Ars Electronica, which is this large, like, um, electronic arts festival and media arts. I think for me, media arts is a way of, like, expressing a worldview, um, using technology like any other form of art. Like, you just kind of like experimenting and, like, reflecting and, and, and expressing a worldview using a piece of, like, a tool. And in this case, like, it happens to be that we like to express it via, like, computers and software, and writing software is a form or an art- of art, and write- making hardware is also s- form of art. One thing I've, I remember early on in my career when I was dabbling between, like, art and, and business, I met this very famous Chilean artist that, he's a photographer, and he was just, like, mentoring me, and, like, we were chatting, and he, he was speaking to me, and he was like, "How do you think about doing these kind of, like, installations?" And like, "I was exposing it at a museum there, like National Art Museum in, in Chile." And I told him, like, "It's, it's a wonderful new world. Like, I've never been exposed to this world." And I remember the answer, it struck with me forever, which was like, "Chris, this is the same world. We all live in the same world, right? It's the same, right? We just build, like, silos and, like, arbitrary definitions of what is what." And it, it just really stuck with me. I think he just said it like he wasn't really thinking about it, but it, I really, like, stuck with that. And, and I think that's, that's how I like to look of the world. Like, it's, it's just the same world. You can apply different points of views and perspective on that. We build arbitrary, like, definitions of, like, "This is, this is art, and that- that's, that's, like, design, and that's econ, and that's business." But I think true creativity and curiosity comes from just, like, looking at it as a whole, and taking things that weren't supposed to be part of one thing and then adapting them. And sometimes it's hard, because you need to, like, learn things that you've never done before, and it's uncomfortable, and it's perhaps you feel like, uh, an imposter. Like, you haven't, you shouldn't be doing this. I've learned not to care, to be honest. (laughs) I just, like, just drive by curiosity, like, you'll figure something out, and I really like that.

   6. EGElad Gil4:32

      That's super cool. Yeah, it seems like a lot of the history of Silicon Valley actually ties in really closely with art and the art scene. So if you go back to, like, the Stewart Brand world of the '70s or some of the early things that were being done on the Mac, or even look at some of the people in technology where the, the art side of them is understated, you know? Like, Paul Graham obviously wrote a whole book on this, Hackers and Painters, and is a painter himself, but there's people like Seth Kombar who started a company, the Google Bot, and has done a lot of crypto-related things. He was a co-founder of Celo. And he, he's exhibited digital art at the MoMA as well, and so it just kind of feels like it's almost under-discussed now in terms of this overlap between technology, art, and the two scenes, except for, you know, occasionally when people go to Burning Man or something, they bring it up.

   7. CVCristóbal Valenzuela5:08

      (laughs)

   8. EGElad Gil5:08

      But other than that, it seems like it's very under-focused on.

   9. CVCristóbal Valenzuela5:10

      Yeah, I agree. To be honest, I've been in New York, like, six years, and at Runway now it's gonna turn four years. And I was also new to, like, just the, the tech world and SF. Like, I've never been to SF, like, three years I go, right? So I'm relatively new to, to the space, and... But I think what, how we approach it was with that same level of curiosity of like, "I'm gonna figure it out, I'm gonna learn about it." And I think that, uh, that there's two, like, sides of that. The one is that it takes time for you to adapt to that, because it's just new. Like everything else, you just, you need to understand it, you need to understand the patterns of that, that subject, that domain, that area, right? But at the same time, I'm looking at it from eyes, with fresh eyes, with things that the ecosystem itself has considered, like, norms, I've, um, I don't consider them norms. I go just like, "Well, I'm gonna try new things," right? And I think that opens the door, again, to do new things and experiment with new things, and that has, I think...... been a consistent, like, path in both my career, but also in Runway as a whole, that we would look at things, we tried to look at things with, like, very fresh eyes, and like, pretty much with, like, a first principles kind of mentality to it. It was like, okay, why are we doing this, but really why? And then go to the basic aspects of it, and then innovate. I think that's ... a lot of innovation comes basically from, from kind of that way of looking at the world.

   10. SGSarah Guo6:22

       Runway's a, a, I think a very creative shape of product. It's not the kind of product you can come up with if you're just, like, casting around for a good idea. It, uh, it obviously comes from creativity and discovery, and maybe what you could do for our listeners actually just, they should go try it, but can you explain how Runway works as a tool, and what people do with it, to set context?

   11. CVCristóbal Valenzuela6:41

       Yeah, totally. Also happy to set a bit of context of, of the company itself, so I think that better

3. 6:46 – 8:36

   How Runway works as a tool

   1. CVCristóbal Valenzuela6:46

      helps contextualize the product itself. For the best way of describing Runway, I would say is to think about it as a, a deeply AI research company. We do core fundamental research on, like, neural networks for both content creation and video animation and driving models. We then transfer those models into an infrastructure, a system to deploy those algorithms and systems in safe ways, and in, in, in, in ways that will make us build products that are useful for people, right? And, and those products can take different shapes and forms. We have around 35 different, uh, what we call AI power tools or magic tools, and those tools help serve a wide spectrum of creative tasks, from traditional like editing, um, editing videos, or just audio or images has been a very expensive, time-consuming, and sophisticated process. And so we build systems that help you do that, so we have tools like Green Screen, for example, which, which a lot of, uh, broadcasting companies and film studios and post-production companies use to reduce the time of rotoscoping, which is if you ever speak with a filmmaker, that's, that's the one thing no one wants to do, and just no one wants to do it, but you have to do it. And so we basically just help you reduce the time. And we also have tools that help you ideate and, and design and craft, and we have a, a set of, like, suites for generative image editing, for generative video editing. The, the best way perhaps to think about it is it's a creative collection of tools and systems that just help you augment your creativity in, in any way you want.

   2. SGSarah Guo8:15

      From an origins perspective, like, you had this thesis project, which were all of these creative tools, and it was really ... I remember, like, watching the presentation. It was around accessibility of, you know, the increasing number of algorithms that could help people in this sort of, uh, creation and editing process for different modalities. Then when we met in 2019, you framed it quite differently as this kind of desktop

4. 8:36 – 12:22

   The origins and early iterations of Runway

   1. SGSarah Guo8:36

      app store for ML models. Can you talk about the iterations from that collection of algorithms that you were experimenting with to, like, the app store idea to where Runway is today?

   2. CVCristóbal Valenzuela8:46

      Yeah, totally. A lot has happened, I would say, over the last decade or so. When I started building Runway, it was perhaps the AlexNet, ImageNet kind of like moment. There was ... image classification was the, the, kind of like the big thing and the breakthrough, and a lot of interesting applications were coming out of that time. It was still very early. Like, TensorFlow was just perhaps a year old. PyTorch might not even have been released at the time. I think PyTorch was 2016. GANs were just, like, very early, early, early, like inception time. But what I kept seeing was there's this neural, like, aesthetic, these neural-like capabilities that are impacting not just, like, the visual world, or like, the, perhaps industries and markets like, uh, self-driving cars that are using a lot of this technologies and hardware. But the outputs are very interesting from a visual perspective, right? There, it seems to be a correlation and a approximation towards the visual domain. And so I started just experimenting with what, what, what does actually that mean, right? What do you mean by ... how do you experiment with this sophisticated algorithms that were very early, that had all this, like, obscure CUDA dependencies and, like, C++, like, libraries that were just very research-centric, uh, because they were basically s- research, right, like core research? But I was just fascinated by the outputs of the research elements. At the same time, in everything I would say that we consider like a baseline today wasn't really, like, there yet at the time. Things have progressed radically. The space has been growing exponentially, but, uh, systems and, like, software and obstructions to tap into that potential wasn't really there. So our first, um, intuition and our first kind of, like, product experimentation was let's build, like, thin layer, right? Like, basically let's take this, this research set of, like, models and the amount of models that are coming out, which is, like, so interesting. Let's add a thin layer of, like, accessibility to those models, specifically target and aim at creatives, right? And so if you're a designer, a filmmaker, uh, an art director, a copywriter, you might wanna tap into some of these things, so you wanna, um, experiment with them, but they're just very hard to get started with. So we built, at the time was, as you're describing, like a model directory. It's an app store of models, right? You had... We had around, at some point, like, 400 different models. It was one of the first, like, I would say model hubs. Uh, I think there are a few out there now that y- you can, like, tap into and use them. This was, like, very, very early, and we built a whole system around it. We built an SDK. We built systems for, like, deploying those models, like, into real-time applications, so we built, like, uh, RESTful API systems where you can use a model, train a model, and then deploy that model. And so people were building web apps and, like, interactive, like, it was GPT-1. Someone was training a model and, like, fine-tuning a GPT-2 model on a specific corpus of data, and then creating an API to build, like, a text generation app. And we had all these, like, very interesting kind of like layers of, like, applications that, to be honest, for us, was just a way of learning, um, uh, learning a lot about the space and a lot about, like, what was feasible, what was possible, who was kind of, like, interesting in, like, building more of this. And from there on, we've kind of, like, continuously iterating. We've learned a lot from that model registry or, like, model hub. Um, we still use a lot of those in our infrastructure kind of like parts on, on the app, but also we gather a lot of insights on, on how to build these kind of systems in, in scalable ways.

   3. EGElad Gil12:02

      How did your technology stack or the approaches that you took, uh, transition over time? Because I think when I look at the evolution of the area, to your point, you know, a lot of people are doing, like, CNN and RNN based things, and GANs, and all the sort of early things in neural networks. And then, the analogy may be, I know a lot of people who started companies right before AWS launched, and their whole, like, infrastructure

5. 12:22 – 15:43

   Product sequencing and roadmapping in a fast growing space

   1. EGElad Gil12:22

      stack got stuck on the past set of approaches, and then later, s- a subset of them transitioned onto AWS, and then subsets just continued with their own private clouds. And I'm just sort of curious how you thought about it, as, you know, obviously diffusion models I think were invented around 2015, transformers, 2017. But it took a couple years for all this stuff to catch on. And so, when did you start transitioning architectures, or have you, or how have you thought about this sort of whole evolution of the field relative to the tools that you provide, and reinventing them over time and everything else?

   2. CVCristóbal Valenzuela12:48

      No. Th- that's, that's a great question. Something obviously we think a lot about when you think about, like, product sequencing and roadmap, which is just, I would say, one of the most important aspects of product building is, like, how do you sequence everything you have to do? And specifically in infrastructure, like, w- what makes the most sense and how do you spend time, like, every single day, like, means a lot in a startup. I think for us was, uh, a few realizations, to be honest. One is that the moment something gets released, like, let's say, transformers or a particular piece of technology that you think would be interesting or could be worth experimenting with, I think it takes a collective set of months, like 12, 24 months sometimes, to understand the implications of that, right? And, and we've seen this with, like, language models, like, GPT-3 has been around for some time, but it took, like, a collected 24 hours of, 24 months of, like, just tinkering and experimenting to truly understand, like, okay, where can you go, and what can you build, and what's possible? Uh, so I think that we embedded that, and we always keep that in mind. The second thing I would say is things are changing really fast, right? And so if you're thinking about building a long-term business and a long-term product, which, which we are, you always have to decide of like, okay, what are long-term bets versus short-term bets? And I think a lot of building, and software engineering, and, and, and developing products is just saying no to a lot of things. Just, like, customers might wanna ask you for, to build something, and could sound good. It could bring you actually revenues and growth, but it actually might move you away from, like, a more consisting long-term plan. I think for us was, was a realization of those kind of, like, things. And then the third one I would say is, the third component of, like, of, of how we think about that stack is really understanding our users, right? Who are we building for? Um, and so early on, it was more a technical product. So you had to know CUDA, and Docker containers, and managing your Docker NVIDIA GPU cards. And, like, you have all this, like, sophistication that I think it's in some part natural when you're- things are so early, because it's just the only way of making sense. And also you have to build more things. But for us, we've always been thinking about artists, and, and, and filmmakers, and creatives at heart, and really those things don't really matter that much. What matters is, like, your idea and how you execute that idea. And so from the stack perspective, we've iterated a lot on the, the kind of, like, backend side of things. But from a user perspective, we iterate even more on how to present those things, and what abstractions and metaphors you need to build to really aim to solve the things that you wanna solve. But yeah, it's a fast-growing space, so there are a lot of things that are changing.

   3. SGSarah Guo15:14

      In an area where the, the research, like, nobody can keep up with the papers, right? Um, the progress is mind-blowing and has been, uh, you refer to Runway as, I think, an applied research lab. Is that the right term?

   4. CVCristóbal Valenzuela15:27

      Yeah.

   5. SGSarah Guo15:27

      Like, where do you decide, given the progress in the community, like, when you need to do in-house research and push the state of the art versus exploit what's out there?

   6. CVCristóbal Valenzuela15:36

      Yeah. I guess going back to that, like, that set of learnings early on, um, I think one thing that we realized is, um, models

6. 15:43 – 19:10

   Runway as an applied research company

   1. CVCristóbal Valenzuela15:43

      on their own are not products, right? A m- a model is, it's, it's a research component. And, and taking a model and productionalizing that model, it's, uh, it's a different problem that actually building one single model, right? Or one single task, or problem, or, uh, improving a metric in a specific kind of direction. There's a lot of nuances of how that model will get deployed, will get, uh, built, how users would interact with it. The unit economics of, like, running these kind of, like, systems as well is very important, right? So they have all these complexities. And as we started, like, leveraging perhaps open source solutions at a time or trying to build our own, we kind of, like, quickly realized that having control is, like, key. Like, you need to be sure that you can understand your stack and you can understand and know how to fix your stack, right? Because if things are changing really fast and you think about going in one particular direction, but it then happens to be the case that there's a breakthrough somewhere else, you need to react really fast, right? And you need to be able to incorporate that. And if you're just relying on third parties or, like, just, uh, some other solutions, then it might be very hard, right? And so for us, it was a survival kind of, like, realization that if we really wanna make and move the standard of, like, creative tools in the ways and, uh, vision that we had, we had to own our stack. And so we started building this research team, right? And this research team has very deep, like, understandings, and knowledges, and perspectives on how to build models. And we've, we've done this when we- we've collaborated and contributed to, like, uh, breakthrough moments in, like, the creative AI space. But most importantly, we have these researchers working really closely with creatives. Like, half of our team have arts backgrounds, right? Which is, which is very unique. And we put a lot of emphasis on finding those, those very unique, like, they're very hard to find folks that can speak both worlds, right? I just went back to the worlds kind of, like, analogy. And so in one single table you can have a, like, a PhD scientist that's been contributing to, like, fundamental research on the space, working really closely with someone who's working on video for 20 years, right? Who's been editing and post-producing films or content, right? And the things they learn from each other is just so, so unique. It's so radically different, and it helps inform how we build products, right? And so we don't treat research as a standalone kind of, like-... department that then comes every six months with, "Here's a paper," and just, like, do something with it. We see it as an applied thing. It's at the core of who we are and, like, how we drive, uh, the product forward. And it helps just drive the product in a different way. I think that the only thing I've learned is that building that muscle takes time, right? It's not that something you can just, like, "I'm gonna hire a bunch of, like, creatives, and a bunch of researchers, and just put them in a room," and, like, you'll figure something out. It's, it's a lot of, like, learning, and, like, a lot of processes, and, like, frameworks of how you make decisions, how you understand what's worth, what's really possible versus what's feasible, and there's a lot of adjust, just, like, nuances of how to do that.

   2. EGElad Gil18:42

      Yeah, it seems like there's a lot of, um, founders now who come from the research community in the AI and ML world. And, you know, you've navigated that extremely well in terms of saying, "Okay, let's be very product-centric and yet still capture the best of what new technology has to offer and new research has to offer." What do you think are common pitfalls that research-centric founders should avoid, or things that they should think about more as they sort of start their own companies?

   3. CVCristóbal Valenzuela19:03

      Yeah. I, I think it's just phenomenal to see, like, the, that progression of more researchers that been perhaps in academia for too long

7. 19:10 – 22:35

   Common pitfalls for founders to avoid

   1. CVCristóbal Valenzuela19:10

      progressing or moving into, like, just the operational world, like building products. I think it's a great realization of, like, you're working on something for six, eight months, a year, but you see something else in the world of someone using something very similar to what you just built, and impacting the world in very, like, meaningful ways. I think this is, that's great to see people transitioning more. I think we need more of that. I still think that there's, um, a lot to be learned around the difference between a model and a product. And again, it, it, there's a lot of, like, back and forth of how you embed models into usable products. And so coming up with training a model or improving some sort of, like, quality of benchmark in some particular way, even you have a very cool demo, it's a long way to go to, like, actually build a business, and, like, a reliable, like, system that it will continuously iterate over that. And so I think having that more product perspective is always just, uh, good. And releasing and working with real people as fast as you can, I think that's just key. I think a lot of researchers just assume how people work (laughs) and how creatives work, and just say, "Oh, we'll just do that." But, like, the realities might be very different.

   2. EGElad Gil20:16

      Mm-hmm.

   3. CVCristóbal Valenzuela20:16

      And so having, having tools being used by people is, I think, the best way of learning how to, uh, develop products.

   4. EGElad Gil20:23

      Are there specific areas of research that you're es- especially excited about when it comes to video or images right now?

   5. CVCristóbal Valenzuela20:28

      Yeah, for sure. I think, I mean, everything we've seen on the explosion of diffusion has been just so exciting to see. I think I'm particularly excited about multi-modalities and, like, combining, uh, different kind of, like, input or, like, outputs in, in ways that they are yet to be explored. I think we're moving away from, like, very siloed, like, domains. So, like, someone who could be an NLP researcher and computer vision researcher, right? I think we're, like, starting to, like, see them gradually converge and mix. And so building a diverse team that can understand, like, those multi domains is really interesting, and I'm excited to see how that's gonna play out in video and, and in images. And I'd like to think also of how you translate... Again, I'll go back to product. I'm, I'm product obsessed. But how you translate that into, into products that are, that are useful, right? I think a, a common, a common natural evolution of just the creative stack or the creative software solutions out there, they tend to be very specific to domains of, of content. So you have a tool that's specialized on, like, image editing, and then you have a tool that's specialized on vector graphics, and you have a tool that's specialized on motion graphics, which is different from video editing, which is different from, like, post compositing, which... And you have all these, like, very sophisticated, like, software stacks. And I think the, the very interesting aspect of, of what I would like to see and what we'll probably see more with multi-modal systems is that you're able to merge all of those. And what I really find interesting about that is that's how we humans think, right? We don't... You don't go to a movie and watch the, the video first, and then you stop and you hear the audio, and then you stop and you read the subtitles. It's a combination of all of those things, right? And an art director thinks in all of those things at the same time as well, right? So having systems that can translate ideas and text descriptions into videos, and then having a conversation with what's the input of those videos into, like, audio, and then I think that's the, the kind of, like, creativity and set of tools that I'm really excited to, to discover and build.

   6. EGElad Gil22:20

      How do you organize your product (...) ? Because I think, to your point, you have a really unique approach in terms of effectively turning research into products, or being product-centric in terms of what you're asking from the research organization. Is there a specific structure? You know, for example, um, at one of the companies I started, Color, we basically would embed somebody

8. 22:35 – 24:22

   How Runway structures teams for effective collaboration

   1. EGElad Gil22:35

      with a very deep bioinformatics background with the systems team, so that they basically inform that team around the needs of what they had, and then the rest of the team would build it. And it sounds like in your case, you have people who kind of are in both worlds. Is there a specific structure where you're like, "I always put three, you know, full stack engineers with a researcher, with a product person," or the, the researcher is the product person? Or how do you kind of approach all that?

   2. CVCristóbal Valenzuela22:55

      Yeah. We're a small team. We've been consistently, historically a s- small team. Until, like, two weeks ago, we didn't have a product person. Product was led by a combination of research, design, and engineering. And I think that drives a lot of, uh, fundamentals of truly understanding the things that need to be explored. We've iterated a lot on building squads, or building, like, teams, or having more autonomy. I think it really depends. I think you have a, you tend to have a different company every, like, four or five, six months. It's if you've successfully built stuff, it con- it's a continuous, like, process. And the thing that worked when we were, like, five people sitting at a table, it's not gonna really work when you're, like, 20 and you have new technologies and system things available. And so I don't think there's one answer in particular. I think we're pretty much with how we think about product, we like to iterate a lot. Right now, we're working a lot with squads. And so we've, we've come to a kind of, like, a place in time where the organization can have a bit more of, like, domain expertises. And, like, instead of having, like, very generalized engineers, we, we tend to, like, more specialize a little more. So you can still jump and be and collaborate, but you tend to have a bit of a focus of area. And we iterated with that and seen how that works.

   3. SGSarah Guo24:01

      Maybe we can talk about an example of, like, what that iteration looks like. So you mentioned rotoscoping, and, like, green screening as, like, a, uh, like, one of the magic tools, uh, Runway creates. When you were, when we were building that feature, like, what was hard? What were the iteration processes like?

   4. CVCristóbal Valenzuela24:19

      I, I think that Greenscreen is a great example of

9. 24:22 – 28:01

   Learnings from how Runway built Greenscreen product

   1. CVCristóbal Valenzuela24:22

      how to build and how to deploy useful, like, AI products at scale. Like, if you've... If... When we were building that model directory, and we're just, like, early stages of understanding limitations and capacities and directions, we quickly realized that the type of, like, user that was coming for segmentation models, right? And at the time, we didn't have a greenscreen tool. It was just, like, a image segmentation model. And those folks were coming from a specific domain, and they were actually applying a model that was image-based into a video task. And so they were, like, exporting themselves with FFmpeg, creating these, like, sequences of images to then render them back in video, and it was like, "Why are you doing that?" (laughs) "Why are you... What's going on?" And the thing is, like, image models don't really work really well with, like, video, right? You have the temporal consistency component, and, like... So it's really hard. And so we started interviewing them, and we, we, we got to a point where it was like, "Oh, wait. It seems like this, this could be something we could, like, improve." And we're bringing in our research team, so we started, like, iterating more on that. But no one ever asked for a one-off, one-click solution for greenscreen, right? If you ask people what they wanted, they wanted a better alternative that was faster to create mask from their ex- current stack, right? And they're probably using something like RotoBrush too, right? So, uh, "Whoa, what I would really like would be, like, a better brush to just brush over my frames," right? And I think customers and people, um, are really good at telling you, like, what their problems are. (laughs) They're really hard at verbalizing, like, solutions. And so you aggregate that amount of data. You see what's possible for research. You see, you chat more with people, and you start prototyping a lot. And then we came to, like, the realization that we could build and we have the expertise to build a system that will help you automate that, right? And most literature around video object segmentation, which is the, in filmmaking, is basically known as rotoscoping or greenscreen, right? Was around, like, fully automated systems, right? You fit in a video, and the video automatically, like, understands, like, subjects, and then rotoscopes are segments, right? One specific central object, or two, let's say. But, like, just a few minutes of chatting with a professional filmmaker will... He'll probably discover that that's he- rarely the case, because the shots, the scenes, and the compositions, and the camera angles really depend. And, and you might wanna... If you have a shot of 10 people, you might wanna rotoscope the one on the left, but maybe you want the arm from the person on the left, and maybe you want the... Depending on your idea, right? It's a s- it's a creative, like, tool, right? So it can be, should be general. And so what we did was we've... Instead of, like, relying on fully, uh, uh, automatic systems, we embedded a human in the loop kind of like component in it, right? Um, and we thought it would be great if before you start doing that, you can guide the model. Like, you can tell what kind of, like, selections or areas of the video, and you can zoom in and define who you want. And that also really helped us train the model, because we train a model on... We built a probabilistic model of, like, human simulated, like, human clicks, uh, on a mask, and the model was trained on that knowledge, right? From the very bo- bare bones. And that helped the, the product itself, because people were using that model in that particular way. Um, and that whole decision was, I would say, like a combination of different things. It was research, like, some research knowledge and understanding of what was feasible. Can you build a segmentation model? What datasets and what, what, what do you need to do it? Who would be using it for? How are we gonna test if it works? And the first version of Greenscreen was working at, like, four frames per second, right? It was, like, incredibly slow. (laughs) It was, like, not as good as the one we have now, which is incredible, but it didn't matter. It was significantly better than anything else that was at the time, right? And people were, like, scrambling to use it, just because it proved to be a percentage

10. 28:01 – 32:34

    Building a long-term and sustainable business

    1. CVCristóbal Valenzuela28:01

       of, uh, um, amount better than anything out there, right? And people were hacking things, and they were trying to, like, incorporate, and it's like, "Great. That means that, like, that you've hit something." And then we started iterating a lot. And so we keep iterating a lot on it, but the fundamental piece of how we build products is still pretty much similar to that.

    2. SGSarah Guo28:18

       Very cool. Let's zoom out and talk about Runway as a business. So you, as you said, now you're very intent on building, like, a, a long-term durable business. Who uses and pays for Runway today?

    3. CVCristóbal Valenzuela28:29

       Sure. Um, again, we're devoted to, like, storytelling and, like, creative exploration and, and ideation. And, and that's a wide spectrum of people who are... You can consider work in the storytelling business, right? (laughs) On the one end, you have professional, really professional people that have been doing this for years, right? Folks working in post-production agencies, VFX agencies, um, broadcasting companies that are creating video as their main business. Like, this is basically what you do, right? It's entertainment. It is sometimes sports. We have a lot, lot of sports companies.

    4. SGSarah Guo29:01

       You know, that's kind of counter... That's kind of counterintuitive, because, like, one of the sort of, uh, beliefs of many people who look at the research, which is fast progressing, is like, you can't get the quality level for like, the sort of highest production value type assets with, with today's research. So it's really interesting that, like, you know, you're talking about V-S- VFX studios and, and sort of that type of content.

    5. CVCristóbal Valenzuela29:23

       Yeah. I think, I think that realization for us is like, what the goal is, right? If you're trying to automate the entire process of like, the whole end-to-end system of making a movie. Yeah. Like, we're not there, right? We're very far from that. There's a lot of things to be... That have to be developed, that have to be the, the, like, researched, and kind of like understood and tested. But going back to the greenscreen, if, if you look into the processes and the nuances of how video is created, and you look at the inefficiencies of how people are doing it right now, and you offer these people like, 100... Even like 10% or 20%, or like, whatever percentage of like, speed and cost reduction, it's just so radically better, right? And it's radically better for two reasons. Of, of course, it has helped reduce the cost of like, you have... You have less... You have... You can do things faster, so it's just easier. At the same time, you can express creatively more, right? And this happens a lot. I was speaking with this director who was, like, working on a film and was using Runway, and he came up with this idea of like, when he was...... chatting with this editor. He was like, "We should just Runway that," right? "Just Runway the thing that you wanna, like, do." And before Runwaying something, they had to marry themselves or, like, just lock one specific idea, right? It was like, "Let's, let's try doing that with that character on the left. And that's it. Because if we try to do two other things, it's gonna take us too much time and we just don't can afford that." Like, every creative is always on a deadline. There's always something to do it on.

    6. SGSarah Guo30:50

       It was very waterfall era, right?

    7. CVCristóbal Valenzuela30:51

       Yeah. (laughs)

    8. SGSarah Guo30:51

       You must choose a direction and do the whole thing, yeah.

    9. CVCristóbal Valenzuela30:53

       Exactly. And now, now he was telling me, like, "Now I can do the three, right? I can just see the three and pick the one that I like the most, right? It's, I'm not constrained by the time and the cost. I'm constrained by whatever idea I think works the best." And that's just phenomenal, right? And so our goal, and I think our goal still is, is not to, like, build this, like, s- kind of, like, autonomous, like, systems that don't engage in any sort of, like, relationship with humans or with creatives. On the contrary, it's like, you have humans coming up with great ideas, and they wanna express those ideas. How do you build systems that will help them get there really quick? And sometimes what you need is to get 80% there, 90% there. And, and research going from 80% to 100% is really hard. I think that you've seen that in, like, autonomous vehicles where, like, it's always, like, two years ahead and it's, like, always 80%. But like that, that 20, 10% is just really hard. It's just really hard. But in, it's really hard in, in that domain because if there's a 1% failure, like, s- someone might die, right? In creative domains, it's not the case. Like, it's even if you're 80% there, the 20%, like, sure, I mean, you could worry about it. I can improve it. I can, like, find ways of, like, work with that. But you've, you've made an incredible progress from that perspective.

    10. SGSarah Guo32:02

        I think that's actually an interesting, like, filter for what domains are interesting for applied research today. Like, areas where there's built-in tolerance for, you know, lower levels of accuracy, um, is, is one way to look at it.

    11. CVCristóbal Valenzuela32:15

        And you always integrate with... Uh, there's ways of, like, combining existing tools, right? So for RotoScape, for example, you can get 80% there. And then if you're a f- professional filmmaker working on Nuke or Flame, you can do the 20% in that stack, right? But you still saved yourself, like, days of work, right? So it's still better

11. 32:34 – 36:34

    Finding Product Market Fit

    1. CVCristóbal Valenzuela32:34

       than anything you were using before, right? So it d- it depends a lot. And I think over time, more models will get to, like, higher numbers and will have higher outputs. But there's a lot yet to be developed. And I think we're still scratching the surface of, of what's coming.

    2. EGElad Gil32:47

       What was the moment... Uh, you know, you mentioned there's an evolution both in terms of the number of tools that you provided as well as their relative quality in terms of 80% versus more or less and things like that. Was there a specific moment where you really felt that you had product-market fit, or where you felt that, "Okay, this is something a lot of people want, and they wanna use"? Is that, was it immediate? Was it after a specific tool came out? Like, when, when was that moment for you?

    3. CVCristóbal Valenzuela33:08

       I like to think of product-market fit as a spectrum of, like, you have either really strong product-market fit or, like, weak product-market fit. And, like, as you build new products and new research, you're always seeking to be very on the strong side of things, (laughs) of course. I think for us, there are a few factors that we've kind of, like, realized that what we were building was beyond just, like, a niche because I think we started with a very niche, like, audience. And everyone, like, dismissed a little bit of what we're doing, like, as toys. It's just sort of like art students building, like, some toys. Uh, and I think you shouldn't dismiss toys. Toys are, like, very interesting to learn a lot. And I've learned that over time. But it's, by the time when you're building those, of course, it's just like you're focusing on the output, and they're glitchy, and they're, like, abstract, and it's just weird and can't make sense of it.

    4. SGSarah Guo33:51

       And it's only 128 by 128 pixels.

    5. CVCristóbal Valenzuela33:54

       Exactly, exactly. I was, I was actually, I remember, like, uh, we had a version of, like, very early, like, gun system that, uh, that did text-to-image translation. We, uh, we actually still have the demo online. And the outputs were so, like, this 128 pixel, like exactly what you're saying, uh, images that were just blurry. It looked like abstract paintings, right? It was just like... I mean, you type, I don't know, blue ocean, and you get, like, a blue form with something. So it was really interesting. If you close your eyes, like, 10 meters away, maybe-

    6. SGSarah Guo34:18

       And you saw beauty.

    7. CVCristóbal Valenzuela34:19

       I saw beauty. I, I really liked it.

    8. SGSarah Guo34:21

       (laughs)

    9. CVCristóbal Valenzuela34:21

       But at the same time, I remember, like, showing it to advertisers. And, like, I, I went to, like, this executive meeting at this top agency in New York, and I was like, "Here guys, here's the thing you will be using to work," right? And they were like, "Chris, this, this is a toy." Like, "Great, I mean, f- fascinating technology, whatever, but, like, we have work to do. Come on. Like, move on," right? And I think I, I... The, the main mistake for me was, like, you're looking at this singular moment in time of that technology. You should really be looking at the rate of progress, right? That thing that it can type a word and send an image wasn't feasible a year ago. It just didn't, didn't exist, right? Now we have this. So just compound and c- try to, like, imagine where we'll be in, like, four or five, six years, right? But the thing is, that's really hard because you can't imagine it. And I remember people at the time, like, when I show some of those demos, and specifically for generative models is people are asking me like, "Hey Chris, how are you collaging these images, right? You're taking existing images, and you're pasting them together, right?" And it's like, "No, this is... You're generating them. This, this model has learned patterns around, for sure, a dataset, and you're then generating them on the fly. But they're, these images don't really exist. They just don't exist." And so there's a lot of, I think, mental models that need to be adjusted to really understand it, and we've, we've been adjusting that, that, those mental models. And from a product perspective and from a product-market fit perspective, I think there's the right moment for the market to use the technology. And I think that moment has matured and we've seen it more as more people have been exposed to generative models and the potential of them. And for us, it's still like, uh, uh, there's a lot to build and to develop and to kind of like, uh, improve. But there are a few realizations where, like, when people were starting using Runway as a verb, right? You just Runway that. Okay, that, that means something. Then you start seeing people just, like, creating tutorials and, like, speaking about the product online, right? With no, we don't, we, we certainly for a long time never had a, like, a marketing team or a content strategy team. Like, everything was just basically people making things and then sharing them online. I think that really drives, drives, I would say, the realization, "Okay, we're into something. Like, people are using this every day. They're coming, and they're sharing with their friends, right? And they're, they're thinking about it every day." And they're like... I remember an artist and, and a person who early Runway adopter, which just fell in love, so in love with the product. He sent me like-... he painted a

12. 36:34 – 48:51

    The influence of AI tools in art as an artistic movement

    1. CVCristóbal Valenzuela36:34

       picture, like, and he just sent me the picture, like, to my, to m- to my home. He's like, "Here," he's like, "I just want you to have the first piece I ever made with AI." And that was, like, 2018.

    2. EGElad Gil36:42

       What was it? A cat?

    3. CVCristóbal Valenzuela36:44

       No, it was th- an abstract painting where it was like he- he generated something that was very abstract, right? And then he painted it on a canvas, and then he used, like, mixed techniques to just, like, improve some styles and, like, change some colors. And it was very new and novel at the time. It's like, what? Wow, that's just, I don't know, interesting and fascinating.

    4. EGElad Gil37:01

       So with all the, uh, wisdom of, you know, Chris four years in and Runway's going well, like, is there, is there anything you'd do really differently if you were gonna start from scratch today?

    5. CVCristóbal Valenzuela37:10

       To be honest, I- I think we're, I'm still learning a lot. (laughs) I think I- I wouldn't consider, like, oh, we're, that's it. We're, we've... I don't know. We have a lot of users and a lot of companies, and that we're- we're basically, like, chilling. Or on the contrary, I'm like, this is the time to be very bold and, like, continuously learn. Like, the one thing I think it's just great for everyone building in this space, um, at least for- for- for- for us from just on a very personal level, is I have to do less of the selling. Like, I don't have to tell people, like, "Hey, this is useful. This could be useful." (laughs) Like, it's just great to, like, have more of a conversation around, yeah, possible use cases. And it's also great to drive the conversation in very productive and positive ways and understand what's feasible and you get more people to build with you, right? I really, we really try, and something I'm- I'm, I've- I tend to code less now, but something I try to do a lot is just really think from a company perspective and cultural perspective. How do we keep those DNA elements of what makes Runway really unique, really, really unique, and really consistent as you grow? Because as you grow, there are other challenges, like organizational-wise of, like, communication and, like, structure, right? And hiring as well. Like, you need to bring, like, the best, the best people that... But there's also, like, needs to be, like, a cultural fit, right? Um, and that's, I'm really obsessed with that. I think what I've come to, like, the realization of, like, I code less, but I code the company in a way.

    6. EGElad Gil38:29

       Yeah.

    7. CVCristóbal Valenzuela38:29

       Uh, and I try to, like, f- find ways of optimizing that- that set of decisions.

    8. EGElad Gil38:34

       Yeah, it feels like every company eventually has two products, the product they sell and then the company as a product. And so to your point, I feel like a lot of founders eventually move into a mode where they're, like, engineering the company or they're designing the company, or they're roadmapping the company. And that's- that's a really important transition. And so it's awesome you're doing that. I feel so many people wait a very long time and then things start breaking because they didn't think ahead, just like you think about your backend scaling. You know, you have to have your company be able to scale.

    9. CVCristóbal Valenzuela38:56

       And I'm- I'm- I'm still, I'm still learning on exactly that. But I think just keeping that learning mentality has been, yes, useful for me.

    10. EGElad Gil39:03

        Yeah. One- one thing I'd love to get your perspective on simply because you have such a unique mix of- of background and skills and customers and everything else is, you know, there's this emerging debate, um, in the art world about the role of AI in art. And I think if you go back through art history, there's always been ongoing questions and contentious, not just around technology and art, but the role of an artist relative to the art they create. And I think, like, the sort of old school canonical example was Marcel Duchamp signing the urinal, you know, with R. Mutt and, um, I think it was called The- The Fountain or something, right? It was a piece that he...

    11. CVCristóbal Valenzuela39:36

        Yep.

    12. EGElad Gil39:36

        ... submitted and it got refused, and it created, uh, a bunch of, uh, sort of scandal at the time. Or, you know, Andy Warhol had, like, The Factory and other people would assemble a lot of the art actually (laughs) with sort of him overseeing it.

    13. CVCristóbal Valenzuela39:49

        (laughs)

    14. EGElad Gil39:49

        And so it seems like there's been a long history of sort of different approaches to art that, at the time, seemed very, um, controversial, and now you're just like, "Yeah, of course, that's- that's how you do things or how things were done." H- h- what do you think about the debates right now in terms of art and AI, and, you know, what do you think are the important threads that people are talking about? And what do you think are the areas that in 10 or 20 years people will look back and say, "Yes, it was just part of sort of this art history debate, but it in hindsight wasn't really that important"?

    15. CVCristóbal Valenzuela40:14

        I like to think a lot about, uh, what d- I guess previous moments in history and- and time as you were referring before that has taught us something about how to, like, both understand art and, like, look at the tools that we use for art. For me, art is- is the way of looking at the world and expressing that view of the world in a particular way, right? And- and an artist's role, I think, should be to explore and experiment with different mediums that would allow you to express that in the best way you think possible, right? Um, and so people experiment with different techniques and different systems and different, like, structures and- and pigments and, like, uh, tools themselves, right? Even before, like, Duchamp and even before, like, Warhol, you had previous moments in times where, like, technical revolutions enabled people to look at the world in very different ways, and then express those views of the world in very different ways to whether was feasible at the time or possible at the time. Um, and- and, uh, an example I go back to often is this idea of in the 1700s, like, before even painting was, like, a massive, like, thing that you can do, uh, in any condition, situation or, like, location, painting was the realm of, like, these very sophisticated painters that were, like, painting in studios, right? Painting was the realm of, like, people who can afford and were able to understand and master the techniques of the masters, right? And- and more importantly from a tools perspective, it was really hard to get pigments, right? (laughs) It's a very practical thing, but, like...

    16. EGElad Gil41:39

        Yeah.

    17. CVCristóbal Valenzuela41:40

        ... you couldn't just... Pigments they didn't exist, right? You couldn't just go to a store and then get all, gotta get red, white, yellow and, like, paint something and I have a canvas. The way you mix pigments was this very sophisticated thing where you had to hire a master that knew, like, these obscure techniques and you were, like, measuring them, and then you- you store them in, like, these sophisticated bladders and you seal them, and it was a f- incredible, like, complex and expensive process. And then someone was like, "Hey, we should just, like, build a tube and then, like, have this, and carry it around," right? And this, maybe it's easier. (laughs) And it was. And- and it was, it was a very radical innovation, very simple at the time. I- I guess it was very radical at the time. It's very simple for us now. But that would allow us, like, for a l- whole new generation of artists to look at art and be like, "Great, I wanna take this painting and there's a mountain that I really like there. It doesn't have to sit on a canvas. I'm gonna paint in plein air." Which is a, which is, which is the thing. You paint in plain air, right? You're painting in air, in the, in the wild, and you're able to look at the world and the sky and you're able to, like, quickly brush, like, the light, right?... and just being outside of the studio was just not feasible. It was ten, like before that, and then gave birth to impressionism, right? And impression was like a whole revolution. Like, impressionism was not really well-received b- because it's like, "Hey, this is, this is not art. This is like, these are just brushes of, like, things. Like, they're not, I mean, no." Right? And then impressionism really started to pick up. People, like, started to r- really understand the medium, and then it evolve, it continues to evolve and evolve, right? And you find similar moments in time where the paint tube metaphor becomes relevant. And, like, photography, for me, was a very similar one, right? Um, and then cinema, for sure. Um, and then the digital world, the transition to, like, film. And every single step of the way, you have artists experimenting with this technology and using them to put a perspective of the world. I think right now, what we're seeing right now with AI, and, and, and there's also been... I, I like to think of two AI art waves. There was like the 2015 to 2022 where, like, the, the VQGAN and, like, the early GAN experimenters, and there was a lot of artists experimenting with it. And then now, the division kind of like, and the transformers kind of like world has enabled a whole new wave of people to experiment with it. The first wave, and, and now this particular wave, I think we're in the paint tubes kind of like moment, right? Where people are taking it and are using it to express something, right? To, to think of the world, and then type that in the world, and generate something, right? I think the artist still remains pretty much at the center, because that's what really art is about. And these are just tools, right? It's hard to understand them at first because they're just new, like every new piece of technology is. (laughs) And I think we're, I guess, to your point, like what are we gonna be asking ourselves in, like, 10, 20 years, 30 years? Uh, I think it's the realization that we'll look back and we'll look at this moment as in like, "Yeah, I mean, it was a natural transition and we needed it. It allowed us to do so many things that we just couldn't thought of before. Great that we had it." And I think we're still early at realizing that.

    18. EGElad Gil44:37

        Yeah, it seems like a extremely exciting time from a, a arts perspective. And I remember in, I can't remember if it was 2018, 2019, the first sort of GAN-based, um, artwork sold, uh, at auction. I don't know if it was like, if it Christie's or Sotheby's or something. And then, um, it almost felt like everybody got really excited, and then there was silence, you know? (laughs) Until this next wave of, like, diffusion-based models and, you know, everything else. Is there anything that you think is needed to encourage that art scene, or do you think it's literally just time now, because we have the tools and we have really interesting things happening? Or do you need to be able to print the art a certain way? I'm just sort of curious, like what are the obstacles for this becoming a bonafide sort of fine arts moment or movement?

    19. CVCristóbal Valenzuela45:16

        I think it's, it's convenience, and it needs to be, like, accessible and usable and understandable by people. I think in the analogy of the paint tubes, it's, we're not yet at the stage where you can just buy a paint tube and use it. We're still, like, in the stage of we're transitioning from, like, this sophisticated pigments to, like, some sort of like a paint tube, right? But, like, early, like, GANs and, like, Rovi Bharath, which I th- I was thinking, I think is the, the artist you mentioned that, that, that w- was behind all of the early works on, on, on that auction in 20, I think it was 2017, 2018. It was very hard to just get started with a model, right? It was very sophisticated process. And now you can just do it from your phone, right? The just very, you're, you're coming closer to the, like, the, we're putting the cap on the paint tube, right? Almost there. I think we're just a f- rate of, like the rate of progress and the expectation of it will become easier and better and... I would say two things. These models and these systems need to become really expressible and controllable, which is somehow the way I like to think about enlightenment is like, you have an intention and you wanna express that intention in a very controllable way, right? These models that are yet not controllable, right? Not exactly as we would like them to be, right? And the reason why, m- because of that is that we're still very early. Like, (laughs) there's a lot...

    20. EGElad Gil46:27

        Yeah.

    21. CVCristóbal Valenzuela46:27

        ... that they had to be, uh, invented to control them and, uh, have them be very expressive, and have them work in the way that you really want them to work.

    22. SGSarah Guo46:35

        So, the art movements that you mentioned, um, they're, they're art movements, but they're also, like, cultural movements fundamentally, right? And, like, we talked about the tools because you're a toolmaker, and it's like, "We gotta have the paint tube." But if you take impressionism or futurism or something, like, it's also, like, it had an aesthetic, it had tools, but it's also very Italian at a certain point in time. It was about optimism, uh, like urbanism and cars and everything. Are there, like, schools or philosophies or scenes that you think are, like, worth paying attention to right now?

    23. CVCristóbal Valenzuela47:03

        Yeah. Um, I mean, I'm biased because I'm, I'm, I'm, I've, I guess, I've been part of this particular scene in, in New York, the media art scene that I've, like, I particularly think will heavily, has, uh, heavily influenced a lot, a lot of this, this learnings in this stage. Uh, to your point, I think every art movement sits in a particular cultural context and historical context. And futurism was like in particular moment in time about, like, technology, and, like, also fascism was around. And there's a lot of, like, things, and you just look at the world in, like, this particular way, and you express in this particular way. And there's an aesthetic, and a line, and a system that if you look back it's like, "Oh, of course." Like, the... And cinema was the same, like, movies early on were, were a way of, like, perceiving the world and expressing them because it was in a very contextual, like, historical moment in time. I think for me, uh, if, if you apply that same kind of, like, principle now, I would tend to look a lot at, like, the weirdos of tech, right? (laughs) People who are at the fringes. People have, who've been always considered like, "Oh, you're just toying around. This is like a f- experiment." Like, like, there's a lot of coding, creative coding communities and people f- experimenting with codes as art. There's, like, a lot of conferences and these communities of people, like, Baby Castles in New York, or WorkHack, or, like, I.O. or... You have all this, they're just, like, interesting just art. It's just very, very, very highly creative and niche. I think those folks will define a lot of what we'll see next in, in tech.

    24. SGSarah Guo48:26

        Yeah. Well, uh, New York or otherwise, weirdos are a pretty good bet in general for people who come from the technology world.

    25. CVCristóbal Valenzuela48:32

        Yeah.

    26. SGSarah Guo48:32

        Chris, this has been amazing. That's all we have time for today. We're looking forward towards, uh, unlocked creativity in the next paint tubes. Thank you so much for joining us on the podcast.

    27. EGElad Gil48:41

        Cool. Thanks a ton.

    28. CVCristóbal Valenzuela48:42

        Of course. Thank you for having me here. It was great.

    29. NANarrator48:46

        (instrumental music)

Episode duration: 48:51