This Insane AI Video Search Technology Selected by NVIDIA and Snowflake | Twelve Labs, Jae Lee

1. 0:00 – 1:07

   Intro

   1. JLJae Lee0:00

      If you try to accommodate all of the advices that you get from your mentors, your company will most likely become like an underwear company, totally different from what you wanted to build. Having some fundamental, like, foundation for yourself and for the company, and be able to say, "Thank you for your advice, but no thank you." Right? So having that, the gut to say no to someone that you, you respect. As a founder, I think I grew a lot. Hi, EO. My name is Jae. I'm one of the co-founders and CEO of Twelve Labs. Twelve Labs is an AI research and product company based here in San Francisco and Seoul. We're building video foundation models for developers and enterprises building video-centric product. We basically build humongous AI models that can understand videos like humans, and we serve it to developers via APIs, uh, that are looking into building really powerful semantic search or classification or summarization into their products. We currently have a little over 20,000 developers that are actively using our search API. We have largest creators of the world adopting Twelve Labs, as well as, uh, media, entertainment, and large sports organizations and law enforcement.

2. 1:07 – 7:06

   Chapter 1. A Startup Founded during Military Service

   1. JLJae Lee1:07

      [piano music] So I was born in Seoul. I spent about 10 years in Seoul. Uh, I had a chance to move to the States. I moved when I was 11, so went through elementary and middle school in Knoxville, Tennessee. I was able to pick up a lot of the, the, the culture and, and, and things like that. I was, uh, very interested in expanding my perspective and, uh, exploring the new world. My first experience with software engineering, or at least coding, was, uh, MATLAB. My uncle, uh, was actually getting his PhD at the University of Tennessee, so I, I see him, uh, kind of like plotting these dis- distribution graphs and things like that, and which made me curious what, what he was doing. That's how I got into playing around with, like, small data sets and doing the same thing that he, he was doing because I wanted to be relevant and I wanted to talk to him about a bunch of things. He probably thought what I was doing was pretty cute, which sparked my interest in learning more about how do we capture all this data and be able to create a system that can really understand the distributions of the things of the world. And it just kind of felt like if you had the understanding of that, it gives you power, predict anything. And I went to Berkeley for college and studied computer science. So I really geeked out on AI and, and software engineering. Um, so spent about 15 years, uh, in the United States, um, so half of my life in Seoul and in the States. I was drafted into, uh, this organization called Korean Cyber Command, where like-minded people were already there, um, with incredible knowledge in software engineering and, and, in AI. I joke about it, you know, I served the country with keyboards rather than, than, than a rifle. I was really fortunate to have met my chief architect, SJ, and then Aiden kind of also joined in, he was... You know, I still clearly remember Aiden with his buzz cut coming in from, from boot camp. But from day one, uh, we knew we had this common interest in AI and what can we do as a, as a young scientist really, like, push the frontier of AI development. And we've spent a lot of times reading papers, discussing, arguing, and it turned out there were two clear paths. Uh, one was after military, we go pursue a career in academia, go become a professor, or start something of our own. And looking back, what we've realized is that we were spending so much time together. And military is this really special setting where, you know, you're basically jamming in 50 to 100 20-year-olds, right, with testosterones. And what we thought was, you know, "If we're having this much fun in military, imagine what we can do when we go out." So it was pretty clear to us that we're gonna start. I think we've spent about a year and a half really thinking about what is the next frontier for AI and how can we contribute to pushing that boundary. There's this seminal paper called Attention Is All You Need, which some people call it the transformer paper, which is making a lot of impact now. But when it was published in 2017, at least for text and image-based, you know, foundation models, probably capital was going to be of mode, whoever raises most amount of money. What we realized was, uh, there's still a lot of unexplored research areas for multimodal video understanding. There, rather than capital, is probably going to be really passionate, smart, but also slightly dumb people that's, like, dumb enough to start it will have really good chance of succeeding. And also we've realized that with the explosion of, uh, the video and, and other complex multimedia data is going to be the infrastructural data for the internet. And we realized that probably developers and enterprises need something better than object detection or, or transcription to make sense of all, all this video data that the humanity is creating. So it was like a no-brainer to start building for video understanding. The, the model that Twelve Labs is building, it's basically trying to map human language into whatever that's happening in video content. So if you can map precise human language to whatever that's happening within video content, that gives you these emergence capabilities, like being able to search for things really well or being able to classify things or summarize. You know, we didn't all join the Korean Cyber Command at the same time. So SJ was already, like, six months ahead of me, and then Aiden was six months behind. So, like, okay, we, we decided we're gonna [laughs] we're gonna start this company, but then SJ's, like, leaving, like, next year, and then I'm leaving the year after, and then Aiden's, like, six months after. So how do we do it? [laughs] So it was genuinely very scary. So we had our ideas. I remember SJ was discharged on Thursday, and he came back To the military base Saturday of that week with our laptops, and he took us out. In front of our military base, there was a bagel shop ca- called Last Bagel, and that was, like, our office where SJ would bring all of our laptops, and we would do our research and, and do some little, little bit of prototyping. And we did that for six months. And then I got discharged, and then I did the whole laptop carrying and, and taking out Aiden. So we did that for, like, a good, like, year so that everyone is out. And then we had bunch of friends that were working in AI and crypto and blockchain and, and, and Web3 was just booming. And we had a, a mutual friend, like the founders had a mutual friend that had, like, a really nice office in Seoul. And he was like, "Oh, you guys can come in and, and use our office space." Uh, so that's what we did. And then after like three weeks, that company went bankrupt. And then really scary, like, people started coming in and, and we had, like, our desktops and our GPUs all set up there, and we got really scared. So we brought everything back out, and we found, like, really tiny, tiny office, probably like a size of a dressing room. [laughs] Uh, that's where, like, all five of us kinda spent, uh, the next six months, uh, before, uh, we raised our proper seed round. Looking back, if we were to do it again, I don't know if we'll be able to do it, but some people say ignorance is bliss, and I think we were just, like, really naive and just really excited about building this company. And I guess not knowing what was ahead, um, allowed us to kinda do what we did. [laughs]

3. 7:06 – 8:06

   Sponsor

   1. SPSpeaker7:07

      Hi, all. I'm Jae from EO. I have a quick question for you. Which is the better prompt? If you're looking to use AI to apply directly to your workflow, I would say this one does the job. This prompt was made using the 5 Essential Resources for Using ChatGPT at Work, a free bundle of AI resources provided by HubSpot, our amazing partner. The bundle has the following resources: Tips on implementing ChatGPT at work in various professional settings, The Ultimate Guide for Adopting AI at Work checklist to make sure your AI-generated content feels authentic, and more. For this prompt, I use Templates for Setting Clear Guidelines, which gives simple guides to dramatically improve your prompting and results using ChatGPT. The best part is that it has over 100 prompts that you can use right away, so you don't have to start from scratch. So for anyone who's tried really hard to use the power of AI at work, give this toolkit a try. Download for free using the link in description and start supercharging your workflow with AI. Thanks to HubSpot for sponsoring our video. Now back to the video.

4. 8:06 – 13:04

   Chapter 2. What a startup with only $2000 can achieve

   1. JLJae Lee8:10

      When we first started the company and we hired our first employees, they had hard time explaining what Twelve Labs does to their parents. Really new thing. What is foundation model? What is video understanding? It's a new concept that's hard to understand for, I guess, uh, people that are not in the space. Nowadays, people talk about the foundation layer, the tooling layer, and the application layer. It all – everyone's very familiar. Uh, when we started Twelve Labs, I think technologically it made total sense. We knew it was going to happen, but what was uncertain about it was will market accept? But knowing that we are at the verge of breakthrough in building an AI that can at least, you know, get to certain level of human understanding of videos. So we were betting on market's acceptance of foundation models. The founders are pretty, pretty much broke, right? Because we spent two years at, in military, and I think we had $2,000 [laughs] uh, to start with. So barely, barely enough to do anything but figuring out what was going to be impactful that we can do given our current resources that'll put us in the map or at least, you know, let the world know that what we're doing is relevant. So our tactic here was, okay, we're gonna talk to a bunch of customers, and there were early believers in Twelve Labs who took our APIs and built awesome things with us, but we needed more exposure basically. So as a team, we've decided, uh, to participate in ICCV's, its International Conference in Computer Vision. They're putting this, like, awesome competition for video understanding. So I- we talked to Aiden. We have nothing to lose and only to gain. The team was extremely supportive of, of Aiden spearheading that effort with the team. All I can do to support is, you know, there was some ideas and, and directional kind of feedback that I gave to Aiden, but we needed compute, and we needed determination to put some serious cash behind. And back then for Twelve Labs, like $200,000 in, in compute was, was a lot of money for us, right? So – and just thinking that, okay, we're gonna blow through $200,000 in 10 days [laughs] in compute was really scary. But the team was able to use that capital, that, that precious capital, and build something incredible, helped us win the competition. Um, so I think the important thing is if you're building something really impactful and you think that it's going to significantly change the industry that you're in, there will always be someone that has very similar thesis. It's just a matter of how do you get yourself out there? How do you let the people know that you, you, you exist? For us, that was the competition. After winning the competition, companies like Index Ventures and Radical Ventures had very strong thesis around multimodal AI. The next idea would be what's next? And video happens to be the most relatable multimodal data. These amazing companies actually came in inbound, right? So they reached out to us, and we started jamming The conversations turned into next conversation, and then we talked about technology and, and it just happened very serendipitously, right? The first pitch deck, I was in Seoul, and, and my first, uh, call with Index Ventures was at, like, I think it was, like, 3:30 AM Seoul time, and we didn't have pitch deck. And we just felt like, hmm, this is our first meeting, and we, we, we knew nothing about fundraising [laughs] then. We didn't even know this was going to be, like, a friendly introduction, but I just kind of felt the need to, like, oh, we don't have pitch deck. [laughs] We need to build one. So I remember staying up till, like, 3:00 AM building it. I think the storyline was, like, quite simple. We didn't have much to sh- much to show for it. The idea was, hey, the problem that we're solving is massive. 80% of the world's data is in video, and there's no adequate solution out there or technology out there for developers and enterprises to make sense of it all. That is the market that we're ta- we're tackling. We wanna index all of that 8- like, 80% of the world's data, and this is the technology underlying research work that we've done. And then that was it. VCs asked a lot of hard questions. The most memorable one, "If I bring you TikTok as a customer, and they wanna index, like, billion hours of content, how long does it take?" And I think we were, [laughs] we were thinking about maybe, like, a million hours. It's, it's going to take, like, 10 years, right? That's when we realized we should never be comfortable of- with what we've built. This whole new incredibly large world out there, you know, that s- maybe some people are impressed with our system being able to index, like, a million hour, but there's others that are thinking about billion, 10 billion, 100 billion hours. And that was, like, a really challenging question because, like, what we said during our pitch is we wanna index all of the world's videos, and that question kinda made me stunned and then think about all of the technical issues that we, that we had at the time. He probably find it, like, funny, right? Where I was trying to give my best answer as a founder. It's like... So the company has raised about $30 million,

5. 13:04 – 19:46

   Chapter 3. Lessons from Acquiring Early Customers

   1. JLJae Lee13:04

      uh, in seed funding.

   2. SPSpeaker13:04

      [instrumental music] My name is Soyoung. I'm one of the co-founders of Twelve Labs, and I currently lead our business development and go-to-market. We had a customer who was paying for our product, um, but they weren't actually using it. Uh, but we had gone through a lot of work to actually get them as a customer through a lot of sales and kind of relationship building and then so on, but I think they were, you know, they were extremely early, and we had almost pushed the sales to happen. And I think what we learned from that experience was we have to optimize even early on. It might have been better for us to not actually make the sale because the customer prob- uh, probably wasn't ready. They didn't have the passion or the innovative drive that o- our other customers had had. But we were optimizing for kind of hearing the yes. We tried so hard to make that no into a yes, and we had succeeded, but at the end, at the end, I think it turned out that we probably should have kept it at a no and focused on all the other customers where the yes was very more clear. And they had a very clear vision of how they can, you know, build their ex- new experiences with the s- the technology. And especially for earlier products and earlier technologies where resources are limited, and you want to, you know, build for your mo- best customers and the innovators in every field, uh, you should probably start to optimize for hearing the no, uh, than the yes because that will help you find the right direction faster. We quickly think we learned through trial and error that the types of companies that we need to-- early customers that we need to find and work with are true innovators in their field, uh, whether they come from, like, the content creation space, law enforcement space, e-learning, and so on. You know, we've had instances of trying to oversell. We were going through sales, you know, sales 101 books or, like, sales methodologies, and we were trying to pitch to the customer, "Hey, here's the use case you could build out with our, you know, with our technology. You know, we'll improve your ROI by X percent with our technology." And we did make some sales from those, but I don't think it was something that we probably should have, you know, spent so much time on doing because if you find the right customer who's innovative, you don't need to explain anything to them. You show them, uh, a use case based demo of, you know, for us it would be, uh, we, we would index, uh, some videos that resemble the customers, and then we show them how you can search or you can generate text very easily, like a, like, just like a person would, um, had they been watching the video. They can draw out the full map of, okay, this is what I want to provide to my customers or my users. Uh, this is the technology that you have, and I can, and they can, you know, fill that gap in pretty easily. And they know already, uh, what the return on that or what the opportunity of that e- experience would be for them. For the kind of really large, high-profile customers that we have right now, it was all the same process. We actually learned all of these from the customers because they had, you know, seeing our demo and seeing the early kind of hints of the technology, uh, they were able to teach us about how they could utilize the technology. And even today, it's not just a single use case that they want to power. Uh, they come to us with four or five different ideas of how, you know, the technology can impact different units, different business units, um, and different, you know, optimize different workflows or build new experiences.

   3. JLJae Lee16:18

      I think we, [laughs] we make stupid mistakes probably every day. The one that I regret the most is we knew we had, like, this conviction around building a foundation model, but then we didn't have any data point as to how do you build that company. And I think I blindly kind of believed that the startup mantra, like identifying a narrow problem and building a very narrow solution for it. I think we've spent a lot of early days thinking about, okay, if we have this really powerful AI that can understand videos, what do we do with it? And we knew, we know from the, from the get-go that we wanted to serve it to developers and enterprises, but I think, you know, we've had mentors and, and, and, and other founders like, "Oh, you should build TikTok 2.0," or, "You should build, like, YouTube 2.0," or, "You should build that." And I think we've spent a lot of time thinking, okay, yeah, maybe, like, TikTok 2.0 makes sense or maybe, like, Gong 2.0, like, sales call analysis. Uh, but that didn't really, like, excite us because we knew, like, we're good at building the infrastructure and, and helping developers build the next thing, and that's probably the stupidest thing that we've done. It's like spending time on thinking about things that we're not excited about. As a founder and CEO, it's really hard to get distracted. If you're a first-time founder or a young founder, your mentor's advice means a lot to you. But Having, like, your own grounding, having your own... Also real- like, relying a little bit on your own gut feeling is very important because what we've realized, if you try to accommodate all of the advices that you get from your mentors, your company will most likely become, like, an underwear company, totally different from what you wanted to build. So my key takeaway is having some fundamental, like, foundation for yourself and for the company, and be able to say, "Thank you for your advice, but no, thank you." Right? So having that, the gut to say no to someone that you, you respect. As a founder, I think I grew a lot, right? Twelve Labs has multi-year compute partnership with Oracle Cloud Infrastructure where we get all of the, uh, state-of-the-art NVIDIA, uh, chips. And Oracle had put together, uh, this small event for their key partners that are building foundation models for. And Aiden and I had chance to meet with Jensen, because Jensen was at that event. And we had, I think, 5 to 10 minutes ta- to talk about Twelve Labs, and I think it seems like he had, he has a special, uh, place in his heart about, uh, computer vision and video understanding as, you know, that was one of the first use cases that NVIDIA chips powered. So we, we got to, uh, meet with, like, NVIDIA folks from that event, and then Twelve Labs was featured in NVIDIA's 2023 GTC. And I think that kind of sparked other people from NVIDIA to be interested in Twelve Labs, and the NVIDIA's venture team reached out to us. It was quite casual. We were talking about Twelve Labs and the future that we're drawing and the future of multimodal video understanding. And I think the venture team also had an idea of how NVIDIA and Twelve Labs can partner up more than just financial investment, but also think about really robust product partnership. From then on, what Twelve Labs is doing and what NVIDIA wants in vision and video understanding was just, like, a perfect match. It happened quite, uh, naturally from, from conversing our, our technology, our roadmap, and, and NVIDIA's future in terms of, like, producing really powerful, uh, chips for edge devices, uh, for smart cities, right? So there's that natural fit of two companies' product really creating

6. 19:46 – 21:25

   Chapter 4. The Best Engineer is not the Best Coder

   1. JLJae Lee19:46

      synergy. Nowadays, I am focusing mostly on hiring. So I think Twelve Labs is group of great people. I spend a lot of time meeting great people. I wanna be able to recognize greatness when he or she comes in. Good engineers, or even, like, just good people in general, have this core values. I would go, uh, near their places and get together at a cafe, and we would speak for three, four hours. My way of deciding whether this person is a good fit for Twelve Labs is, like, I am able to learn from their core values. Everyone's really good at coding nowadays, but great engineers can apply their core values and their skillsets, and is able to talk about the company that they're excited about and how they wanna impact it. How do you see the product evolving? How do you see our interfaces ev- evolving? The best- some of the best engineers are not the best coder, but having that perspective, really strong perspective, and, and groundedness is very important, and, and I try to look for that. Twelve Labs' vision in the next two years is really, uh, becoming horizontal video understanding infrastructures for all of the businesses and developers that are working with video data. We wanna enter into, uh, streaming as well, so real-time, uh, video data, and, uh, really become a, a visual cortex for modern video applications. [upbeat music]

Episode duration: 21:25