How Intelligence Measures the Frontier

1. 0:00 – 3:21

   How Design Arena Works

   1. AGAnkit Gupta0:00

      [upbeat music] I'm very excited to be here with Grace, the CEO of Intelligence, the creator of Design Arena. Grace, thanks so much for coming.

   2. GLGrace Li0:11

      Thank you so much for having me.

   3. AGAnkit Gupta0:12

      Tell us a little bit about what Intelligence does.

   4. GLGrace Li0:14

      So we benchmark the frontier models on the hard-to-verify domains, and we primarily do this in a product-led way. So our first product, Design Arena, is one of the default places that folks go to go build creative things with the model, so their own websites, images, videos, et cetera. And on the back end, we turn that into the most accurate benchmark in the world for where these models are in creative domains, and we work with the frontier labs to help improve their models on these hard-to-verify domains.

   5. AGAnkit Gupta0:40

      So how does this actually work? Someone goes to your website, what do they see? What do they... W- what are they going in with the intent to do? And then-

   6. GLGrace Li0:46

      Yeah

   7. AGAnkit Gupta0:47

      ... how do you actually make this into a benchmark?

   8. GLGrace Li0:49

      Yeah, great question. So somebody will come into the website with the intent to make a website, for example, or they'll make an image, or they'll make a video. And what they're putting in is the human-computer interaction, right? So something that they're requesting from the model, the back and forth, the, uh, edits of what they like, what they don't like. And on the back end, you can think of us almost like a PM, transforming all of these, like, raw user signals into, um, whether or not the user is enjoying the output from this particular model, and that is a real world eval. The same way that PMs, for example, will take a look at what users are doing on a product and then tell their engineering team, "Okay, you gotta fix X, Y, Z about the product," we take a look at what users are doing when they're interacting with the models, and then tell the model teams, "You gotta fix X, Y, Z about the models."

   9. AGAnkit Gupta1:31

      So you are now, um, a year into building this company.

   10. GLGrace Li1:34

       Yeah.

   11. AGAnkit Gupta1:34

       It's, it's a company that is very unique in what you're building, and I'm actually... I'm curious to dive into a little bit about how you actually got here. So maybe before you started this company, tell us a little bit about what you were doing before. How did you arrive at starting the company in the first place, and of this idea that is sort of an inconceivable idea even a few years ago? So, uh, how'd you get here?

   12. GLGrace Li1:52

       Yeah. I, I didn't start off here, right? Kind of started in the whole big tech, some- some time in quant, and the feeling there is like you're just working at a system that is, like, not very leveraged. What do I mean by that? You could put in, like, 120% of work, but you would get the same, like, 30% out from the company because Apple is not gonna move 20X faster because you're running 20X faster, right? It's a phenomenal team, but it's just a, a very different setup. And so, um, you know, we were really looking for a place where, okay, if you put in, like, 200X of effort, can you actually get, like, 400X of returns? And what's very true about early-stage startups is if you're in the right space, it feels that way, right? Like, don't get us wrong, this thing is, like, incredibly hard, but you also happen to do something that's wanted by the market and wanted by a lot of people. If you put in two X of effort, you actually get four X of returns. So, um, that's how we ended up in this spot and, and, you know, ho- hopefully continue to stay in, in this trajectory.

   13. AGAnkit Gupta2:43

       And where did it start with this particular company then?

   14. GLGrace Li2:46

       Yes.

   15. AGAnkit Gupta2:46

       Like, what was the first idea where you were like, "Okay, maybe we wanna make this into a company"? And then where did the particular idea you're working on now come out of?

   16. GLGrace Li2:52

       Yeah. So a incredibly successful initial idea of being a gaming company, actually. [laughs]

   17. AGAnkit Gupta2:56

       Okay.

   18. GLGrace Li2:57

       So we noticed that the models could make, like, relatively okay video games that were functional. Um, and we were-

   19. AGAnkit Gupta3:03

       This was, like, a while ago.

   20. GLGrace Li3:04

       This was a while ago. This was, um, about a year and a half ago, where even the most frontier frontier models could only output one page of code at a time, so, like, a single long HTML file, right?

   21. AGAnkit Gupta3:15

       We did not have coding agents as good as the way we do now.

   22. GLGrace Li3:17

       There was, there was just things the hardest.

   23. AGAnkit Gupta3:18

       There was like-

   24. GLGrace Li3:19

       Yeah.

   25. AGAnkit Gupta3:19

       There was, there wasn't Cloud Code yet.

   26. GLGrace Li3:20

       No, no.

2. 3:21 – 6:43

   From AI Games to Creative Benchmarks

   1. AGAnkit Gupta3:21

      Yeah.

   2. GLGrace Li3:21

      Like, you would basically just copy-paste a bunch of these, like, in chat output files. And I remember we realized, like, okay, they can make functional games, but the games aren't fun, right? And you, you could break that down, like, they're not well designed, they don't have good storylines. They, they're like... Y- you didn't have, like, a kind of addictiveness to the game. Like, what is going on? And the canonical wisdom at the time is that these models would keep getting better at, like, math, chemistry, and physics, these domains with correct answers that we later learned were called verifiable domains, and have a much harder time getting better at the non-verifiable domains, like design or game development.

   3. AGAnkit Gupta3:56

      Right. You c- it's hard to make an RL environment for, like, this looks good.

   4. GLGrace Li3:59

      Yes, exactly. And that just seemed like a bunch of bogus to us because people can get better at these non-verifiable domains, right? Um, if you are a game developer for three years, you're much more skilled than you were before you started that three-year journey. So there must be some signal that can be learned. It's just probably learned in a different way than getting better at math, right? So we thought, well, what if they're... the reason why there, there isn't, like, as much progress or as much attention even on the field is just because there's no North Star, like, forcing function of what better even means in these subjective domains. And so then we, we took a look at what we had built, which was this game engine, and we realized, well, there is actually some ground truth signal for whether a game is good. How many people are playing it, right? Like, what's the retention rate on the game? Steam uses something very similar to figure out, you know, whether games are good or bad. So you actually can rate these games. Okay, cool. So what if we have a really rare opportunity to make a double-sided product, right?

   5. AGAnkit Gupta4:53

      Mm.

   6. GLGrace Li4:53

      That, yes, is for humans to access superintelligence and make things that they love, but also, and, and maybe, like, just as importantly, for a superintelligence to access humans to learn what it is that people love at scale. So that's how the kernel of the idea of Design Arena was born, and we've since scaled that to over seven million users now across, um, 192 countries.

   7. AGAnkit Gupta5:14

      And so in that first version, you're basically saying it, it's basically a comparison website, right?

   8. GLGrace Li5:18

      Yes.

   9. AGAnkit Gupta5:18

      You're coming up with two different versions of a game or a website or an image-

   10. GLGrace Li5:22

       Yeah

   11. AGAnkit Gupta5:22

       ... or whatever.

   12. GLGrace Li5:23

       Yeah.

   13. AGAnkit Gupta5:23

       And people are selecting, you know, which one they prefer or are, are using more, for example, and that gives some useful signal that you can't just capture in, like, a simple verifiable reward.

   14. GLGrace Li5:33

       Yes, exactly. But critically, the framework shift to have as the user is, um, th- they're not actually just rating things for the sake of rating things. The same way that you go and commission a designer to design your room maybe, and they'll come up with three variants of mock-ups for you to pick which one you like the most, it turns out the best form factor to deliver something to a client when it's a visual deliverable is to show that client multiple visuals of the thing, because by seeing the output, they will have, like, better understanding of what they want.

   15. AGAnkit Gupta5:59

       Oh, interesting. So, so you're saying it actually improves the experience-

   16. GLGrace Li6:02

       Exactly

   17. AGAnkit Gupta6:02

       ... of website building to give multiple options-

   18. GLGrace Li6:04

       Yeah

   19. AGAnkit Gupta6:04

       ... as in the way that a designer will give you multiple options.

   20. GLGrace Li6:06

       Yeah. Like, critically, you cannot, um- Deteriorate the UX experience of the product by asking somebody for, like, labels, for example. You have to do your best job possible to embed those labels into the UX experience of the product. Like, one way to think about it is PMs who are making products better for users actually do this all the time, right? They, they have buttons where they know to be, like, critical buttons to measure. Um, but you, you don't ask the user 24, "Hey, are you having a good experience right now?"

   21. AGAnkit Gupta6:33

       Right.

   22. GLGrace Li6:33

       Yes. Yeah, yeah.

   23. AGAnkit Gupta6:35

       You mentioned earlier that it's not just humans accessing super intelligence, but it's how super intelligence understands what humans prefer.

   24. GLGrace Li6:42

       Yes.

3. 6:43 – 9:56

   Measuring Human Taste at Scale

   1. AGAnkit Gupta6:43

      Could you, could you say a bit more, maybe just more broadly, about, you know, why do evals matter exactly? Like, how are the frontier labs using these to decide what to do?

   2. GLGrace Li6:50

      Yeah. Uh, very good question. So, so the way to think about it is, is this way. Unless you know where, like, the model breaks, like, unless you know where the models fail, you don't actually know how to improve the models. And particularly in subjective domains, those failures actually can't be diagnosed, we found. They can't be diagnosed on an individual scale. Like, just because you don't prefer X to Y doesn't mean you know whether the population level agrees with you, or, like, the certain cohort that you're trying to optimize for agrees with you. So, you actually need signal at scale to figure out whether there is a failure. So, specifically in the realm of non-verifiable domains, evals matter because they help you diagnose failures, and the specific shape of these evals is it's harder to have an individual expert, for example, be the verifier of whether something is good or bad. Actually, you need, like, population-level data. One, maybe one way to think about it is, um, TikTok has essentially built a verifier, a subjective verifier for a person, right? Like, you're-

   3. AGAnkit Gupta7:41

      Based on how long you spend, right, on each video.

   4. GLGrace Li7:43

      Exactly, exactly. Like, they are optimizing for the goal of, like, the videos you're least likely to leave, right? The, the, the videos that, like, keep you there. And you are the verifier for that product, right? Like, how long you stay on there is, is the verifier. Um, TikTok probably in aggregate has the best information on whether a video is, like, going to do well or do poorly. But it's not because there's, like, one dude at TikTok who, like, you know, knows this. What... No, they, they've built a, a system with an algorithm and a bunch of population-level data to be able to predict well whether a video is, is going to do well or do poorly. Not that they have, like, a house of, like, super high taste, like X, Y, Z individuals because it turns out what's high taste in the high taste cohort is probably low taste in another cohort that is, like, just as valuable to another, to another provider. Yeah.

   5. AGAnkit Gupta8:27

      Probably why ByteDance's video models are so compelling.

   6. GLGrace Li8:29

      Yeah, potentially.

   7. AGAnkit Gupta8:30

      Yeah.

   8. GLGrace Li8:31

      Yeah.

   9. AGAnkit Gupta8:31

      Uh, related to that, I mean, you, you guys probably have some of the best insight in the world of what the current state of frontier AI looks like and what this full landscape of companies, which has dozens, if not more providers-

   10. GLGrace Li8:46

       Yeah

   11. AGAnkit Gupta8:46

       ... is acting on and optimizing for.

   12. GLGrace Li8:49

       Yeah.

   13. AGAnkit Gupta8:50

       So, I'm curious what observations you've had. Like, are the-

   14. GLGrace Li8:52

       Yeah

   15. AGAnkit Gupta8:52

       ... things that you guys can see from your vantage points that are all public information-

   16. GLGrace Li8:55

       Yeah

   17. AGAnkit Gupta8:55

       ... but that may not be obvious to people-

   18. GLGrace Li8:56

       Yes

   19. AGAnkit Gupta8:57

       ... who, who don't have your vantage point?

   20. GLGrace Li8:58

       Yeah, very good question. I, I'd say the biggest vantage point we have is, like, data at scale. So, there are two things. One is we c- it, it's like the future is already here, just unevenly distributed. And from the place where we sit, you get to see what the landscape looks like the next, like, three to six months out, not necessarily, like, just reacting to what's happening today. And what we found is, like, okay, you can kind of classify people's preferences into, like, stated preferences versus revealed preferences, right? Like, what, what, what is it that they're saying that they want, and then what do they ultimately want? And e- even if we're just analyzing the portion of the data of, like, what is it that they're saying that they want, this, like, drifts over time. So, like, our Q1 of people's stated preferences are, like, very different than Q3 or Q4 people's stated preferences. And what we're seeing is, like, for some of the frontier labs, they're thinking about how to use this, like, stated preference drift, for example, to improve their model's outputs based on what you predict in Q4 and in the next quarter people will start to want given, like, the, the trends of, of-

   21. AGAnkit Gupta9:54

       Yeah

   22. GLGrace Li9:54

       ... uh, how that's changed over time.

4. 9:56 – 13:52

   A Marketplace for Specialized AI

   1. AGAnkit Gupta9:56

      And are all the labs, I guess, optimizing for the same thing or, or is it more like there's many intelligences getting made? How do you think about that?

   2. GLGrace Li10:02

      Yeah. What, what's another thing that's, like, very interesting is that different frontier labs will have, like, different target demographics for who they ultimately want to appeal to. Major frontier lab X might want to appeal to a different thing than open source, like, small neo lab Y, right? Like, they're, they're probably appealing to a different end user cohort. And so for subjective domains, being better is different in, in these two cases, so you're actually looking for, like, a different set, like, post-training data to improve to. So, let's say if you, you really want to serve the Asia population, for example, your definition of how a UX should look is, like, quite different than if you're looking primarily to serve folks in, in North America. Um, so it actually looks very similar to the ads landscape is what I'd say. Um, there's kind of this marketplace happening where, uh, intelligence will, will, um, objectively have different correct functions, uh, depending on the task that is being fulfilled and depending on who is requesting the task.

   3. AGAnkit Gupta10:58

      Is that how you then think about where you yourself fit into this? Like, you described it as, you know, like an ad marketplace or I guess an intelligence marketplace.

   4. GLGrace Li11:08

      Yes.

   5. AGAnkit Gupta11:08

      As these models get better and as you see more of them developing-

   6. GLGrace Li11:11

      Yeah

   7. AGAnkit Gupta11:11

      ... which I assume is the trend line we're on right now, where do you see yourself fitting into that?

   8. GLGrace Li11:15

      Yeah. So, one of the things that we noticed is, well, Decagon had this really good piece a while ago where they said 70% of their model usage is open source models, and that makes sense because it turns out to, like, answer the phone and go, "Hi, how are you? Can I get an account started for you?" You don't need, like, a very Stokes level intelligence. You probably need, like, a cheap model that's relatively fast and, you know, responds quickly.

   9. AGAnkit Gupta11:36

      Right.

   10. GLGrace Li11:36

       And so you see this idea of specific intelligence. So, if an open source model that's, like, relatively small and is very good at that and, like, very accurate at that, why would you want to use, like, a generalized model that's, that's more expensive, right? So, it looks like, um, the frontier of intelligence might actually look like a system that chains together many, like, specifically intelligent components. And the way that we see it is as long as it's true that- That these individual, specifically intelligent components are not all coming from, like, one lab, that there is, like, a diversity of providers, like open source or, um, application layer or, or Neo Lab or Frontier Lab, um, that there exists a place-

   11. AGAnkit Gupta12:14

       Which probably some mix of capitalism and antitrust would enforce anyways.

   12. GLGrace Li12:17

       E- exactly. Also, also similarly, like, how easy it is to post-train a model to get it to be specifically good at, like, one capability, and, like, whether you can put a moat on that, um, and, like, how cheap general intelligence gets. It, like, looks to be the case. And so we see ourselves as the marketplace for specific intelligence. Like, we will be able to, given your price and your time and your preferences, give you the best intelligence always, without any stake in the game of, like, only providing our own models, for example, since we offer humanity's models and not just our own.

   13. AGAnkit Gupta12:50

       Is that why you also think about product-led evals as the right way to kind of measure how well these models do, or at least an, a really important way to measure them in the context of this multi-intelligence world?

   14. GLGrace Li13:01

       Yes. Like, at the end of the day, we are making our living on measuring what people want. And in order to understand what people want, you actually just have to have a lot of signal of what people are wanting and, like, what they're doing. Not necessarily, like, sit in an ivory tower and say, like, "I'm just going to be the best at knowing what everybody should want," and, like, do that, right? Um, it would be very different if you're trying to make benchmarks for, like, math, chemistry, or physics. Those are, like, verifiable domains that in the end state probably don't even need a human in the loop. You can run some, like, very tight, like, real-world verification loop without a person in the loop. But at the end of the day, we want to be at a place where intelligence is serving humans. And as long as it's true that humans are wanting something, and that thing that they're wanting is changing, which has been true for as long as history's been true, um, there's always something for the models to get better at, which is better serving what people want.

5. 13:52 – 17:04

   The Small Team Behind a $60M Run Rate

   1. AGAnkit Gupta13:52

      So I wanna change gears a little bit and talk a little about the kind of company that you're building.

   2. GLGrace Li13:55

      Yeah.

   3. AGAnkit Gupta13:55

      So it's been incredible watching you build this company over the last year. You guys have gone from zero to, you announced a few months ago, from zero to 60 million in, in annualized run rate in less than a year or about a year, which is nuts. Like, a few years ago, this would've been an unheard of rate for any company to hit. And you're a relatively, uh, tight team that's doing this also. How do you get a team of people to invent so much new stuff?

   4. GLGrace Li14:24

      Yeah.

   5. AGAnkit Gupta14:24

      To invent a whole new business model and a whole new vision for what thing needs to exist? It's not like you're in an existing market.

   6. GLGrace Li14:30

      Yeah.

   7. AGAnkit Gupta14:30

      You're building a new market, and all point in the same direction.

   8. GLGrace Li14:34

      Very, very, very good question. Um, I think what we've found is a lot of what will slow you down when you get into, like, a, a, as a bigger team is just when people don't have the same kind of framework for solving problems or, like, the same, like, problem-solving framework. Specifically what I mean by that is it becomes easier and easier to give up on people. So think you're, like, a 2,000-person company, right, and you have a disagreement with your manager. It's like, well, what are you gonna do? Are you gonna change their mind? Are they gonna change your mind?

   9. AGAnkit Gupta15:01

      [laughs]

   10. GLGrace Li15:01

       Like, no, you're probably just gonna be like, "Ah, you know, we have a disagreement, whatever. I'm just gonna do the thing that they, they wanna do," right? But then what you start to do is, like, you develop a story that, like, you guys solve problems in a different way. God forbid you develop the story that they don't solve problems in a good way. And then you're not just two people on the same team solving a problem together. You're, like, both trying to solve the problem and trying to manage the problem with your manager. And then, like, the whole thing slows down, right? Um, what we find is, like, very advantageous of a small team is you can develop a culture of, like, never giving up on a person. So for example, let's say you, uh, want to solve or, you know, you, you think you should solve it way A, somebody else thinks you should solve it way B. You can afford to never give up and, like, never say, "Oh, it's not worth sharing with this person why I wanna solve it with A, way A, 'cause they're not gonna listen to me." And likewise, the other person, like, will, will feel like, "No, no, this person is, like, a rational person that I trust, and so I will explain to them why I think it should be solved way B," and then you guys will both end up, like-

   11. AGAnkit Gupta15:54

       Hmm

   12. GLGrace Li15:54

       ... in the, in the better decision-making framework. So we found that alignment to be, like, critical because that means that 80% of the time you don't have to give a damn about, like, why somebody picked a thing. You just inherently trust that, like, they, they are kind of on the same, on the same page as you.

   13. AGAnkit Gupta16:08

       Now, for that to be true, though-

   14. GLGrace Li16:09

       Yeah

   15. AGAnkit Gupta16:10

       ... that means you probably have to get people that have some kind of shared decision-making framework or, like, some sub- some, some sort of shared beliefs-

   16. GLGrace Li16:18

       Yes

   17. AGAnkit Gupta16:18

       ... that all point the same direction.

   18. GLGrace Li16:19

       Yeah.

   19. AGAnkit Gupta16:19

       How do you look for those things?

   20. GLGrace Li16:20

       Yeah. Very, very good question. Um, I mean, I think early days what's nice is you're doing, like, recruiting from the people who, you know, were in very similar circles to you. In the first, like, 80% of our team is, is from Harvard, who's, like, all, all folks that we generally know. So before a company passes by, like, the 25-person threshold, I think this is, like, relatively the same. Ultimately, though, I think it's solved with culture. Like, you, you just need to be able to, like, teach the, the ground truth beliefs, so, like, truth seeking or, like, very good at communication or, like, you know, trust or what, or whatnot that, like, build this foundation so that even if somebody who's, like, a couple degrees removed from this, like, coming in will, will, like, pick up on, on, on these

6. 17:04 – 20:56

   Hiring Fast Learners and Taking Action

   1. GLGrace Li17:04

      skills. Yeah.

   2. AGAnkit Gupta17:05

      One of the things that strikes me on that is it, it, it seems likely that the kind of person who would thrive at a company like yours today might actually be kind of different in skill set than-

   3. GLGrace Li17:13

      Yeah

   4. AGAnkit Gupta17:14

      ... a company founded by, uh, by folks five or six years ago.

   5. GLGrace Li17:18

      Yeah.

   6. AGAnkit Gupta17:18

      Um, do you have thoughts on what the almost, like, the skill pack of, like, an optimal-

   7. GLGrace Li17:25

      Yes

   8. AGAnkit Gupta17:25

      ... great engineer looks like or a great first hire or-

   9. GLGrace Li17:27

      Yeah

   10. AGAnkit Gupta17:28

       ... early hire looks like at a company like yours that's on, especially one that's on this type of really rapid trajectory?

   11. GLGrace Li17:32

       Yes. There, there's, like, one skill that kind of comes above all else, and it's, like, the willingness, belief, and ability to, like, learn anything. Um, I think this is especially critical now because things are changing so quickly that the best way to learn a thing is, like, no longer phoning up somebody with expert domain or, like, hiring somebody with expert domain. It's actually usually just, like, inhaling that, that expert domain information yourself and, like, figuring it out and, and, and whatnot. And what we found to be true is, um, bringing on somebody who has a skill is, of course, like a, um, a massive amplifier. But just as true, bringing on somebody right now who, like, doesn't have that skill isn't just, like, not an amplifier. It's a massive liability because things are, are moving at a faster rate. And so if, if you've got one person on the team who you, you're worried about, like, not, you know, leveling themselves up as quickly or, or as, um, expeditiously, that actually becomes something that you have to... It's like a, a big hole problem that gets bigger as, as things progress. So that's the number one trait that we look for in, in new hires. Um, it's, it's very rare to find, um, but what's also great is, like, once you do find s- they teach you so much. It's usually, like, an order of magnitude improvement that you'll see in the, in the company. Yeah.

   12. AGAnkit Gupta18:42

       You, you have this, uh, line you shared with me about idiots in motion.

   13. GLGrace Li18:45

       Yes.

   14. AGAnkit Gupta18:46

       Do you wanna tell us about idiots in motion?

   15. GLGrace Li18:47

       Yes. Um, so I think one very painful lesson that I've learned in the last five years or so is that an idiot in motion will always go farther than a genius at rest. I, I think I spent most of college being, like, a, a genius at rest, where I would just, like, pontificate about, like, men in the arena. I'd be like-

   16. AGAnkit Gupta19:02

       [laughs]

   17. GLGrace Li19:02

       ... "Oh, yeah, like, you should do this thing, or you shouldn't do this thing," or like X, Y, or Z. And the reality of the situation is you learn so much by doing the thing that once you start to do it, A, you drown out all of the noise of the people who are like, "Oh, you should do A," or, "You shouldn't do B," 'cause you realize how hard it is to do the thing and you're like, "Yeah, okay. All right, peanut gallery, like, you can sit down."

   18. AGAnkit Gupta19:20

       [laughs]

   19. GLGrace Li19:21

       Um, but at the same time, you, you realize that a lot of the biggest successes that will come your way is just because you were, like, kind of doing something in the area, um, before it became obvious to a bunch of people. And so then when it became obvious, you were lucky in that you were already kind of doing something in that area.

   20. AGAnkit Gupta19:39

       Yeah, it strikes me as particularly true when you are basically building a new market-

   21. GLGrace Li19:43

       Yes

   22. AGAnkit Gupta19:43

       ... that didn't already exist. Like, it'd be one thing if you were like, "Okay, I'm t- I'm gonna take an industry that already exists and I'm gonna make a better version of something in that industry."

   23. GLGrace Li19:49

       Yeah, yeah.

   24. AGAnkit Gupta19:49

       Versus I'm... The industry you're in does not exist.

   25. GLGrace Li19:53

       Yeah.

   26. AGAnkit Gupta19:53

       You guys are inventing it.

   27. GLGrace Li19:54

       Yeah.

   28. AGAnkit Gupta19:54

       And every day you're moving is a day you're experiencing new things to figure out what the shape of that industry actually looks like-

   29. GLGrace Li20:00

       Yeah

   30. AGAnkit Gupta20:00

       ... while other things are moving with you.
7. 20:56 – 25:19

   Handling Rejection and Living in the Future

   1. AGAnkit Gupta20:56

      On the flip side, though, you probably are constantly pushed up against things where you are wrong.

   2. GLGrace Li21:02

      Yeah, yeah.

   3. AGAnkit Gupta21:02

      Right? Or, or where your, your expectation of how the market works is not correct and you face a bunch of rejection-

   4. GLGrace Li21:07

      Yeah

   5. AGAnkit Gupta21:07

      ... on all sorts of elements-

   6. GLGrace Li21:08

      Yeah

   7. AGAnkit Gupta21:08

      ... of what you might be doing. How do you think about rejection, or how do you think about continuing forward and building something new while facing probably a very high rate of rejection all the time?

   8. GLGrace Li21:18

      Yes. I, I would say, like, this is the, the number one thing that I wish I could go back to my past self and, and say. I will say the feeling of rejection is way more amplified when you are the person getting rejected than when somebody is, is rejecting you. I'll, I'll give you a, a good anecdote here. Really early on, there's this, like, friend that I wanted to, to recruit, and then I, you know, I kinda called him up. I was like, "Oh, you should do this thing," like, X, Y, or Z, and he's kinda like, "Oh, I'm, like, pretty happy where I'm at," like, blah, blah, blah. And I'm like, "Oh, okay," like, you know, I, I tried X, Y, Z.

   9. AGAnkit Gupta21:43

      Like, move on.

   10. GLGrace Li21:43

       Yeah, move on, like, on, on to the next... That, that was my rejection. I was like, "Oh, pretty painful," whatever. Six months later, he calls me, he goes, "Dude, why the hell didn't you try to recruit me harder?"

   11. AGAnkit Gupta21:52

       [laughs]

   12. GLGrace Li21:52

       Like, "Honestly, I'm offended." Like, I... And then I was like, "What do you mean? Like, I, I tried." So my feeling of him rejecting me was, like, 100x, 1000x amplified on the receiving end than, than actually what he was doing. Um, and I think this is true for all of the rejection [laughs] that you face. Like, whether it's from people you're trying to hire or the market, or, like I s-... Like, the fall is always more painful to you than, than it is in, like, the objective kind of grand scheme of things. So, um, the same way that, you know, when you're doing public speaking, you're supposed to speak slower than what feels normal to you because the perceiver is, like, perceiving it to be faster than, than what you think. When you're facing, like, doing hard things and facing rejection, you should just set yourself to expect to, for it to be much more painful than, than you think it ought to be, and, like, that usually is kind of the, the right way to calibrate it.

   13. AGAnkit Gupta22:39

       And I guess when it goes right, well, when, when you don't get rejected-

   14. GLGrace Li22:43

       Yeah

   15. AGAnkit Gupta22:43

       ... you guys must be among the first in the world to discover that things are possible just all the time with what you're building.

   16. GLGrace Li22:49

       Yes.

   17. AGAnkit Gupta22:50

       Right? Like, what, what does that look like?

   18. GLGrace Li22:51

       Yeah, very good question. I mean, one of the things that makes me most excited about the company is, like, the inherent nature is you have to live in the future a little bit. Like, you have to be kind of the first fools to think the other thing is possible, where you're specifically measuring the delta between what's not yet possible and what will be in the future.

   19. AGAnkit Gupta23:06

       Oh, like, yeah, to literally even build your product correctly-

   20. GLGrace Li23:09

       Yes

   21. AGAnkit Gupta23:09

       ... you have to be living-

   22. GLGrace Li23:10

       Yeah

   23. AGAnkit Gupta23:10

       ... uh, just at the cusp of what's possible and beyond.

   24. GLGrace Li23:12

       Yeah. And I think the scariest thing about being, like, a 20-year-old right now is that you find yourself in a position where in order to do your job well, you have to, like, put blinders on to what is potentially possible. Like, in order for you to do your job as an entry-level software engineer, you have to be against the grain of, like, models replacing software engineering, for example. Um, but our company's the inverse. Like, in order to do your job well at our company, you have to have massive dreams of what the models can potentially do in order to potentially, like, create benchmarks or evals in, in that direction. So what oftentimes happens is, like, the mini experiments that we'll run will actually turn us into, like, the first people ever probably anywhere in the world to be doing a certain thing. So for example, like, recently the, the models got very good at making videos, and, like, internally we were one of the first teams to be able to, like, put out a production-level video with the kind of new model capabilities. And, like, we kind of all looked around the room and we're like, "Oh, my God, we're probably some of the first in the world to, to be able to do that." And that moment happens, like, multiple times a month, like, very, very frequently. The nice feeling is, of course, idiot in motion, but you, you also gotta be, like, in, in the right space. If you're always on the frontier, then you're never kind of lagging behind on, on, like, a set of beliefs or ideologies that, like, are, are no longer true given the reality of today.

   25. AGAnkit Gupta24:28

       So you were in YC a year ago.

   26. GLGrace Li24:30

       Yes.

   27. AGAnkit Gupta24:30

       Demo Day was a little over a year ago.

   28. GLGrace Li24:31

       Oh, wow.

   29. AGAnkit Gupta24:31

       It's now been, it's now been a very insane year for you.

   30. GLGrace Li24:34

       Yeah, yeah.

Episode duration: 25:19