Paul Buchheit: Why Evals, Not Code, Are the Real AI Moat

1. 0:00 – 1:17

   Intro

   1. GTGarry Tan0:00

      The deadline to apply for the first YC spring batch is February 11th. If you're accepted, you'll receive $500,000 in investment, plus access to the best startup community in the world. So apply now, and come build the future with us.

   2. PBPaul Buchheit0:17

      I think with AI, there's- there's sort of two forks in the road. There's- there's the bad direction and there's the (laughs) good direction. And the good path, which I think we're, you know, we're moving towards, is looking to say, how do we maximize human agency and freedom, um, and our, just potential to be kind of our best- the best versions of ourself?

   3. HTHarj Taggar0:34

      This is the first time no one's saying no. (laughs) Everyone is saying yes, and like, more. Like, there's just, like, unprecedented amounts of demand for just AI stuff.

   4. PBPaul Buchheit0:44

      There's a whole category of businesses or products that would not have been economically viable or even possible to create before, um, that are now possible. And so, we've actually just like, expanded the universe of possible businesses.

   5. GTGarry Tan0:57

      Never been a better time to be a founder, that's for sure. Welcome back to another episode of The Light Cone, and we've got a special one today, because we are in Sonoma, and we just wrapped up a 300-person retreat

2. 1:17 – 5:04

   Retreat.

   1. GTGarry Tan1:17

      of some of our top AI founders. And we also have a very special guest today, the creator of Gmail and our partner at YC, Paul Buchheit. Harj, why is this such a special episode? What are we doing here?

   2. HTHarj Taggar1:31

      Well, we're filming from a different place, so ...

   3. GTGarry Tan1:33

      (laughs)

   4. HTHarj Taggar1:33

      Um, uh, this weekend, we put on a AI retreat for some of our alumni companies to share ideas about AI and what they're seeing as they're building their startups, and we learned a bunch of really interesting stuff. So we thought we would film an episode to talk about it.

   5. GTGarry Tan1:48

      So PB, back in the day when we were working with companies, you know, what was sort of a aspirational growth rate? What would we tell people to try to do week on week?

   6. PBPaul Buchheit1:56

      Well, 10% week on week is a- is an amazing metric to hit.

   7. GTGarry Tan2:00

      Yeah, and I think back then, if, uh, you were like, maybe the top one or two perc- you know, maybe even the top one or two companies in the whole batch, you'd be able to achieve that. And since summer of last year, the wildest thing is realizing that, uh, both summer and fall batch in aggregate, on average over the batch in 12 weeks, averaged 10% week on week growth. So not just the very best, the Airbnb of the batch, but the batch overall.

   8. PBPaul Buchheit2:31

      It's amazing.

   9. JFJared Friedman2:32

      And it's not just during the batch. Um, Diana and Harj, you guys have companies that you've worked with that have continued an insane growth rate long after the batch is over. Do you wanna talk about those ones?

   10. DHDiana Hu2:42

       One of the ones that really stands out is a particular company that went from zero to 12 million in 12 months. I'd never seen any growth like that, and I think we've seen this not to be just the exceptional different company of the batch, but actually more of them do that as well, right?

   11. HTHarj Taggar3:03

       Yeah, those were- my general pickup from this weekend was that just the- the rate of execution of startups is going- is going much higher, and you can see it in both like, how quickly companies are hitting like, a million dollars in ARR. Like, we used to, I think, say you should aim for that like, 12 to 18 months out of the batch, and that's like, the equivalent of the 10% week on week growth. Like, that's what you should aspire to. Um, now, it feels to me like that's probably like, the minimum (laughs) -

   12. DHDiana Hu3:27

       Mm-hmm. (laughs)

   13. HTHarj Taggar3:27

       ... I feel like, in a AI startup. Um, we have companies hitting it within six months. And then I was just talking to some founders this weekend just about their goals for this year. Like, some of them have hit a million dollars ARR just now. Um, I- I had one company say that their goal is 20. Like, another company said they're aiming for 10 at least. Like, just-

   14. JFJared Friedman3:44

       Wait, g- going from one to 20 million ARR-

   15. HTHarj Taggar3:46

       Yes.

   16. JFJared Friedman3:47

       ... in one year?

   17. HTHarj Taggar3:47

       Yes. Yeah, and like-

   18. DHDiana Hu3:48

       (laughs)

   19. HTHarj Taggar3:48

       ... this is a goal. Founders have goals, and like, we help them hit- we hope they hit them. But my point is, I think like, saying that, like even a couple of years ago, "Hey, my goal is to go from like, one to 20," this-

   20. JFJared Friedman3:58

       People would have thought you were just asking.

   21. HTHarj Taggar3:59

       Yeah, you would have just- you would have been either like, uh, like, just like, "That's total nonsense. Like, that's never gonna happen," or like, um, it just- you just wouldn't have said it. And I just think that the general level of ambition has gone way up because of AI.

   22. DHDiana Hu4:10

       The things are starting to work, and let's talk about why that is the case. Does anyone have any thoughts on that?

   23. GTGarry Tan4:16

       Well, I guess I have a meme that I showed you guys earlier. You know, I think the classic thing is you have a boss who's sort of like, slave driving, and then, you know, I s- still believe this, like, if you're a leader, you're not, you know, sort of slave driving from the back. You're way in the front, like, sort of leading. And then the meme has, you know, this one person pulling the cart, uh, alone, (laughs) uh, and that's the introvert. And what might happen now is the introvert with AI can pull (laughs) three times as many carts all alone, actually. (laughs) Once intelligence is truly on tap, then it's actually a force multiplier for founders and people with really, really strong senses of agency.

   24. HTHarj Taggar4:53

       Should be- wh- why- why specifically is this happening? One of the interesting talks I heard was from Aaron Levie, the C- CEO of Box, and he was talking about, um, he's been through like, multiple cycles of enterprise

3. 5:04 – 8:00

   Demand for AI.

   1. HTHarj Taggar5:04

      software. Uh, and he said that usually when there's sort of like, a new cycle shift for, like, cloud or mobile, there's always people in the, um, room, decision-makers at the big enterprise software companies saying like, "No," like, you know, like, "We're never gonna shift to cloud." There's apparently like, a famous quote from Jamie Dimon, um, where like, whenever like, "Mobile's not gonna be a thing. It's not that important." Um, but with AI, it's different. Like, this is the first time no one's saying no. (laughs) Everyone is saying yes, and like, more. Like, there's just like, unprecedented amounts of demand for just AI stuff.

   2. JFJared Friedman5:37

      Yeah, it's notable that all these companies that are having these incredible growth rates, they're all the same flavor of startup, right? They're all basically selling AI agents to businesses. I mean, the- there's other companies that were-... funded that are doing well, but, like, all of the ones that you guys were talking about, right, they're all AI agents for businesses.

   3. HTHarj Taggar5:52

      Yeah.

   4. JFJared Friedman5:52

      And so they're all essentially, I think, riding on this wave of, like, enterprises are, like, have enormous pressure internally to adopt AI.

   5. PBPaul Buchheit6:00

      This seems like it's, goes back to our fundamental base advice, which is make something people want. And in this case, traditionally, the challenge was convincing people that they wanted the product.

   6. HTHarj Taggar6:09

      (laughs)

   7. PBPaul Buchheit6:10

      And I, it sounds like what's driving the growth is that the demand is already there.

   8. JFJared Friedman6:14

      Yeah.

   9. PBPaul Buchheit6:14

      And so you just have to show up with a product that works, and you don't even have to be that good at sales.

   10. HTHarj Taggar6:19

       The point is that actually building the product that works is quite hard. Like, a lot of, a lot of what the, the demand we're seeing is not for, is for software that can actually do the work of a person. So it's essentially services. Um, and doing that to the equivalent level of a human doing the job, whether it's, like, customer support, sales phone calls, whatever it is, um, is actually very, very hard. So I think just, like, a trend I noticed is a lot of our heavily technical CEOs, who aren't necessarily the strongest at sales, are able to win big enterprise contracts now. Because although there's 10 or 15 other companies competing for the same contract, um, it's very, very hard to build a product. And so just building the thing that actually does the work well is enough to win these huge deals.

   11. DHDiana Hu7:04

       A lot of the details of how they build the products, they're really inventing a lot of new patterns on how to build the products. Because nobody knew how to really get LMs to behave, let's say, correctly, and give very predictable results, and people thought that was impossible. That's because they only tried maybe surface-level. They play with ChatGPT, and oh, sometimes hallucinate, and then people give up. That could be the, the random person just does that. But a lot of the technical founders, they don't. They find ways and wizardry around on how to really state a problem, how to properly prompt it to actually be very accurate. And it is possible, because we're actually seeing a lot of these products getting bought at businesses to handle all these complex tasks.

   12. JFJared Friedman7:50

       One thing I noticed this weekend is that a lot of the talks that the founders gave were around evals and, like, testing, which I don't think that would have ever been true at a previous

4. 8:00 – 10:00

   Evals.

   1. JFJared Friedman8:00

      YC conference. Like, testing was sort of this, like, afterthought thing-

   2. HTHarj Taggar8:03

      Yeah. (laughs)

   3. JFJared Friedman8:03

      ... that you try to do as little of as possible. Um, I, I heard one really interesting comment from a founder, um, who's building an AI agent, who said that he thinks that the most valuable thing that his company has built is not the code base. It's the eval set. It's a gold standard labeled set of data of, like, this is what, this is the correct answer for the AI to do. And, uh, that, that was sort of a mental change for me, that, like, there's, I think, this perception that would, that, like, companies have, like, data assets. But just, like, general random data is actually not that valuable. The thing that's really valuable is, like, a gold standard, meticulously labeled eval set.

   4. GTGarry Tan8:42

      I mean, this is exactly why the whole ChatGPT rapper meme is wrong, in that actually it's the model that is changing very quickly. There are clearly five or more AI labs, all of which are right there at the frontier. So, now there's a lot of alternatives to which model, but then the thing that, you know, nobody has that is actually hard to get is the eval set, and I'd argue the prompting. Which is sort of, like, mirrors basically all of the opportunity for the people watching right now. It's, like, basically agency and taste prompting, just knowing what to tell someone to do or, you know, what to tell the agent to do (laughs) , and then eval's taste. It's like, is it good? Is it beautiful? Is it useful?

   5. HTHarj Taggar9:30

      I heard one very interesting tidbit from a founder who said that his designer, they- they've actually stopped using Figma mockup things, and the workflow is interesting. So the designer is designing entirely with Claude and going from text to JavaScript. It was just counterintuitive to me 'cause you assume design is this, like, very visual-

   6. PBPaul Buchheit9:49

      Visual.

   7. HTHarj Taggar9:50

      ... thing. But apparently their designer has figured, like, it has enough taste to be able to turn that into just, like, text prompts and, like, via prompt engineering essentially,

5. 10:00 – 14:37

   Product iteration.

   1. HTHarj Taggar10:00

      get to a actual, like, lines of code that, like, are, like, as tasteful and as good as any Figma mockup would have been.

   2. PBPaul Buchheit10:09

      So it's like the pattern, as always, is whoever can iterate the fastest wins, right?

   3. HTHarj Taggar10:13

      Yeah. (laughs)

   4. PBPaul Buchheit10:13

      And AI is an incredible tool for rapid iteration.

   5. GTGarry Tan10:16

      I guess people are, you know, sort of worried that the jobs are gonna go away, but earlier we were talking about, um, there's this great Milton Friedman, uh, quote, where he's visiting a developing country.

   6. HTHarj Taggar10:28

      (laughs)

   7. GTGarry Tan10:28

      And he sees this large group of workers digging a canal using shovels, and he asks, uh, his government official host, you know, "Why are you not using machinery? What's going on?" And the guy says, "It's a jobs program, (laughs) actually." And, uh, what does Friedman say? (laughs)

   8. PBPaul Buchheit10:45

      He says, "If it's a jobs program, you should give them spoons, not shovels."

   9. GTGarry Tan10:48

      I think that that's actually the most useful mental model for sort of the fear about job loss, at least for now.

   10. PBPaul Buchheit10:56

       Certainly, you know, the, the potential of AI is because it is this incredible tool, where we're moving not from spoons to shovels or shovels to bulldozers, but to the point where the AI can do so much work that we're actually just able to create dramatically more wealth. And I think that's really the dream that we have for, you know, peering 10 years into the future, um, is I, I think there's a potential for an unprecedented level of certainly scientific discovery. Um, the AI is incredibly good at reading thousands of papers, um, you know, digesting textbooks, very good at chemistry. Um, so I think we're gonna see incredible levels of productivity.

   11. GTGarry Tan11:37

       The story is fascinating to me because the alternative is what? Replacing people's shovels with spoons. Like, I, you know, I think it's absurd on its face.

   12. PBPaul Buchheit11:47

       Right.

   13. GTGarry Tan11:48

       Like, who would ne- Like, that actually is a little bit like torture. Like, I feel like-

   14. PBPaul Buchheit11:51

       Right.

   15. GTGarry Tan11:52

       ... you know, growing up, my dad would force me to work in the gardens.

   16. JFJared Friedman11:55

       (laughs)

   17. GTGarry Tan11:56

       That alone was barbarous to me, (laughs) but, like, if you made me do it with a spoon, what would we call that? That'd be torture, actually.

   18. PBPaul Buchheit12:01

       Yeah, yeah. I think the question that, that I, I posed, uh, to Sam that, that everyone-... seemed most interested in is, is essentially, you know, are any of these startups actually gonna exist-

   19. GTGarry Tan12:12

       (laughs)

   20. DHDiana Hu12:12

       (laughs)

   21. PBPaul Buchheit12:12

       ... in 10 years?

   22. DHDiana Hu12:12

       That was certainly relevant to the audience, yeah. (laughs)

   23. GTGarry Tan12:15

       Yeah. (laughs)

   24. PBPaul Buchheit12:15

       Yeah. Every- everyone felt that was, that was relevant because, yeah, if we, if we do achieve AG- AGI, you know, what does that mean? How, how quickly can that actually just displace all of the work that we're doing here? And honestly, like, no one's quite sure. (laughs)

   25. GTGarry Tan12:29

       (laughs)

   26. DHDiana Hu12:29

       (laughs)

   27. PBPaul Buchheit12:29

       Which makes this a very exciting time in technology. Um, but, you know, it's, it's, it's very clear that we're able to achieve a lot more. And I think, like, throughout history, every time we've found ways to create more wealth more rapidly, that's actually worked out really well. You know, historically, 97% of people were farmers or something like that. And now, it's, you know, I think maybe 3% or, or, or even less. Um, and so we seem to be very good at inventing new work for ourselves and new, new ways to, to find purpose and meaning.

   28. GTGarry Tan13:01

       Well, what was the answer to, uh, your question? I think, uh-

   29. PBPaul Buchheit13:04

       Luxury real estate-

   30. DHDiana Hu13:05

       (laughs)
6. 14:37 – 16:16

   Balance.

   1. GTGarry Tan14:37

      I mean, a bunch of the studies around UBI are sort of showing that there are sort of nice benefits here and there. But fundamentally, it's not, uh, creating a greater sense of well-being in the way that people hoped maybe like five or 10 years ago.

   2. PBPaul Buchheit14:51

      Yeah. It's, uh, definitely had mixed results. And I think a lot of that really comes down to, um, you know, people still need guidance in life, and especially a lot of the people who are targeted with UBI are, are not necessarily people who are in a great sort of social position to begin with. And again, I think that there's a lot of potential for AI to actually kind of act as like a life coach. And again, like, you know, if you're fortunate enough to grow up with great parents and a great culture or something, you have a lot of advantages that a lot of other people maybe didn't have. Um, and so again, I think like the great promise of, of AI is kind of taking like the best of what we have available and then just making it universally accessible because we're able to drive the cost so low.

   3. GTGarry Tan15:31

      I mean, honestly, this past December, I spent a couple weeks in Vietnam, and you're in a developing country, and then you realize like, there is so much that needs to be developed. It's, uh, you know, it's actually, you know, there's roads, there's infrastructure. Like, the whole country seems like it's under construction. I imagine that's what China probably looked like in, you know, maybe the mid-'80s or mid-'90s.

   4. PBPaul Buchheit15:52

      Yeah.

   5. GTGarry Tan15:52

      But there's also like this crazy optimism as it's building. If you have a robot, like, it could build your house, it could clean your house, it could, you know, take care of all of these things for you. Uh, and that would like radically change your day-to-day, your standard of living. And how much more direct can you be if you, you know, rather than sort of just giving people, uh, more human money-

   6. PBPaul Buchheit16:14

      Yeah.

   7. GTGarry Tan16:15

      ... let's give them like

7. 16:16 – 19:31

   Automation.

   1. GTGarry Tan16:16

      the better, uh, way to live. And that's sort of, you know, everyone can get here, but then I think there's like a, a special thing here around the human money, where like the really remarkable things, like, you know, nobody's guaranteed to have, you know, beachfront property in California-

   2. PBPaul Buchheit16:31

      Right.

   3. GTGarry Tan16:31

      ... or something like that. And that, that's where human money might go. Like-

   4. PBPaul Buchheit16:35

      For sure.

   5. GTGarry Tan16:35

      ... everyone has the basics, and actually something that's probably five or 10 times better than what even the most wealthy people have today.

   6. PBPaul Buchheit16:44

      Yeah. Yeah, the way I like to think about it is I, I kind of think with AI, there's, there's sort of two forks on the road. There's, there's the bad direction and there's the good direction. Um, and I think the bad direction is, is one where it's used to just like constrain and control and, and essentially like imprison us. Um, and, and, and the good path, which I think we're, you know, we're moving towards is looking to say, "How do we maximize human agency and freedom, um, and our just potential to be kind of our best, the best versions of ourself?" And we even have that, you know, today with some of the creative tools, where, you know, I don't have like a lot of artistic ability, but with like AI image generation or something, I can convey, you know, funny concepts or whatever. Um, and, and we see that again, like with the design tools, you know, someone who can't code can now all of a sudden create basic apps and things like that, and we're able to, um, realize our visions in a way that we were never able to do before.

   7. DHDiana Hu17:38

      A conversation we were having earlier is how we are actually now in the good timeline on how AI is shaping.

   8. PBPaul Buchheit17:44

      Yeah, absolutely.

   9. DHDiana Hu17:45

      Because 10 years ago, this was very different view how AI could turn out. Y- you wanna kinda talk about that?

   10. PBPaul Buchheit17:49

       Yeah, exactly. So I, I, you know, I like to think about things on a time, 10-year timescale, um, in part because, you know, that's kind of how our startups work, roughly speaking. We seed fund them, they come through YC, and then 10 years later, they, they IPO. Um, and so I've been asking a lot of people about the year-... 2035, you know, what do you want to see in 2035? Um, but also thinking backwards to 2015. And so if we go back to 2015, you know, 10 years ago was where we were having discussions inside of YC about artificial intelligence, because we believed that we had sort of crossed a threshold basically in the early teens. Somewhere around 2012 is where we started to really believe that actually f-... we had broken through. I think everything kind of prior to 2012 was fake, in my opinion. Um, (laughs) but it, it was really deep learning, um, that, that, that s- started to really deliver on AI. Um, but when we were looking at this 10 years ago in 2015, th- one of the big questions was, it was all reinforcement learning, and what is the thing that we're reinforcing, 'cause at the time, they were playing video games and trying to make the score go up. Um, and, and this is, I think, also kind of where the paperclip maximizer concept and fear came from, is like if you gave it the wrong objective function. And so we had a lot of fear that based on our own evolution, our intelligence arose as a survival mechanism, that we became intelligent and, and other animals became intelligent as a way to survive and perpetuate themselves. And we thought that if AI did the same thing, it would, by its very nature, want to, like, wipe us out in order to maximize its own odds of survival. Um, and what's happened in the last 10 years is we

8. 19:31 – 22:21

   Predictive models.

   1. PBPaul Buchheit19:31

      actually found the right objective function, which is simply to predict the next token. Um, (laughs) and that actually intelligence in its most raw form is simply predicting what comes next. And so all of our really at a root level what we're, we're predicting, what our reinforcement function is simply predicting what comes next, and that is the fundamental core of intelligence. And the great thing about that is we've been able to create this intelligence that doesn't have this drive to survive. It doesn't mind that we spin up an intelligence and it does some work and then it disappears, um, because it's, it's just based on, on that ability to predict patterns.

   2. GTGarry Tan20:10

      I would argue, like, the most important part of this is actually the agency piece. Venkatesh Rao has this crazy thing, uh, he, he talks about, um, and this is sort of a, a function of, you know, maybe the Uber and DoorDash era of things, where in society there's this, like, API line. So, um, either you are above the line, meaning you create Uber, or you, uh, drive for Uber.

   3. PBPaul Buchheit20:31

      Mm-hmm.

   4. GTGarry Tan20:32

      And, uh, obviously that's a distillation of, like, sort of the last idea. And then in this AI world, basically b-... if you're below the API line in the old model, like, you don't have agency. You sort of have to, like, you know, play this, uh, never-ending game. Like, the human is being... doing the paperclip maximizing. And so there's this other sort of world that I'm hoping we live in, where it's humans not just writing the prompt and then the machine runs software, and then this vast machinery and that's it, and you can never change the prompt. Like, that would be tyranny probably. It's conceivable that in the future we might have... and I don't know if this is the right thing, but, you know, this so-... sounds like something the EU would do, for instance.

   5. PBPaul Buchheit21:12

      (laughs)

   6. GTGarry Tan21:13

      It would probably mandate there to be a human in the loop on... you know, maybe the CEO of a company has to-

   7. PBPaul Buchheit21:19

      Spoons. (laughs)

   8. GTGarry Tan21:19

      (laughs) Yeah. And that might be the case, right? That, you know-

   9. PBPaul Buchheit21:22

      Yeah.

   10. GTGarry Tan21:22

       ... it might be a form of, like, we cannot use shovels here.

   11. PBPaul Buchheit21:25

       Right.

   12. GTGarry Tan21:25

       Like, we must use tiny little spoons just for this one part.

   13. PBPaul Buchheit21:28

       Right. Yeah, I think kind of the fundamental error they keep making over and over again is taking a very static view of the world and then s-... essentially trying to just regulate the current structure, um, a- a- and that closes off our ability to, uh, to evolve and see into the future. And it's, you know, again, very difficult and oftentimes implausible. So, you know, again, going back to 2015, the conclusion of our, of our thinking was actually that we needed to create our own AI lab, because at the time, you know, all of the best AI work was being done at Google and they had... you know, Google had all the money, all the data, all the users, all the researchers, um, and it kind of seemed possible that they were gonna have essentially a monopoly, and it was all gonna be locked up inside of that system. And so we had this very, um, sort of, like, loony moonshot idea to start... at the time we called it YC Research, but it eventually got renamed, um, OpenAI.

9. 22:21 – 24:58

   OpenAI.

   1. PBPaul Buchheit22:21

      And so OpenAI, you know, we were gonna take on Google with a small nonprofit, which was... sort of doesn't quite pass the laugh test, right?

   2. GTGarry Tan22:28

      (laughs)

   3. PBPaul Buchheit22:28

      Like, like, how is this little nonprofit, uh, you know, gonna... going to be the one that, that actually develops AGI when, you know, the other companies have dramatically more resources? And then here we are 10 years later, it actually happened. And at the time, it just seemed incredibly implausible. Like, no one would have believed it, yet here we are on, I think, basically the best timeline. Like, we actually delivered it. We have, uh, an open and competitive market with I would say at least kind of like six basically, uh, you know, y- you know, foundation models that are competing, um, including an open source one from Meta, and I think that's our best shot for preserving freedom, is, is, is, is choice and competition.

   4. DHDiana Hu23:08

      And talking a bit about Google, it actually is digging their traffic too. Do you want to talk a bit about that? About the, the stat that we're-

   5. GTGarry Tan23:15

      Yeah. I mean, some of it is, like, I don't think it's out there in the annual reports yet.

   6. PBPaul Buchheit23:19

      Yeah.

   7. GTGarry Tan23:19

      And certainly, like, we did some, uh, you know, research prior to this episode. We couldn't really find, uh, anything that conclusive. But maybe purely anecdotally, because we're in this pool of people who are very, very early adopters, very much software engineers, and, you know, our behavior interacting with the internet has changed already, it's not a surprise to me some people are starting to report, uh, in their referral traffic, Google referrals are down maybe 15% in the last year, and that certainly probably mirrors my own behavior. Like, I still use Google, but I'm includ-... increasingly not clicking on any links in Google, because there's sort of the snippet at to-... at the top, or the first thing I think of is using ChatGPT with web or using Perplexity directly.

   8. PBPaul Buchheit24:03

      Yeah, exactly. I mean, if you want to understand the future, I think you always have to look at where the early adopters are. And so you say, (laughs) you know... again, now if we go back 25 years, (laughs) right? If we go back to, to, to the year 2000 or 1999, you know, the early adopters were the people using Google. So at the time, you know, people were like, "Well, Google is just kind of this fringe thing that, you know, maybe tech-... techie people use or something." Um, but-At this point in history, those same people, or those same kinds of people who were the early adopters of Google, um, are now switching their behavior to where your default action if you're looking for information is, you know, ChatGPT, or Perplexity, or one of these things. Um, and even just, you know, observing my own behavior, I'll use Google mostly for kind of navigational, like if I'm just looking for a specific website, and I know it's gonna give the same thing. But it's starting to have that weird kind of like legacy website, like I'm using eBay or something,

10. 24:58 – 26:23

    AI tools.

    1. PBPaul Buchheit24:58

       vibe to it.

    2. DHDiana Hu24:59

       Even earlier sign was the drop in traffic for Stack Overflow that actually started back in 2022, even before ChatGBT, and this was primarily because of, uh, GitHub Copilot.

    3. GTGarry Tan25:10

       And they're down 60% this year.

    4. DHDiana Hu25:12

       Yeah.

    5. HTHarj Taggar25:12

       Yeah, the pool of people here are quite, have quite a good track record of predicting trends (laughs) , right? If you think of, um... Uh, by the pool, I mean just like technical startup founders at YC. Like I remember 2007, sort of Apple was back on the rise, but you could tell because just everybody who was in a YC batch was using a Mac. Uh, you could see the rise of AWS and like the shift from rack service to everything being in the cloud because all of the founders in the batch just started like using AWS. Same thing now. I've, I've spoken to a bunch of founders of just personal productivity, like they just have ChatGBT open all day.

    6. DHDiana Hu25:47

       Mm-hmm.

    7. HTHarj Taggar25:48

       They had founders like say they're just constantly screenshotting their desktop and just like sending it to ChatGBT if they need to debug something or figure out how to, like, navigate a government website.

    8. PBPaul Buchheit25:56

       (laughs)

    9. HTHarj Taggar25:57

       It's like one random example is like, "I need to do, set up some registration, like, you know, here's a screenshot. Just tell me, like, exactly where I need to click in order to do this quickly."

    10. DHDiana Hu26:05

        Uh, the only thing that we saw last year in the summer batch was how so much of the batch was using Cursor and is one of the companies that's been growing a lot, uh, very quickly. We're... Anecdotally, they hit $50 million in revenue.

    11. HTHarj Taggar26:19

        I think we, we may have mentioned this in another episode, but yeah, I can't think of another tool that's

11. 26:23 – 29:30

    Cursor.

    1. HTHarj Taggar26:23

       got adoption so quickly within a YC batch as Cursor. It just went from nothing to...

    2. DHDiana Hu26:28

       From one batch to the other (laughs) .

    3. HTHarj Taggar26:29

       Yeah (laughs) .

    4. DHDiana Hu26:30

       It went to... Like, up to like 80%-

    5. HTHarj Taggar26:31

       Yeah.

    6. DHDiana Hu26:31

       ... of the batch using it from one to the other when the previous batch was like single digit percent.

    7. GTGarry Tan26:35

       They had some people mentioned it felt like a, like a technical conference a little bit. And a lot of people were trading notes on how to hire the best engineers. And a few people said, "You know what? Like if someone comes in, and I ask them if they use Cursor or any code gen tools, and they say no, right now I can't hire them 'cause they're not (laughs) going to be able to be as productive as the rest of my team."

    8. HTHarj Taggar26:57

       I think that's an extension of, um, something Stripe started a decade ago, actually. Like, in general, engineering interviews and technical interviews, most of them valley- copied Google, I would say. It was like whiteboard CS problems, which probably made sense for Google and what Google was looking for. But I think Stripe were the first, around like 2011 I think they started doing this, where like, "We don't really need you to whiteboard CS problems. We need you to develop web apps really fast. And so just give someone a laptop." And like the idea was you basically sit in a room, and you just like build like a to-do list app or whatever you can, as quickly as you can. And you're basically measured on your, like your max output in those like two or three hours. And so I think if you follow that line through, then it's like, well, it doesn't really matter. Like if there's, whatever tools they use, the question is just like the bar moves higher. Like, you've got three hours, build what you can build. And it's just like you should be able to build a lot more with Cursor than before. If you still believe that you're sort of looking for fundamentally how clearly can people think or solve hard, hard architecture problems, then you're sticking to whiteboards.

    9. GTGarry Tan27:56

       What do you think this means for, uh, SASS? 'Cause you know, one of the crazier things we've been seeing is that Klarna claims that they're not even buying new SASS tools anymore. They're using CodeGen and, uh, not even hiring new engineers anymore. Using their existing engineering set of people. They're just gonna replace the, all the SASS tools they use to run their fintech. And, uh, I definitely heard stories like that. One of the unconference talks was actually specifically about that. This is a company I think we mentioned before, is a company called Jerry that is now halfway to $100 million a year in revenue. But a few years ago, they were like still burning like five or $10 million a year. They had crazy customer support problems and basically GPT-4 dropped, they implemented it, and then now it totally changed the way they hire. Like, the prompting itself is actually in the hands of their head of customer support. And so they have a PM, they have the head of customer support. The engineers made it, they don't have to touch it. It's mainly a prompt management and workflow tool, and it literally, uh, cut their customer support, uh, team, and their budget for that side by half. And it turned a company that was not able to grow, uh, and burning $10 million a year to a profitable company that is cash flowing, that is also compounding its growth at north of 50% a year, which is like a dream scenario.

    10. PBPaul Buchheit29:22

        Yeah. This is a great example, actually, I think of the, the way in which AI is creating wealth, right? Because there's a whole category of businesses or products

12. 29:30 – 33:16

    Scaling.

    1. PBPaul Buchheit29:30

       that would not have been economically viable or, or even possible to create before, um, that are now possible. And so we've actually just like expanded the universe of, of possible businesses.

    2. GTGarry Tan29:42

       Yeah. It's never been a better time to be a founder (laughs) , that's for sure.

    3. HTHarj Taggar29:45

       There's definitely been a vibe shift in, um, the attitude towards just building companies, like, uh, there's like to- start with like hiring a number of people, for example. If you're like... Certainly 10 years ago, the general sense was if you were growing, if the company was growing fast and revenue was ramping up, then you would go out and raise round, and you would sort of... A, a metric you would hear a lot was like, "How many people are you at?" Like, "How many people did you hire this year? How many people are you gonna hire next year?" So like a bit of a vanity metric. Um, it just seems to me now, like the companies that are reaching these numbers we're talking about, like a millionaire are trying to get to 10 or 15 or 20, are doing it with less people and expect to do it with less people.

    4. DHDiana Hu30:22

       Which is the new thing. This is why so many of them really haven't even raised to Series A, which the, there's less need for...... for hiring a lot of people to do a lot of the operations on, or, or maybe going to your analogy, Gary, the previous generation of, uh, startups had this concept below the API or above the, below or above the API, so you had a bunch of people that had to kind of build and operate the API, like if you had to build a business like Uber or Lyft, DoorDash, marketplaces, you had to do that hyperscale of hiring lots of people.

    5. GTGarry Tan30:52

       The f- funny thing about that era was there's this concept that I think was probably appropriate for that era called blitzscaling. Uh, there was an entire book about it, and the idea was basically, I think it was born out of this descending interest rate world, while at the same time, like, if you put more money into something, like you had these network effects. So if you played that out, yeah, you want to blitz- blitzscale. You want to hire as many people as possible, you want to grow faster than everyone else, and then because of the winner take all dynamic, like, the world capital markets were just gonna funnel you tens of billions of dollars, hundreds of billions of dollars even, to, you know, subsidize growth to be the winner. And, you know, that was the game. And I think, like, from what we can tell from all the people here, we have more than 300 founders right here sort of sharing their stories, and I don't think I heard blitzscaling or I'm trying to hire as many people as possible, like, at all. Nobody is bragging about, "Hey, you know who I'm hanging out with? These unicorn, you know, I'm gonna be a unicorn." Like People are literally not bragging about that.

    6. PBPaul Buchheit31:56

       Yeah. It's all about leverage, right?

    7. DHDiana Hu31:57

       Yeah.

    8. PBPaul Buchheit31:57

       Now, now the, the real thing is how much you can do with a little bit of resources because we have these magical tools that give us-

    9. GTGarry Tan32:04

       Yeah.

    10. PBPaul Buchheit32:05

        ... superhuman leverage.

    11. DHDiana Hu32:07

        Part of it's like this, there's gonna be a longer tail of businesses that are possible only now because of AI, and this longer tail is gonna be also fatter. It's not just companies that are doing 20, 30 million revenue, but more than hundreds of millions of revenue. And it goes back to the episode when we talked about vertical SaaS. There's just more willingness to pay for this new category that people are still trying to figure out how to price.

    12. PBPaul Buchheit32:30

        Yeah.

    13. DHDiana Hu32:30

        That's why there's just so much willingness to pay because people want it. It doesn't go just on the software budget for a company, is there's budget from the AI chief officer or something. I don't know if that's like a title that has come out yet, but...

    14. HTHarj Taggar32:43

        I really made this point too that, I mean, one thing I'm certainly noticing is that the, the companies that are hitting these big revenue numbers, trying to sign these contracts, it's actually sort of usage-based pricing, um, uh, in the day. Like, it's not necessarily they're paying per use, but the pricing is tied to like how much you use the product, which is definitely how, i- it's close to how you would think about it as like selling services and software per se.

    15. PBPaul Buchheit33:05

        Obvious ROI, right?

    16. HTHarj Taggar33:07

        Yes.

    17. PBPaul Buchheit33:07

        So, so the problem a lot of times with selling a product is, is the customer doesn't really know if they're getting the ROI, and so that makes for a long and painful sales cycle.

13. 33:16 – 36:37

    ROI.

    1. HTHarj Taggar33:16

       Yeah.

    2. PBPaul Buchheit33:16

       But if you're able to drop in something that pays for itself in the same month-

    3. HTHarj Taggar33:19

       Yeah.

    4. PBPaul Buchheit33:19

       ... that's an easy sale, right?

    5. HTHarj Taggar33:21

       Yeah.

    6. DHDiana Hu33:21

       I think the way they've kind of priced it is more like services and is really akin to this is how intelligence is getting priced.

    7. HTHarj Taggar33:28

       So know that, that on the, on the spectrum of like people thinking, not worrying about just, uh, the big picture, is AI gonna make us all obsolete on the one end, and the other end like existential, like philosophical conversations, some of the stuff I thought that's interesting in the middle is it's just hard to predict the timeline of the tools themselves. So there was some interesting talks about like RAG for example. I, I think Sam maybe seeded this with, with his talk about like if you have like infinite context, well, huge context windows, do you even need like RAG or retrieval tools at all? And I think that's like, that's the kind of thing where it's like it's, as a startup or a builder right now, like I think people are more concerned about, am I using the right tools and like is this gonna still make sense-

    8. GTGarry Tan34:07

       Yeah.

    9. HTHarj Taggar34:07

       ... in three to six months?

    10. GTGarry Tan34:09

        I think that's like actually a direct consequence of like, you know, if you're an AI lab, you're like on the frontier and so how you know that your thing is working is actually like your model is bigger, you're like farther along on the scaling law. And so when I meet people from AI labs, like they almost all talk about bigger, better models, but they're model makers.

    11. PBPaul Buchheit34:30

        Mm-hmm.

    12. GTGarry Tan34:31

        And then obviously, you know, we also spend a lot of time with very scrappy founders-

    13. PBPaul Buchheit34:35

        Yeah.

    14. GTGarry Tan34:35

        ... who have very little capital. There are just as many talks, um, sort of on the other side, which was, you know, I, I went to one that was very much about systems level programming. Mm-hmm. Like if you want to have, I think it was, uh, Tavis.

    15. DHDiana Hu34:48

        So Tavis is building this real time AI avatars with video and audio that are very realistic. Part of the trick is they got it to very low latency, which is-

    16. GTGarry Tan34:59

        Yeah, 600 milliseconds, which is really fast.

    17. DHDiana Hu35:02

        Which was even too fast that some of the customer, "Oh no, no, no, it's too fast."

    18. GTGarry Tan35:05

        (laughs)

    19. HTHarj Taggar35:05

        (laughs)

    20. DHDiana Hu35:05

        It's a bit uncanny when it's too fast.

    21. GTGarry Tan35:07

        And it's like th- now it's being rude. It's just interrupting me every time. (laughs)

    22. DHDiana Hu35:11

        Yeah. So a lot of, uh, the, they build this, uh, SDK for, uh, other companies, so a lot of the products that are getting built with this Zoom video interface with another human, it is using them.

    23. GTGarry Tan35:23

        So I, I love their talk 'cause it's, uh, a good illustration of like, yes, like the labs are going to continue to do their thing, you know, and maybe on a more fast timeline than we even imagined. Like, you know, nine months, 18 months, like, you know, maybe it's even every three months there's sort of these like breakthroughs. So if I had to guess, like that's sort of, you know, when people are in their heads being like, "Why should I do any of this because OpenAI models are just gonna be infinitely smart and I should just lie down on a bed." You know? (laughs)

    24. HTHarj Taggar35:51

        (laughs)

    25. GTGarry Tan35:51

        But you know, what I would say is like I'm actually heartened by all the stories that I heard, like will the models change? Will the technology change? Will, you know, will Tavis change its stack? Like yes, like they've already seemingly rewritten their stack multiple times to take advantage of what's been going on. Their product has only gotten better in the marketplace as time goes on. And then what will it look like? Will there be like a, you know, a model model? Maybe not. The same AI labs today that are talking about, you know, there's gonna be a, a trillion token context, it's like, man, how much is that gonna cost? Ultimately engineering and systems, like those matter. Those are actually the most valuable things right now. And then along the way you're gonna have these golden evals. I don't know, I hate to bring in, you know, consulting speak, but like what are the moats, right? And the moats

14. 36:37 – 38:30

    Startup success.

    1. GTGarry Tan36:37

       in the end are brand, it's uh, you know, data that no one else has. Uh, sometimes it actually literally is caring about customers that, you know, the giant company will never care about, right?

    2. DHDiana Hu36:50

       Actually, I think the other moat is going to be ultimately startups move quickly. One of the re- remarkable things that I observed...A lot of the founders actually had rebuilt a lot of their tech stack, uh, to be with the latest. They were very willing to, "Oh, this particular approach to RAG doesn't work on vector database, throw it away." PG Vector became the better thing. Let's use that, and just throw it away and use the best thing. So what was fun to see is, I think the best startups are gonna be the ones that can build the fastest and be willing to be at the bleeding edge, and be willing to reevaluate assumptions on what's the best approach. And I heard a lot of how, a lot of how things got built. They redo it, or they do it again with the best, with the latest and best.

    3. HTHarj Taggar37:32

       Which would also ex- another reason explain why they're securing enterprise contracts and these big contracts faster than ever, right? Like, big companies have never been great at continuing to build great software. But now, like yeah, if you need to constantly rip and replace the tool you're using every three months (laughs) -

    4. DHDiana Hu37:48

       (laughs)

    5. HTHarj Taggar37:48

       ... to be like at that bleeding edge, like it's gonna take three months to get, like, the meeting scheduled to discuss-

    6. SPSpeaker37:53

       (laughs)

    7. HTHarj Taggar37:54

       ... like (laughs) -

    8. SPSpeaker37:54

       (laughs)

    9. HTHarj Taggar37:54

       ... like whether we should reevaluate the tools.

    10. DHDiana Hu37:55

        We're gonna plan in the next ... and it'll take, whatever. It's gonna take like three months.

    11. HTHarj Taggar37:58

        Yeah. We can get to that in like, you know, 2029 for sure. (laughs)

    12. SPSpeaker38:02

        (laughs)

    13. DHDiana Hu38:02

        Yeah. And these companies are getting to the six or 12 million example, I know they actually have rewritten a lot of their tech stack a lot of times and the architect, every time I actually talk to them, "Oh yeah, we threw away that thing that you, that we told you." It's like, "I- it's this new way of doing it." It's like, "Okay." And that's like every month, every other month.

    14. SPSpeaker38:20

        From talking to founders this weekend, what was your sense of the overall vibe?

    15. PBPaul Buchheit38:24

        I think it's pretty exciting. I mean, I, I, I don't know that there's ever been a better time. You know, again, just kind of looking back historically,

15. 38:30 – 39:32

    Outro

    1. PBPaul Buchheit38:30

       really the foundation of YC, if we jump back not 10 years to when we were starting OpenAI, but 20 years to the Summer Founders program, the, the thesis behind why Paul and team started, um, YC was the realization that it was getting easier to build startups. You know, you didn't need to raise a mountain of capital and hire a giant team. That actually just a couple of smart kids could build a web app, um, a- a- and that trend has only accelerated now with AI where you can build, you know, an entire (laughs) , uh, $12 million business or something with just a handful of employees. And so it, it again goes back to technological leverage enables, um, people who have sort of ambition and insight to do incredible things.

    2. SPSpeaker39:13

       Well, that's all we have time for today, but we'll catch you next time on The Light Cone. (music)

Episode duration: 39:32