Kevin Weil: Why evals are the new core skill in AI products

1. 0:00 – 5:16

   Kevin’s background

   1. KWKevin Weil0:00

      The AI models that you're using today is the worst AI model you will ever use for the rest of your life. And when you actually get that in your head, it's kind of wild. Everywhere I've ever worked before this, you kind of know what technology you're building on. But that's not true at all with AI. Every two months, computers can do something they've never been able to do before and you need to completely think differently about what you're doing.

   2. LRLenny Rachitsky0:21

      You're Chief Product Officer of maybe the most important company in the world right now. I want to chat about what it's just like to be inside the center of the storm.

   3. KWKevin Weil0:29

      Our general mindset is, in two months there's going to be a better model and it's going to blow away whatever the current set of limitations are. And we say this to developers too. If you're building and the product that you're building is kind of right on the edge of the capabilities of the models, keep going because you're doing something right. Give it another couple months and the models are going to be great. And suddenly the product that you have that just barely worked is really gonna sing.

   4. LRLenny Rachitsky0:51

      Famously, you led this project at Facebook called Libra.

   5. KWKevin Weil0:55

      Libra is probably the biggest disappointment of my career. It fundamentally disappoints me that this doesn't exist in the world today because the world would be a better place if we'd been able to ship that product. We tried to launch a new blockchain. It was a basket of currencies originally. It was integration into WhatsApp and Messenger. I would be able to send you 50 cents in WhatsApp for free. It should exist. To be honest, the current administration is super friendly to crypto. Facebook's reputation is in a very different place. Maybe they should go build it now.

   6. LRLenny Rachitsky1:27

      Today my guest is Kevin Wheel. Kevin is Chief Product Officer at OpenAI, which is maybe the most important and most impactful company in the world right now being at the forefront of AI and AGI and maybe someday super intelligence. He was previously head of product at Instagram and Twitter. He was co-creator of the Libra cryptocurrency at Facebook, which we chat about. He's also on the boards of Planet and Strava and the Black Product Managers Network and the Nature Conservancy. He's also just a really good guy and he has so much wisdom to share. We chat about how OpenAI operates, implications of AI and how we will all work and build product, which markets within the AI ecosystem companies like OpenAI won't likely go after and thus are good places for startups to own, also why learning the craft of writing evals is quickly becoming a core skill for product builders, what skills will matter most in an AI era and what he's teaching his kids to focus on, and so much more. This is a very special episode and I'm so excited to bring it to you. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. If you become an annual subscriber of my newsletter, you get a year free of Perplexity Pro, Linear, Notion, Superhuman and Granola. Check it out at lennysnewsletter.com and click bundle. With that, I bring you Kevin Wheel. This episode is brought to you by Eppo. Eppo is a next generation AB testing and feature management platform built by alums of Airbnb and Snowflake for modern growth teams. Companies like Twitch, Miro, ClickUp and DraftKings rely on Eppo to power their experiments. Experimentation is increasingly essential for driving growth and for understanding the performance of new features. And Eppo helps you increase experimentation velocity while unlocking rigorous deep analysis in a way that no other commercial tool does. When I was at Airbnb, one of the things that I loved most was our experimentation platform where I could set up experiments easily, troubleshoot issues and analyze performance all on my own. Eppo does all that and more with advanced statistical methods that can help you shave weeks off experiment time, an accessible UI for diving deeper into performance and out of the box reporting that helps you avoid annoying prolonged analytic cycles. Eppo also makes it easy for you to share experiment insights with your team, sparking new ideas for the AB testing flywheel. Eppo powers experimentation across every use case including product, growth, machine learning, monetization and email marketing. Check out Eppo at geteppo.com/lenny and 10X your experiment velocity. That's geteppo.com/lenny. This episode is brought to you by Persona, the adaptable identity platform that helps businesses fight fraud, meet compliance requirements and build trust. While you're listening to this right now, how do you know that you're really listening to me, Lenny? These days it's easier than ever for fraudsters to steal PII, faces and identities. That's where Persona comes in. Persona helps leading companies like LinkedIn, Etsy and Twilio securely verify individuals and businesses across the world. What sets Persona apart is its configurability. Every company has different needs depending on its industry, use cases, risk tolerance and user demographics. That's why Persona offers flexible building blocks that allow you to build tailored collection and verification flows that maximize conversion while minimizing risk. Plus, Persona's orchestration tools automate your identity process so that you can fight rapidly shifting fraud and meet new waves of regulation. Whether you're a startup or an enterprise business, Persona has a plan for you. Learn more at withpersona.com/lenny. Again, that's with P-E-R-S-O-N-A dot com slash lenny.

2. 5:16 – 8:13

   OpenAI’s new image model

   1. LRLenny Rachitsky5:17

      Kevin, thank you so much for being here and welcome to the podcast.

   2. KWKevin Weil5:23

      Thank you so much for having me. We've been talking about doing this forever and we made it happen.

   3. LRLenny Rachitsky5:27

      We did it. I can't imagine how insane your life is so I really appreciate you, that you made time for this. And we're actually recording this, uh, the week that you guys launched your new image model which is a happy coincidence. Uh, my entire social feed is filled with skibbily vacations of everyone's life and family photos and everything. So good job.

   4. KWKevin Weil5:44

      Yep, mine too. My wife-

   5. LRLenny Rachitsky5:46

      Okay.

   6. KWKevin Weil5:46

      ... uh, Elizabeth sent me one of hers so I'm, I'm right there with you.

   7. LRLenny Rachitsky5:51

      Uh, let me just ask, did you guys expect this kind of reaction? It feels like this is the most viral thing that's happened in AI which is a high bar, uh, since, I don't know, ChatGBT launched. Just like did you guys expect it to go this well? Uh, what does it feel like internally?

   8. KWKevin Weil6:03

      You know, there have been a handful of times in my career when you're working on a project or product internally... and the internal usage just explodes. Uh, this was true by the way when we were building stories at Instagram. More than anything else in my career, we could feel it was gonna work because we were all using it internally. And we'd go away for a weekend, you know, before it launched, we were all using it and would come back after a weekend and we would know what was going on and be like, "Oh hey, I saw you were at that camping trip. That- how was that?" We were like, "Man, this thing really works." Image Gen was definitely one of those. Uh, so we've been playing with it for, I don't know, couple months. And, um, uh, it- it, when it first went live internally to the company, there was kind of a- a- a- a little gallery where you could generate your own, you could also see what everyone else was generating. And it was just like non-stop buzz. So yeah, we had a sense that this was gonna be a lot of fun for people to play with.

   9. LRLenny Rachitsky6:58

      That's a really cool- like, that should be a measure of just like, uh, confidence in the something going well that you're launching is internally everyone's going crazy for it.

   10. KWKevin Weil7:05

       Yeah, especially social things because-

   11. LRLenny Rachitsky7:08

       Mm. Yeah.

   12. KWKevin Weil7:08

       ... um, it, you have a very tight network as a company socially, so you know each other and you're experts in your product hopefully. And so there's some sense in which if you're doing something social and it's not taking off internally, you- you might- you might question what you're doing.

   13. LRLenny Rachitsky7:23

       Yeah. Uh, and by the way, the Ghibli thing, is that something you guys seeded or how did that even start? Was that like an intentional example?

   14. KWKevin Weil7:29

       I think it's just the style people love.

   15. LRLenny Rachitsky7:31

       Yeah. Wow.

   16. KWKevin Weil7:31

       And the model is- is really capable at- at emulating style or understanding what... You know, it's very good at instruction-following. That's actually something that I think people- I'm starting to see people discover with it. But you can do very complex things. You can give it two images, you know, one is your living room and the other is a whole bunch of photos or memorabilia or things you want. And you say like, "Tell me how you would arrange these things." Or you can say, "I'd like you to show me what this will look like if you put this over here and this thing to the right of that and this one to the left of this, but under that one." And the model actually will understand all of that and do it. It's incredibly powerful. So I'm- I'm- I'm just excited about all the different things people are gonna figure out.

   17. LRLenny Rachitsky8:11

       Yeah. All right. Well good job. Good job team,

3. 8:13 – 11:42

   The role of chief product officer at OpenAI

   1. LRLenny Rachitsky8:13

      OpenAI. Uh, let's get serious here and let's kind of zoom out a little bit. The way I see it is you're chief product officer of maybe the most important company in the world right now, uh, just not to set the bar too high, but you guys are ushering in AI, AGI at some point, super intelligence at some point, no big deal. Uh, I've had- I have more questions for you than I've had for any other guest actually. Put out a call out, uh, on Twitter and LinkedIn and my community just like, "What would you want to ask Kevin?" And, uh, had 300, over 300 well-formed questions and we're gonna go through every single one, so let's just get started. I'm just joking.

   2. KWKevin Weil8:45

      Cool.

   3. LRLenny Rachitsky8:45

      Uh, I picked out the best and there's a lot of stuff I'm really curious about.

   4. KWKevin Weil8:47

      Well it's- it's- it's 01:00 P.M. here, it doesn't get dark for a while, so let's do it.

   5. LRLenny Rachitsky8:51

      (laughs) Okay. Here we go. Okay, so first of all, I'm just gonna take notes here, uh, when is AGI launching? When does this

   6. NANarrator8:57

      (laughs)

   7. KWKevin Weil8:58

      I mean, we just launched a good image gen model. Does that count?

   8. LRLenny Rachitsky9:00

      (laughs) It's, uh, it's getting there. It's getting there.

   9. KWKevin Weil9:04

      The- there's this, um, there's this quote I love which is, "AI is whatever hasn't been done yet."

   10. LRLenny Rachitsky9:11

       Mm.

   11. KWKevin Weil9:11

       Because once it's been done when it kind of works, then you call it machine learning. And once it's kind of ubiquitous and it's everywhere, then it's just an algorithm. Um, so I- I've always loved that- that it- we call things AI when they still don't quite work and then, you know, by the time it's like an AI algorithm that's recommending you follow, you know, oh, that's just an algorithm. But this new thing like self-driving cars, that's a-

   12. LRLenny Rachitsky9:34

       Yeah. (laughs)

   13. KWKevin Weil9:35

       Uh, I think- I think to some degree we're always gonna be there, and the next thing is always gonna be AI. And the current thing that we, you know, use every day and is just a part of our lives, that's an algorithm.

   14. LRLenny Rachitsky9:46

       It's so interesting 'cause yeah, like, uh, in- in the Bay Area you see self-driving cars driving around and it's so normal now. When like th- four years ago and I don't know, three years ago, you would've thought- you would've seen this and you'd be like, "Holy shit. What is... wow, we're in the future," and now we just so take it for granted.

   15. KWKevin Weil10:01

       It's- I mean, there's something like that with everything. If I showed you e- when GPT-3 launched, right? I wasn't at OpenAI then, I was just, uh, I was just a user. But it was mind-blowing. And if I gave you GPT-3 now, I just plug that into ChatGPT for you and you started using it, you'd be like, "What is this thing?" I- like-

   16. LRLenny Rachitsky10:20

       (laughs)

   17. KWKevin Weil10:20

       ... it's like mess, uh...

   18. LRLenny Rachitsky10:22

       Slop. Slop.

   19. KWKevin Weil10:23

       And it- there's- I had the same experience when I- when I first got into a Waymo, right? Your- your very first ride, at least my very first ride, my first like 10 seconds in a Waymo it starts driving and you're like, "Oh my God, watch out for that bike." And you're- you're holding onto whatever you can. And then like five minutes in, you've calmed down and you realize that you're getting driven around the city without a driver and it's working. And you're just like, "Oh my God. I am living in the future right now." And then like another 10 minutes, you're bored, you're doing email on your phone, answering Slack messages and, you know, suddenly this miracle of human invention is just an expected part of your life from then on. And I- and there is really something in the way that we all are adapting to AI that's kind of like that. These miraculous things happen and computers can do something they've never been able to do before. And it blows our mind collectively for like a week, and then we're like, "Oh yeah," like, "oh yeah, now- now it's just machine learning on its way to being an algorithm."

   20. LRLenny Rachitsky11:23

       The craziest thing about what you just shared actually is like, I don't know, ChatGPT which is like now feels terrible, uh, 3.5 was like a couple years ago and, uh, imagine what life will be like in a couple years from now. We're gonna get to that where things are going, what you think is gonna be the next big leap. But I want to start with the beginning of your journey at OpenAI.

4. 11:42 – 15:59

   His recruitment story and joining OpenAI

   1. LRLenny Rachitsky11:43

      Uh, so you worked at Twitter, you worked at Facebook, you worked at Planet, Instagram. Uh, at some point you got recruited to go come work at OpenAI. I'm curious just what that story was like of the recruiting process of joining Opi- OpenAI as CPO. Is there any- are there any fun stories there?

   2. KWKevin Weil12:00

      Uh, if I'm gruning- remembering the timeline right, we communicated, uh, Planet I was leaving, and I was planning to just go take some time. You know, like, I wasn't gonna stop working but, um-But I was also happy to take the summer. This is like maybe April or something. It was like, "Cool, I'm gonna have the summer with my kids. We're gonna, you know, go up to Tahoe or something, and I'll actually get to hang out rather than what I usually do going up and down and all that." Uh, a- and then, you know, Sam and I had known each other lightly for a bunch of years. And he's, he's always involved in so many interesting things, you know, like companies building fusion and, and all these things. So he'd always been somebody that I would, like call occasionally if I was starting to think about my next thing, um, because I like working on big, like tech forward sort of, you know, next, next wave kinda things. And, um, and so, uh, I called him, and I think Vinod also helped, uh, put us in touch again. And, and this time, it wasn't like, "Oh, you should go talk to, like these guys working on fusion." It- he said, "Actually, you know, th- we're thinking about something, you should come talk to us." I was like, "Okay, that sounds amazing, let's do it." And it goes really fast, really, really fast. Like, I met, uh, you know, most of the management team in a brief period of time, a few days, and they were telling me, "Look, we're gonna- we're basically gonna move as fast as we, as we wanna move." And, uh, it kind of, i- if everyo- if you talk to everyone, everyone likes you, we're ready to go. Uh, Sam came over for dinner, uh, and we had, we had a great evening together just like talking about OpenAI and the future and getting to know each other better. And at the end, I was like, I, I and I- I, I was gonna go in the next day for, like a bigger round of interviews. And, uh, um, Sam was saying, you know, "Hey, it's going really well. We're really excited." And I said, "Cool, so how do I think about tomorrow?" And he said, "Oh, you'll be fine. Don't worry about it. And if it goes well, like we're basically there." And so I go in the next day, meet a bunch of people, have a great time, like I really enjoyed everybody I met with. In- in any interview, you can always second guess yourself. You know? Like, "Oh, I shouldn't have said that thing," or, "I- I- that thing I gave a bad answer on, I wish I could redo." But I- I- I came away feeling like, "I think that went pretty well." And ex- I was expecting to hear, like that weekend basically, 'cause they'd sort of set expectations as soon as, y- you know, if this goes well, we're ready to go. And, uh, I didn't hear anything. And then it was like Monday, Tuesday, Wednesday, I still didn't hear anything. And, uh, I reached out to, uh, to folks on the OpenAI side a couple of times, still nothing. And I was like, "Oh my God. I screwed it up. Like, I don't know where I screwed it up, but I totally screwed it up. I can't believe it." And I was going back to Elizabeth, my wife, and being like, "What did I do? Like where, where do you think I..." You know, getting all crazy about it. And, um, and then, uh, still nothing. And finally it was like, it was like nine days later, they finally got back to me. And it turned out, you know, there was like a bunch of stuff happening internally and this, that, and the other thing. And, uh, you know, there's just a million things happening. An- and they finally were like, "Oh yeah, that went well. Let's do this." And I was like, "Oh, okay, cool. Let's do it." Uh, but uh, it was like nine days of agony, and they were just super busy on some internal stuff. And, uh, there I was, like fretting every single day and re- re-going over every line of our interview process.

   3. LRLenny Rachitsky15:33

      It makes me think about when you're like dating someone and you have texted them and then they just- you're not hearing anything back. And all you- like, you assume something is wrong.

   4. KWKevin Weil15:40

      Yeah, totally.

   5. LRLenny Rachitsky15:40

      And they, they might just be busy. Uh, and yeah.

   6. KWKevin Weil15:43

      I, I, I give her a hard time about it still, so. (laughs)

   7. LRLenny Rachitsky15:47

      Uh, that's wild. Uh, I love- I love that it worked out. Uh, and I guess, I guess the lesson there is don't, don't jump to conclusions.

   8. KWKevin Weil15:55

      Yeah. Ha- and have a little bit of chill.

   9. LRLenny Rachitsky15:57

      (laughs)

5. 15:59 – 18:44

   Working at OpenAI

   1. LRLenny Rachitsky15:59

      Speaking of that, I wanna chat about what it's just like to be inside the center of the storm. Again, you worked at, uh, a lot of, let's say, traditional companies, even though they're not that traditional. Twitter and Instagram and Facebook and Planet, and now you work at OpenAI. I'm curious to what is most different about how things work in your day-to-day, uh, life at OpenAI.

   2. KWKevin Weil16:19

      I think it's probably the pace. Uh, maybe it's two things. One is it's the pace. The second is, you know, everywhere I've ever worked before this, you kinda know what technology you're building on. So you spend your time thinking about what, what problems are you solving? Who are you building for? You know, how are you gonna make their lives better? How are you gonna- is this a big enough problem that you're gonna be able to, to change habits? You know, do people care about this problem being solved? All those, like good product things. But the stuff that you're building on is like kinda fixed, you know? You're talking about databases and things, and I bet the database you use this year is probably 5% better than the database you used two years ago. But that's not true at all with AI. It's like every two months, computers can do something they've never been able to do before, and you need to completely think differently about what you're doing. Th- there's like something fundamentally interesting about that, makes life fun here. There's also something, uh, y- you know, we'll maybe like talk about evals later, but it also really, in this world of, um, you know, e- everything we're used to with computers is about giving a computer very defined inputs. You know, if you look at Instagram, for example, there are buttons that do specific things, and you know what they do. And then when you give a computer defined inputs, you get very defined outputs. You're confident that if you do the same thing three times, you're gonna get the same output three times. LLMs are completely different than that, right? They're good at fuzzy, subtle inputs, the- all the nuances of human language and communication, they're pretty good at. And also, they don't really give you the same answer. You, you probably get spiritually the same answer for the same question. But it's certainly not the same set of words every time. And so you're much more, it's fuzzier inputs and fuzzier outputs. And it- when you're building products, it really matters whe- whether, you know, y- there's some use case that you're trying to build around.... if the model gets it right 60% of the time, you build a very different product than if the model gets it right 95% of the time, versus if the model gets it right 99.5% of the time. And so there's also something that you have to get really into the weeds on your use case, on the evals, and things like that, in order to understand the right kind of product to build. So, that is just fundamentally different. You know, if your database works once, it works every time. And that's not

6. 18:44 – 24:40

   The importance of evals in AI

   1. KWKevin Weil18:44

      true in this world.

   2. LRLenny Rachitsky18:45

      Let's actually pull this thread on evals, I definitely wanted to talk about this. So we had this, uh, legendary panel, uh, at the Learning Friends Summit, it was you and Mike Krieger, and Sir Guo, uh-

   3. KWKevin Weil18:56

      Yeah, that was fun.

   4. LRLenny Rachitsky18:56

      ... moderating. So fun. And, uh, the thing that I heard that kind of stuck with people from that panel was a comment you made, where you said that writing evals is gonna become a core skill for product managers.

   5. KWKevin Weil19:08

      Yeah.

   6. LRLenny Rachitsky19:08

      And I feel like that probably applies further than just product managers. A lot of people know what evals are. A lot of people have no idea what I'm talking about. So could you just briefly explain what is an eval, and then just why do you think this is gonna be so important for people building products in the future?

   7. KWKevin Weil19:23

      Yeah, sure. I, I think the easiest way to think about it is almost like a, a quiz for a model, a test to, to gauge how much it, how well it knows a certain set of subject material, or how w- how good it is at responding to a certain set of questions. So in the same way you, you know, you take a calculus class and then you have calculus tests that see if you're, if you've learned what you're supposed to learn, you have evals that test how good is the model at, at creative writing? How good is the model at, uh, at, you know, graduate level science? How good is the model at competitive coding? Uh, and so you have these set of evals that basically, you know, perform as benchmarks for how smart or capable the model is.

   8. LRLenny Rachitsky20:04

      Is like a simple way to think about it, like unit tests for-

   9. KWKevin Weil20:07

      Y- yeah.

   10. LRLenny Rachitsky20:07

       ... model output?

   11. KWKevin Weil20:07

       Unit tests, tests in general for models, totally.

   12. LRLenny Rachitsky20:10

       Great. Great. Okay. And then, uh, why is this so important for people that don't totally understand what the hell's going on here with evals? Why is it so, so key to building AI products?

   13. KWKevin Weil20:20

       Uh, well it gets back to what I was saying, you need to know whether your model is going to... there are certain things that models will get right 99.95% of the time, and you can just be confident. There are things that they're gonna be 95% right on, and things they're gonna be 60% right on. If the model's 60% right on something, you're gonna need to build your product totally differently. And, by the way, these things aren't static either. So a, a big part of evals is if you know y- you're, you're building for some use case. So let's say, let's take our deep research product, which is one of my favorite things that we've released, maybe ever. Um, all right, the idea is with deep research for people who haven't used it, you can give ChatGPT now a, an arbitrarily complex query. Like, it, it's not about returning you an answer from, you know, a search query, which we could also do. It's, it's here's a thing that if you were gonna answer it yourself, you'd go off and do, you know, two hours of reading on the web, and then you might need to read some papers, and then you would come back and start writing up your thoughts and realize you had some gaps in your thinking, so you go out and do more research. And you might, it might take you a week to write some like 20-page answer to this question. You can let ChatGPT just like chug for you for 25, 30 minutes, you know, it's not the immediate answers you're used to, but it might go work for 25, 30 minutes and do work that would have taken you a week. So a- as we were building that product, we were designing evals, uh, e- sort of i- at the same time as we were thinking about how this product was gonna work, and we were trying to go through like hero use cases, you know, he- here's a question you want to be able to ask, here's an amazing answer for that question. And, and then turning those into evals, and, and then hill climbing on those evals. So it's not just that the model is static and we hope it does okay on a certain set of things. You can teach the model. You can make this a continuous learning process. And so as we were fine-tuning our model for deep research to, to be able to answer these things, we were able to test, is it getting better on these evals that we said were important measures of how the product was working? And it's when you start seeing that and you start seeing performance on evals going up, you start saying, "Okay, I think we have a product here."

   14. LRLenny Rachitsky22:35

       You made a kind of a comment along these same lines around evals that, uh, that AI is almost like capped in how amazing it can be by the- how good we are at evals. Does that resonate? Any more thoughts along those lines?

   15. KWKevin Weil22:48

       These, I mean, these models are, are, they're intelligences. In- intelligence is so fundamentally multi-dimensional. So you can talk about a model being amazing at competitive coding, which may not be the same as that model being great at front end coding, or back end coding, or taking a whole bunch of code that's written in Cobalt and turning it into Python, you know? Like, and that's just within the software engineering world. And so, I, I think there's a sense in which you can think of these models as incredibly smart, very like factually aware, uh, intelligences. But still most of the world's data, knowledge, process, is, is not public. Uh, it's behind the walls of companies or governments or other things. And same way if you were gonna join a company, you would spend your first two weeks onboarding, you'd be learning the company-specific processes, you'd get access to company-specific data. I- i- it's, you can teach these mo- the models are smart enough, you can teach them anything. But they need to have the, the sort of the raw data, uh, to, to learn from. And so there's a, there's a sense in which, um, yeah, I think the future is really gonna be incredibly smart, broad base models that are fine-tuned and, and, and, um, tailored with company-specific or use case specific data, so that they perform really well on company-specific or use case specific things. Um, and-You're gonna measure that with custom evals. And so, you know, w- what I, what I was referring to is just, like, these models are really smart. You need to still teach them things if the data's not in their training set. And there's a huge amount of use cases that are not gonna be in their training set, because they're relevant to one industry or one company.

7. 24:40 – 26:34

   Opportunities in the space

   1. LRLenny Rachitsky24:40

      I'm just gonna keep following the thread that you're, uh, leading us down, and... But I'm gonna come back, 'cause I have more questions around some of these things. So, you ca- you came to, uh, a, a space that I think a lot of AI founders are thinking about, is just, where is OpenAI not gonna come squash me in the future? Or one of the other foundational models. And so it's unclear to a lot of people, just like, "Should I build a startup in this space or not?" Is there any advice you have or any guidance for where you think OpenAI, or just foundational models in general, likely won't go, and where you have an opportunity to build a company?

   2. KWKevin Weil25:09

      Well, uh, one of my... So, th- this is something that Ev Williams used to say, um, back at Twitter, that's always stuck with me. Which is, no matter, no matter how big your company gets, no matter how, like, incredible the people are, there are way more smart people outside your walls than there are inside your walls. And that's why we are so focused on building a great API. We have three million developers using our API. Uh, no matter how ambitious we are, how big we grow... By the way, we don't wanna grow super big. It, uh, there are going to be... There, there are so many use cases, places in the world where AI can fundamentally make our lives better. We're not gonna have the people. We're not gonna have the, the, you know, the, the knowhow to build most of these things. And I think, like I was saying, the data is, is industry specific, use case specific, you know, behind certain company walls, things like that. And there are immense opportunities in every industry and every vertical in the world to go build AI-based products that improve upon the, uh, the state of the art. And there's just no way we could ever do that ourselves. We don't want to. We couldn't if we did want to. And we're really excited to power that for three million-plus developers and way more in the future.

   3. LRLenny Rachitsky26:24

      Coming back to your earlier point about the, the, the tech changing constantly and getting faster, not exactly knowing what you'll have by the time you launch something, in terms of the power

8. 26:34 – 29:47

   Shipping quickly and consistently

   1. LRLenny Rachitsky26:34

      of the, uh, the model. Uh, I was... I'm curious what allows you to ship quickly and consistently, and such great stuff. And it sounds like one answer is bottoms-up, empowered teams, versus the very top-down roadmap that's, you know, planned out for a quarter. What, what are some of those things that allow you to ship such great stuff so often, so quickly?

   2. KWKevin Weil26:53

      Yeah. I mean, we try and, we try and have, uh, a, a sense of where we're trying to go. You know, point ourselves in a direction so that we have some rough sense of alignment. Um, like, thematically. Uh, I don't for a second... And we do quarterly roadmapping. You know, we, we laid out sort of a year-long strategy. I don't for a second believe that what we write down in these documents is what we're gonna actually ship, you know, three months from now, let alone six or nine. But that's okay. There's a, um... I think it's, like, an Eisenhower quote. "Plans are useless, planning is helpful." Uh, which I totally subscribe to, especially in this world. It's really valuable, if you think about quarterly roadmapping, for example. It's really valuable to have a moment where you stop and go, "Okay, what did we do? What worked? What went well? What didn't go well? What did we learn? And now, what do we think we're gonna do next?" And, by the way, everybody has some dependencies. You, you know, you need the infrastructure team to do the following things, partnership with research here. And so you wanna have a second to kind of check your dependencies, make sure you're good to go, and then start executing. We try and keep that really lightweight, because it's not gonna be right. You know? (laughs) We're gonna throw it out halfway, because we will have learned new things. So the moment of planning is helpful, even if you're only gonna... You know, it's only partially right. So that's p-... I think b- just expecting that you're gonna be super agile, and that there's no sense writing a three-month roadmap, let alone a yearlong roadmap, because the technology's changing underneath you so quickly. We really do try and go, like, very strongly bottoms up, kind of subject to our overall directional alignment. Uh, we have great people. Um, we have engineers and PMs and designers and researchers who are passionate about the products they're building and have strong opinions about them, uh, and are also the ones building them. And so they're, they have a, they have a real sense of what the capabilities are too, which is super important. And so I think you want to be more bottoms up in, in this way. And so we operate that way. We are happy making mistakes. We make mistakes all the time. It's one of the things I really appreciate about Sam. He pushes us really hard to move fast, but he also understands that with moving fast comes, "Uh, we didn't quite get this right," or, you know, "That we launched this thing, it didn't work, we'll roll it back." You know, look at our naming. Our naming is horrible. You know?

   3. LRLenny Rachitsky29:14

      There was a lot of questions people had for you. (laughs)

   4. KWKevin Weil29:17

      Yeah.

   5. LRLenny Rachitsky29:17

      On model names, yeah.

   6. KWKevin Weil29:18

      It, it, it's absolutely atrocious, and we know it. Um, and we'll, we'll get around to fixing it at some point. But it's not the most important thing, and so we don't spend a lot of time on it.

   7. LRLenny Rachitsky29:27

      But it also shows you how it doesn't matter. Uh, again, ChatGPT, the most popular, fastest-growing product in history. Uh, models are... M- it's the number one AI, API model, so clearly it doesn't matter that much.

   8. KWKevin Weil29:39

      And, and we name things like O3 Mini High.

   9. LRLenny Rachitsky29:42

      (laughs)

   10. KWKevin Weil29:42

       And... (laughs)

   11. LRLenny Rachitsky29:45

       Oh, man. I love it.

9. 29:47 – 32:53

   Product reviews and iterative deployment

   1. LRLenny Rachitsky29:47

      Um, okay. So you talked about roadmapping, um, and bottoms up. And I'm really curious how you... K- is there, like, a, a cadence or a ritual of aligning with you or Sam or he? Or you review everything that's going out? Like, is there a meeting every week or every month where you guys see what's happening?

   2. KWKevin Weil30:03

      On key projects. So we do product reviews and things like that, like you would expect. Um, e- there isn't a ritual because there isn't a... We, we, I, I would never want us to be blocked on launching something, you know, waiting for a review with me or Sam. If we can't get there, if I'm traveling or Sam's, you know, busy or whatever, that's a bad reason for us not to ship.So obviously, for the biggest, most high priority stuff, we have a pretty close beat on it. But we really try not to, frankly. Um, like, we want to empower teams to move quickly. And, uh, I think it's more important to ship and iterate. So we have this philosophy that we call iterative deployment. And the idea is, like, we're all learning about these models together. So there's a real sense in which it's way better to, like, ship something even when you don't know the full set of capabilities and iterate together, like, in public, and we- we kind of co-evolve together with the rest of society as we learn about these things and where they're different and where they're good and bad and weird. I really like that philosophy. Um, there's also a bit of... I- I think the other thing that- that, like, ends up being a part of our- our product philosophy is, uh, this sense of, like, model maximalism. The models are not perfect, they're gonna make mistakes. You could spend a lot of time building all kinds of different scaffolding around them. And by the way, sometimes we do because sometimes there are things, y- you know, kinds of errors that you just don't want to make. But we don't spend that much time building scaffolding around the parts that don't match that because our general mindset is, in two months there's gonna be a better model and it's gonna blow away whatever, you know, the current set of limitations are. And so if- if you're building... And we say this to developers too. If you're building and- and the product that you're building is kind of right on the edge of the capabilities of the models, keep going 'cause you're doing something right. Because you give it another couple months and the models are gonna be great and suddenly, the- the product that you have that just barely worked is really gonna sing and, uh, you know, that's- that's kind of how you make sure that you're really pushing the envelope in building new things.

   3. LRLenny Rachitsky32:18

      I had, uh, the founder of Bolt on the podcast. Uh, StackBlitz is the company name. And he- he shared this story that they've been working on this product for seven years behind the scenes and it was failing. No- nothing was happening. And then all of a sudden, uh... It was, sorry to mention a competitor, but Claude, uh, came out. Or Sonnet 3.5 came out. And a- all of a sudden, everything worked. And they've been building all this time and finally, it worked. And I hear that a lot with YC, just like things are... That never were possible now are just becoming possible every few months with the updates in the models.

   4. KWKevin Weil32:48

      Yeah, absolutely.

   5. LRLenny Rachitsky32:50

      Let me actually ask this. I wasn't planning to ask this, but I'm curious if you have any

10. 32:53 – 36:03

    Winning consumer awareness

    1. LRLenny Rachitsky32:53

       quick thoughts. Just why- why is, uh, Sonnet so good at coding and kind of thoughts on, uh, your stuff getting as good and better at actual coding?

    2. KWKevin Weil33:01

       Yeah. Uh, I mean, kudos to Anthropic. They've built very good coding models. Uh, no doubt. We, uh, we- we think that we can do the same. Um, maybe by the time this, uh, podcast is shipped, we'll- we'll have more to say. But either way, uh, all credit to them. I think, uh, mo- this intel- intelligence is really multi-dimensional. And so I think there's... The- the- the model providers, it used to be that OpenAI had this, like, massive model lead. You know, 12 months or something ahead of everybody else. That's not true anymore. You know, I like to think we still have a lead. I'd argue that we do. But it's certainly not a massive one. And that means that there are gonna be different places where, you know, the Google models are really good or where Anthropic's models are really good or where we're really good and- and our competitors are like, "Ah, we gotta get better at that." And it actually is easier to get better at a certain thing once someone's proved it possible than it is to, you know, forge a path through the- the jungle in doing something brand new. So I just think, yeah, as an example, it was like nobody- nobody could break four minutes in the mile and then finally somebody did and the next year 12 more people did it. I- I think there's that all over the place. And it just means that competition is really intense and consumers are gonna win and developers are gonna win and businesses are gonna win in a big way from that. It's part of why the industry moves so fast. But, um, you know, all respect to- to the other big model providers, models are getting really good. We're gonna move as fast as we can and I think we've got some good stuff coming.

    3. LRLenny Rachitsky34:35

       Exciting. Uh, this makes me also think about, uh, in many ways other models are better at certain things but somehow ChatGPT is like the... Like, if you look at all the awareness numbers and usage numbers, it's like no matter where you guys are in the rankings, people seem to just, like, think of AI and ChatGPT almost as- as the same. What do you think you did right to kind of win in the consumer mindset, at least at this point, in awareness in the world?

    4. KWKevin Weil35:01

       I think being first helps, which is one of the reasons why we're so focused on moving quickly. Um, you know, we like being the first to launch new capabilities, things like deep research. Uh, we've also... Our models are very... They can do a lot of things, right? So they can... They can take real-time video input, they can... You have speech to speech, you can do speech to text and text to speech. Um, they can do deep research, they can operate on a canvas, they can write code. And so ChatGPT can kind of be this one-stop shop where all the things that you want to do are possible. Um, and as we... as we go forward and it... You know, we have more agentic tools like Operator where it's browsing for you and doing things for you on the web. You know, m- more and more, you're gonna be able to come to this one place to ChatGPT, give it instructions and have it accomplish real things for you in the world. There's just, like, something fundamentally valuable in that. And so, you know, we- we think a lot about that. We think... And it- it... We- we move... We try to move really fast so that we are always the most useful place for people to come to.

11. 36:03 – 40:56

    Designing thoughtful experiences

    1. KWKevin Weil36:03

    2. LRLenny Rachitsky36:04

       What would you say is, uh, the most counterintuitive thing that you've learned after building AI products or working at OpenAI? Something that's just like, "I did not expect that."

    3. KWKevin Weil36:14

       I don't know. Maybe I should have expected this, but one of the things that's been funny for me is, um, the extent to which you can kind of reason... When you're trying to figure out how some product should work with AI-You can often, or even why some AI thing happens to be true, you can often reason about it the way you would reason about another human. And it kinda works. Yeah, so, uh, maybe a couple of examples. When we were first launching our, um, our reasoning model, right? We were the first to, to build a, a model that could reason, that could, that could... Instead of giving you just a quick, you know, system one answer right away to every question you asked, who was the third emperor of the Holy Roman Empire, like, you know, here's an answer, you could ask it hard questions and it would reason the same way that if I asked you to do a crossword puzzle, you couldn't just like snap fill in everything. You would be, "Well, okay, um, this one across, I think it could be one of these two, but that means there's an A here, so that one has to be this. Oh, wait..." You know, and like backtrack, kind of step by step build up from where you are, same way you answer any, any difficult, uh, logistical problem, any scientific problem. So this reasoning breakthrough was big, but it was also the first time that a model needed to sit and think. And that's a weird paradigm for a consumer product. You don't normally have something where you might need to hang out for 25 seconds after you ask a question. And, and so we were trying to figure out, you know, what's the UI for this? Because it's also not... Like, with deep research, where the model is gonna go and think for 25 minutes sometimes, it's actually not that hard because you're not gonna sit and watch it for 25 min- you're gonna go do something else. You're gonna go to another tab or go get lunch or whatever, uh, and then you'll come back and it's done. When it's like 20, 25 seconds or 10 seconds, it's a long experi- it's a long time to wait, but it's not long enough to go do something else. And so you actually need s- and, you know, so you could, you could think, like, w- if you asked me something that I needed to think for 20 seconds to answer, what would I do? I, I wouldn't just, like, go mute and not say anything and kind of, um, you know, shut down for 20 seconds and then come back. So we shouldn't do that. We shouldn't just, like, have a slider sitting there. That's annoying. But I also wouldn't just start, like, babbling every single thought that I had. Um, so we probably shouldn't just, like, expose the whole chain of thought as the model's thinking. But, you know, I might go like, "Huh, that's a good question. All right, I might approach it like that and then think..." And, you know, you're sort of like maybe giving little updates, and that's actually en- what we ended up shipping. Y- you have similar things where you can, like, you can find situations where, um, m- you get better thinking sometimes out of a group of models, uh, that all trying to attack the same problem, and then you have a model that's looking at all their outputs and integrating it and then giving you a single answer at the end. I mean, sounds a little bit like brainstorming, right? I, like, I certainly have better ideas when I get in a room and brainstorm with other people 'cause they think differently than me and... So, anyways, there's just like all these situations where you can actually kind of reason about it, like a group of humans or an individual human and it sort of works, which I don't know, maybe, maybe I shouldn't have been surprised, but I was.

    4. LRLenny Rachitsky39:27

       That is so interesting because when I see these models operate, I, like, I never even thought about you guys designing that, uh, experience. Like, to me, it just feels like this is what the LLM does. It just sits there and tells me what it's thinking. And I love this point you're making of, like, we, like, let's make it feel like a human operating, and how does a human operate? Well, they just talk out loud. They think, "Here's something I should explore." And I love that DeepSeek went, like, to the extreme of that, right? Where they're just like, "Here's everything I'm doing and thinking and I..." And people actually like that too. I guess, was that, was that surprising to you? Like, oh, maybe that could work too. People seem to like everything.

    5. KWKevin Weil40:02

       Yeah, we learned from that actually.

    6. LRLenny Rachitsky40:03

       Hmm.

    7. KWKevin Weil40:04

       Um, because we, um, w- when we first launched it, we kind of gave you, like, the, the subheadings of what the model was thinking about but not much more. And then DeepSeek launched and there were, it was a lot. And we kind of went, "You know, I don't know if everyone wants, like, that." There's some novelty effect to seeing what the model's really thinking about. We felt that too when we were looking at it internally. It's interesting to see the model's chain of thought. But it's not... I, I, you know, I think at the scale of like 400 million people, you don't want to see the model kind of like babble a bunch of things. Um, and so what we ended up doing was summarizing it in interesting ways. So instead of just getting the subheadings, you're kind of getting like one or two sentences about how it's thinking about it, and you can learn from that. So we kind of tried to find a middle ground that, that we thought was an experience that would be meaningful for most people, but, you know, showing everybody like three paragraphs, uh, is probably not

12. 40:56 – 45:21

    Chat as an interface for AI

    1. KWKevin Weil40:56

       the right answer.

    2. LRLenny Rachitsky40:57

       This reminds me of something else you said at the summit that has really stuck with me, this idea that chat, people always make fun of, like, chat is not like the future interface for how we interact with AI, but you made this really interesting point that may argue the other side, which is like as humans we interface by talking, and the IQ of a human can span from really low to really high and it all works 'cause we're talking to them, and chat is the same thing and it can work on all kinds of intelligence levels. Uh, maybe just share, maybe I just shared it, but, uh, I guess anything there about just why chat actually ends up being such an interesting interface for LLMs.

    3. KWKevin Weil41:29

       Yeah, I don't know if, uh, maybe I'm, uh, maybe this is one of those things I believe that most people don't believe, but I actually think chat is an amazing interface because it's so versatile. Um, i- u- people tend to go, "Oh, chat." Yeah, well, that's just like, you know, we'll figure out something better and I kind of think, I kind of think this is, uh, it, it's a, it's, it's incredibly universal because it is the way we talk. Like, I can talk to you verbally like we're talking now. I can, you know, we can see each other and interact. Uh, we can talk on WhatsApp and, you know, be texting each other. But all of these things is this sort of like unstructured, uh, you know, method of communication, and that's how we operate. If I had to... And if I had some more rigid interface that I was allowed to use when we spoke, I would be able to speak to you about, you know, far fewer things and it would actually get in the way of us having, like, maximum communication bandwidth. So there's something magical. And, and by the way, in the past it never worked because models, there, there wasn't a model that was good at understanding,... all of the complexity and nuances of human speech. And that's the magic of LLMs. So to me, it's like a, an interface that's exactly fit to the power of these things. And that doesn't mean that it always has to be just, like... I don't necessarily always want to type, but if you, you do want that very open-ended, flexible communication medium. It may be that we're speaking and the model's speaking back to me, but you still want that, like, that, that very sort of lowest common denominator, um, no restrictions way of, of interacting.

    4. LRLenny Rachitsky43:04

       That is so interesting. That's really changed the way I think about this stuff, is that point, that chat is just so good for this very specific problem of talking to superintelligence, basically.

    5. KWKevin Weil43:13

       By the way, I think there are like... It, it's not that it's only chat either. Like, there are... I- if you have high volume use cases where it, they're more prescribed and the... Y- you don't actually need the full generality, there are, there are many use cases where it's better to have something that's less flexible, more prescribed, faster at a specific task. And those are great too. And, you know, you can build all sorts of those and, um... But you still want chat as, like, this baseline for anything that falls out of-

    6. LRLenny Rachitsky43:43

       (laughs)

    7. KWKevin Weil43:43

       ... whatever, you know, vertical you happen to be building for. It's like a catchall for, like, every possible thing you'd ever want to express to a model.

    8. LRLenny Rachitsky43:50

       I'm excited to chat with Christina Gilbert, the founder of OneSchema, one of our longtime podcast sponsors. Hi, Christina.

    9. NANarrator43:57

       Yes, thank you for having me on, Lenny.

    10. LRLenny Rachitsky43:59

        What is the latest with OneSchema? I know you now work with some of my favorite companies, like Ramp, Vanta, Scale, and Watershed. I heard that you just launched a new product to help product teams import CSVs from especially tricky systems like ERPs.

    11. NANarrator44:15

        Yes, so we just launched OneSchema File Feeds, which allows you to build an integration with any system in 15 minutes, as long as you can export a CSV to an SFTP folder. We see our customers all the time getting stuck with hacks and workarounds. And the product teams that we work with don't have to turn down prospects because their systems are too hard to integrate with. We allow our customers to offer thousands of integrations without involving their engineering team at all.

    12. LRLenny Rachitsky44:37

        I can tell you that if my team had to build integrations like this, how nice would it be to be able to take this off my roadmap and instead use something like OneSchema, and not just to build it, but also to maintain it forever?

    13. NANarrator44:48

        Absolutely, Lenny. We've heard so many horror stories of multi-day outages from even just a handful of bad records. We are laser-focused on integration reliability to help teams end all of those distractions that come up with integrations. We have a built-in validation layer that stops any bad data from entering your system, and OneSchema will notify your team immediately of any data that looks incorrect.

    14. LRLenny Rachitsky45:08

        I know that importing incorrect data can cause all kinds of pain for your customers and quickly lose their trust. Christina, thank you for joining us. And if you want to learn more, head on over to OneSchema.co. That's OneSchema.co.

13. 45:21 – 48:05

    Collaboration between researchers and product teams

    1. LRLenny Rachitsky45:23

       I'm gonna come back to... That you talked about researchers and the relationship with product teams. Uh, I imagine a lot of innovation comes from researchers, just like having an inkling and then building something amazing and then releasing it. And some ideas come from PMs and engineers. How do, how do those teams collaborate? Like, does every team have a PM? Is it a lot of research-led stuff? Just, like, what... Give us a sense of just where ideas and products come from, mostly.

    2. KWKevin Weil45:49

       It's an area where we're evolving a lot. I'm really excited about it, frankly. I, I think if you go back, you know, couple years when ChatGPT was just getting started... Uh, obviously I wasn't at OpenAI, so, um... But, uh, it... We were more, we were more of a pure research company at the time. ChatGPT, if you remember, was a low-key research preview.

    3. LRLenny Rachitsky46:12

       (laughs)

    4. KWKevin Weil46:13

       Um, it wasn't-

    5. LRLenny Rachitsky46:13

       For many years.

    6. KWKevin Weil46:14

       Yeah. It, it wasn't a thing that the team launched thinking it was gonna be this massive product.

    7. LRLenny Rachitsky46:19

       Oh, ChatGPT. Yeah.

    8. KWKevin Weil46:21

       And it, it was just, uh, a way that we were gonna let people, you know, play with and iterate on the models. Um, and so we were, we were primarily a research company, a world-class research company. And as ChatGPT has grown and as we've built our B2B products and our APIs and other things, it... Now we're more of a product company than we were. I still think we can't... We're... OpenAI should never be a pure product company. We need to be both a world-class research company and a world-class product company. And the two need to really work together. And that's the thing that's, um, that I think we've been getting much better at over the last, like, six months. If you, if you treat those things separately and, you know, the researchers go do amazing things and build models, and then they get to some state, and then the product and engineering teams go take them and do something with them, we're effectively just an API consumer of our own models. The best products, though, are gonna be... It was like I was talking about with deep research. It's a lot of iterative feedback. It's understanding the products you're trying to solve or the, the problems you're trying to solve, building evals for them, using those evals to go gather data and fine-tune models to get them to be better at the- these use cases that you're looking to solve. It's a huge amount of back and forth, uh, to do it well. And I think the best products are gonna be eng product design and research working together as a single team to, to build novel things. So that's, that's actually how we're trying to operate with basically anything that we build. It's a new muscle for us because we're kind of new as a product company. But, um, but i- i- it's one that people are really excited about, because we've seen every time we do it, we build something awesome. And so, you know, now every

14. 48:05 – 53:06

    Hiring product managers at OpenAI

    1. KWKevin Weil48:05

       product starts like that.

    2. LRLenny Rachitsky48:07

       How many product managers do you have at OpenAI? I don't know if you share that number, but if you do...

    3. KWKevin Weil48:11

       Not that many, actually. I don't know. 25? Um, maybe it's a little more than that. But, uh, my personal belief is that you want to be pretty PM-light as an organization, just in general. I say this with love because I am a PM, but too many PMs causes problems. You know, we'll, like, fill the world with decks and ideas versus...

    4. LRLenny Rachitsky48:33

       Mm-hmm.

    5. KWKevin Weil48:33

       ... execution. So I think that the, the... I, I think it's a good thing when you have...... a PM that has, uh, that, that is working with maybe slightly too many engineers because it means they're, they're not gonna get in and micromanage. You're gonna leave a lot of, of, you know, influence and responsibility with the engineers to make decisions. It means you wanna have really product-focused engineers, which we're fortunate to have. We have an amazingly product-focused, like, high-agency engineering team. But when you have something like that, you have a team that feels super empowered. You have a, a PM that's, you know, trying to really understand the problems and kinda gently guide the team a little bit but has too much going on to get too far into the details, and you end up being able to move really fast. So that's kind of the philosophy we take. Uh, we want, we want producty eng leads and, and producty engineers all the way through. Um, we want not too many PMs but really awesome high-quality ones. Um, and so far, that seems to be working pretty well.

    6. LRLenny Rachitsky49:36

       I imagine being a PM at OpenAI is like a dream come true for a lot of people. Uh, at the same time, I imagine it's not a fit for a lot of people. There's researchers involved, very product-minded engineers. What do you, what do you look for in the PMs that you hire there for folks that are like, "Maybe a pro- I shouldn't go work there. I shouldn't even think about that."

    7. KWKevin Weil49:54

       I think, uh, I, I've said this a few times, but, like, high agency is something that we really look for, people that are not gonna come in and kinda wait for everyone else to allow them to do something. They're just gonna see a problem and go do it. Um, that's, uh, uh, it's just a core part of how we work. I think people that, that are happy with ambiguity because there is a massive amount of ambiguity here. This is not the kind of place... Uh, and we have, we have trouble sometimes with, um, with more junior PMs because of this, because it's just not the place where s- someone is gonna come in and say, "Okay, you know, here's, here's the landscape. Here is your area. I want you to go do this thing." A- and that's, that's what you want as a, as an early career PM. We just... I mean, no one here has time, and the, nobody... uh, the problems are too ill-formed, and we're figuring them all out as we go. And so, um, high agency, very comfortable with ambiguity, ready to come in and help execute and move really quickly. That- that's kind of our, our recipe. And I think also, happy leading through influence because, uh, I mean, as usual, as a PM, people don't report to you. Uh, your team doesn't report to you, et cetera. But you also have the, the, the complexity of a research function, which is even more sort of self-directed, and y- it's really important to build a good rapport with the research team. Uh, and so, uh, you know, that... I, I think the EQ side of things is also super important for us.

    8. LRLenny Rachitsky51:24

       I know in most companies, a PM comes in, and they're just like, "Why do we need you?" And as a PM, you have to, uh, earn trust and help people see the value. And I feel like at OpenAI, it's probably a very extreme version of that, where they're like, "Why do we need this person? We have researchers, engineers. What are you gonna do here?"

    9. KWKevin Weil51:40

       Yeah, I think people appreciate it done right. Um, but you got... you bring people along. I, I think one of the most important things a PM can do well is be decisive. So it's, it's... there's a real fine line. You don't wanna be making... Uh, I mean, it's kind of like, uh... I, I don't love the PM is the CEO of the product, uh, illusion all the time. But, but just like Sam in his role would be making mistakes if he made every single decision in every meeting that he was in, and he would also be making mistakes if he made no decisions in any meetings that he was in, right? It's a... it's the... it's understanding when to defer to your team and to, like, let, let people i- innovate. And when there is, like, a decision to be made that people either don't feel comfortable with or don't feel empowered to make or a decision that, that, you know, has too many different, like, disparate pros and cons that are spread out across a big group and someone needs to be decisive and make a call, that's a really important trait of a CEO. It's something Sam does well. And it's, it's also a really important trait of a PM kind of at a, at a more microscopic level. And so y- y-... because there's so much ambiguity, it's not obvious what the answer is in a lot of cases. And so having a PM that can come in and, like... And by the way, this doesn't need to be a PM. I'm perfectly happy if it's anybody else. But I kinda looked at the PM to say, like, "If there's ambiguity and no one's making a call, you better make sure that we get a call made and we move forward."

15. 53:06 – 1:04:34

    How OpenAI uses AI: vibe coding, AI prototyping, and more

    1. KWKevin Weil53:07

    2. LRLenny Rachitsky53:07

       This touches on, uh, a few posts I've done of just where is AI gonna take over work that we do versus help us with various work. So let me come at this question from a d- few d- different direction of just how AI impacts product teams and hiring, things like that. So first of all, there's all this talk of, uh, LMs doing our coding for us, and 90% of code is gonna be written by AI in a year. Dario at Anthropic said that. At the same time, you guys are all hiring engineers like crazy, PMs like crazy. You know, every dis- function is dead, but you're still hiring every single one. (laughs) Uh, I guess just, first of all, let me just ask this. How do, how do you and the team, like say engineers, PMs, use AI in your work? Is there anything that's, like, really interesting or things that you think people are sleeping on in, in how you use AI in your day-to-day work?

    3. KWKevin Weil53:52

       We use it a lot. I mean, every one of us is in ChatGPT all the time, i- uh, summarizing docs, using it to help write docs with GPTs that, you know, write product specs and things like that, all, all the stuff that you would imagine. Uh, I mean, talk about writing evals, like, y- you can actually use models to help you write evals, and they're pretty good at it. That all said, I still don't... I'm still sort of disappointed by, by us and just by... I really mean me, um, i- in... If I were to, if, if I were to just, like, teleport my five-year-old self leading product at some other company into my day job, I would recognize it still. And I think we should be in a world, certainly a year from now, probably even more now that, um, where I almost wouldn't recognize it because the workflows are so different and I'm using AI so heavily, and I'd still recognize it today. So I think, in some sense, I'm not doing a good enough job of that. You know, just to give an example, like-Why shouldn't we be, like, vibe coding, uh, demos right, left, and center? Like, um, yeah, instead of showing stuff in, like, Figma, we should be showing prototypes that people are vibe coding, you know, over the course of 30 minutes to illustrate proofs of concept and to explore ideas. That's totally possible today, and we're not doing it enough. Our... Actually, our Chief People Officer, Julia, was telling me the other day she vibe coded an internal tool that she had at a previous job that she really wanted to have here at OpenAI, and she opened, I don't know, Windsurf or something and vibe coded it. Like, how cool is that? And if our chief people officer is doing it, we have no excuse to not be doing it more.

    4. LRLenny Rachitsky55:34

       (laughs) That's an awesome story. Okay. And some people may not have heard this term, vibe coding. Can you describe what that means?

    5. KWKevin Weil55:40

       Yeah. Uh, I think this was, uh, I think this was Andrei's, uh term.

    6. LRLenny Rachitsky55:45

       Karpathy, yeah.

    7. KWKevin Weil55:46

       Uh, Andrei Karpathy, yeah. Um, where it's just... So you have these tools like Cursor and Windsurf that, and GitHub Copilot, that are very good at suggesting, uh, what code you might wanna write. So you can give them a prompt and it'll write code. And then as you go to edit it, it's suggesting what you might wanna do. And the, the, the way that, that everyone started using that stuff was give it a prompt, have it do stuff, you go edit it, give it a prompt, you know, and you're kind of, like, really going back and forth with the model the whole time. As the models are getting better and as people are getting more used to it, you can kind of just, like, uh, let go of the wheel a little bit, and when the model's suggesting stuff, it's just like tab, tab, tab, tab, tab. Like, keep going, yes, yes, yes, yes, yes. And of course the model makes mistakes or it does something that doesn't compile, but when it doesn't compile, you paste the error in and you say, "Go, go, go, go, go." And then you, you test it out, and it, like, does one thing that you don't want it to do. So you enter in an instruction and say, "Go, go, go, go, go." And you just kind of, like, let the model do its thing. And it's not that you would do that for production code that needed to be super, uh, tight today yet. But for so many things, you're trying to get to a proof of concept, you're getting to a, a demo, and you can really take your hands off the wheel and the model will do an amazing job. And that's what... That's, that's vibe coding.

    8. LRLenny Rachitsky57:05

       That's an awesome explanation. I think, like, the pro version of that, which is I think the way Andrei even described it, is you talk. You do like a... There's a step, like Whisper or Super Whisper, something like that, where you're, like, talking to the model, not-

    9. KWKevin Weil57:16

       Yeah.

    10. LRLenny Rachitsky57:16

        ... just, not even typing.

    11. KWKevin Weil57:17

        Yeah, totally.

    12. LRLenny Rachitsky57:19

        Oh, man. So let me, let me just ask, I guess, when you look at product teams in the future, you talked about how you guys should be doing this more instead of designs, having prototypes. What do you think might be the biggest changes in how product teams, uh, are structured or built? Where do you think things are going in the next few years?

    13. KWKevin Weil57:36

        I think you're definitely gonna live in a world where you have more, um... Where you have researchers built into every product team. And I don't even mean just at, at, like, foundation model companies, because I, I think the future... Actually, frankly, one thing that I'm sort of surprised about, uh, about our industry in general is that there's not a greater use of fine-tuned models. Uh, like, a lot of people... You know, the- these models are very good, so our API does a lot of things really well. But when you have particular use cases, you can always make the, the model perform better on a particular use case by fine-tuning it. It's probably just a matter of time. You know, l- folks aren't, like, quite comfortable yet with doing that in every case. But to me, there's no question that that's the future. Every... You know, models are gonna be everywhere, just like transistors are everywhere. AI is gonna be just a part of the fabric of everything we do. But I think there are gonna be a lot of fine-tuned models, because why would you not want, uh, to, to more specifically customize a model against a particular use case? And so I think you're gonna want sort of quasi researcher, uh, machine learning engineer types a- as part of pretty much every team, because fine-tuning a model is just gonna be part of the core workflow for building most products. So that's, that's one change that maybe, you know, you're starting to see at foundation model companies that will propagate out to more teams over time.

    14. LRLenny Rachitsky58:57

        I'm curious if there's a concrete example that makes that real. And I'll share one that comes to mind as you talk-

    15. KWKevin Weil59:03

        Sure.

    16. LRLenny Rachitsky59:03

        ... which is when you look at Cursor and Windsurf l- something I learned from those founders, is that they, they use, like, a sonnet, but then they also have a bunch of custom models that help along the edges that make the specific experience that's not just generating code even better, like autocomplete and looking ahead to where things are going. So may- is that one or any, any other examples of what you w- what, what, what is a fine-tuned model that you're

    17. NANarrator59:26

        Yeah.

    18. LRLenny Rachitsky59:26

        ... that you... Do you think teams will be building with these researchers on their teams?

    19. KWKevin Weil59:29

        Yeah. I mean, so when you're fine-tuning a model, one of the... You're, you're basically giving the model, uh, a bunch of, of examples of the kinds of things you want it to be better at. So it's, it's here's a problem, here's a good answer. Here's a problem, here's a good answer. Uh, or here's a question, here's a good answer, you know, times 1,000 or, or 10,000. Uh, and suddenly you're, you're teaching the model to be much better than, than it was out of the gate at that particular thing. We use it everywhere internally. Um, we also... We, we use ensembles of models much more internally than people might think. Um, so it's not here is... I, I have 10 different problems. I'll just ask, you know, baseline GPT-4o about a bunch of these things. If we have 10 different problems, we might, we might solve them using, uh, you know, 20 different model calls. Some of which are using specialized fine-tuned models. They're using models of different sizes 'cause maybe you have different latency requirements or cost requirements at different... For different questions. They are probably using custom prompts for each one. Like, basically, you want the... To teach the model to be really good at... You wanna break the problem down into more specific tasks versus some broader set of high-level tasks. And then you can use models very specifically to get very good at each individual thing. And then, you know, you have an ensemble that sort of tackles the whole thing.I think a lot of good companies are doing that today. I still see a lot of companies, uh, uh, kind of giving the model single generic broad problems, versus breaking the problem down. And I think you- there will be more breaking the problem down, using specific models for specific things, including fine-tuning.

    20. LRLenny Rachitsky1:01:15

        And so in your case, because this is really interesting, is, is that you're using different, uh, levels of ChatGPT, like 01, 03, and stuff that's earlier-

    21. KWKevin Weil1:01:23

        Yeah.

    22. LRLenny Rachitsky1:01:23

        ... 'cause it's cheaper?

    23. KWKevin Weil1:01:24

        There'll be, there'll be parts, uh, of our internal stack. So we do, if you give you an example, uh, customer support with 400-plus weekly, uh, 400-plus million weekly active users, we get, you know, a lot of inbound tickets, right? I don't know how many customer support folks we have, but it's not very many, 30, 40, I'm not sure. Way f- way smaller than you would have at any comparable company. And it's because we've automated a lot of our flows. We've got, you know, most questions, using our internal resources, knowledge base, you know, uh, guidelines for how we answer questions, what kind of personality, et cetera. You can teach the model those things and then have it do a lot of its answers automatically. Or where it doesn't have, uh, you know, the full confidence to answer a particular question, it can still suggest an answer, request a human to look at it, and then that human's answer actually is its own sort of fine-tuning data for, for the model. You're telling it the right answer in a particular case. And, uh, and y- we're using, at various places, you know, some of these places you want a little bit more reasoning. It's not super latency sensitive, so you want a little more reasoning, and we'll use one of our O series models. In other places, you want a quick check on something, and so you're fine to use like 40 Mini, which is super fast and super cheap. And in general, it's like specific models for specific purposes, and then you, you, you ensemble them together to solve problems. By the way, again, not unlike how we as humans solve problems. A company is arguably an ensemble of models that have all been, you know, fine-tuned and based on what we studied in college and what we have, like, learned over the course of our careers. We've all been fine-tuned to have different sets of skills. And you, like, group them together in different configurations, and the output of the ensemble is much better than the output of any one individual.

    24. LRLenny Rachitsky1:03:20

        Kevin, you're blowing my mind. That sounds exactly correct. Uh, and also, different people, (laughs) are, you pay them less. Uh, they, they cost less to talk to. Some people take a long time to answer. Some people hallucinate. (laughs) This is a I'm telling you. (laughs)

    25. KWKevin Weil1:03:37

        Thi- this is like a, this is a mental model that really does work in, in thinking about it.

    26. LRLenny Rachitsky1:03:41

        Oh, man. Yeah. This is great. Some people are visual. They wanna draw out their thinking. Some people wanna talk, word cell. Wow, this is a really good metaphor. So again, coming back to your advice here, 'cause I love that it- we circled back to it. It's you're finding a really good way to think about how to design great AI experiences and LMs, I guess, specifically, is think about how a person would do this.

    27. KWKevin Weil1:04:01

        Well, it's, it's, it's maybe not always the answer is to think about how a person would do it, but, but sometimes to gain intuition for how you might solve a problem, you think about what an equivalent human would do in those situations and use that to, to, you know, at least gain a different perspective on the problem.

    28. LRLenny Rachitsky1:04:18

        Wow. This is great. There's

    29. NANarrator1:04:20

        I-

    30. KWKevin Weil1:04:20

        There's just like, like, you know-
16. 1:04:34 – 1:08:07

    Raising kids in an increasingly intelligent AI world

    1. KWKevin Weil1:04:34

    2. LRLenny Rachitsky1:04:34

       Okay. So speaking of humans, I wanna chat about the future a little bit. So you have three kids, and someone, a community member asked me this hilarious question that I think it's, it's something a lot of people are thinking about. So this is Patrick Sorrell, I worked at him- with him at Airbnb. He asks, so he says, "Ask what he's encouraging his kids to learn to prepare for the future. I'm worried my six-year-old by the year 2036 will face a lot of competition trying to get into the top roofing or plumbing programs and need a backup plan." (laughs)

    3. KWKevin Weil1:05:02

       That's funny. Um, so our kids are, we have a 10-year-old and eight-year-old twins. So they're, they're still pretty young. Uh, they're, they're kind of, I mean, it, it's amazing how AI native they are. Like, they just, it's completely normal to them that there are self-driving cars, that they can talk to AI all day long. Um, they have full conversations with ChatGPT and Alexa and everything else. I don't know. I think, uh, who knows what the future holds? I, I think, you know, things like coding skills are gonna be relevant for a long time. Who knows? But I, I think if you teach your kids to be curious, to be independent, to be self-confident, you teach them how to think, I don't know what the future holds, but I think that those are gonna be skills that are gonna be important in, in any configuration of the future. And so, you know, uh, it's not like we have all the answers, but that's how Elizabeth and I think about, uh, our kids.

    4. LRLenny Rachitsky1:06:02

       And do you find that AI... 'Cause there's a lot of talk about AI tutoring. Is that something you guys are doing? Anything you're... I know they're using ChatGPT. I love, I love all the photos you post, where they're playing with prompts and stuff.

    5. KWKevin Weil1:06:11

       (laughs)

    6. LRLenny Rachitsky1:06:12

       But I guess is there anything there you're, you're experimenting with or you think is gonna become really important?

    7. KWKevin Weil1:06:16

       Th- this is something that, uh, it's maybe the most, uh, important thing that, that AI could do. Maybe that's a, maybe that's a grand statement. There are lots of important things that AI can do, im- im- including like speeding up the pace of fundamental science research and discovery, which I, m- maybe is actually the most important thing AI can do. But, but one of the most important things would be personalized tutoring. And it kind of blows my mind that there is still... I, I know there are, there are a bunch of good products out there, like, you know, Khan Academy does great things. They're a wonderful partner of ours. Uh, Vinod Khosla has a nonprofit that has, uh, that, that's doing some really interesting stuff in this space and is making an impact. But I kind of want, like, I'm kind of surprised that there isn't like a two billion kid-... you know, AI personalized tutoring thing, because the models are good enough to do it now. And every, every study out there that's ever been done seems to show that when you have, you know, classrooms is still, classroom too is, like education is still important. But when you combine that with personalized tutoring, you get like multiple standard deviation improvements in learning speed. And so it's just, it's uncontroversial. It's good for kids. It's free. ChatGPT is free. You don't need to pay for, and, and the models are good enough. Like it still just kind of blows my mind that there isn't something amazing out there that, you know, our kids are using and your future kids are using and like people in all sorts of places around the world that aren't, uh, as lucky as our kids to be able to, like have this sort of built in solid education. Again, ChatGPT is free. People have Android devices everywhere. Like this could, I, I really just think this could change the world, and I'm surprised it doesn't exist and I want it

17. 1:08:07 – 1:14:20

    Why Kevin feels optimistic about our AI future

    1. KWKevin Weil1:08:07

       to exist.

    2. LRLenny Rachitsky1:08:08

       This kind of touches on something I wanna spend a little time on, which is a lot of people also worry a lot about AI, where it's going. They worry about jobs it's gonna take, they worry about, you know, super intelligence squashing humanity in the future. What's kind of your perspective on the, on that and just kind of the optimistic case that I think people need to hear?

    3. KWKevin Weil1:08:27

       Uh, I mean, I'm a big technology optimist. I think if you look over the last 200 years, uh, maybe, maybe more, technology has driven a lot of the advancements that have made us the, the world and the society that we are today. It drives economic advancements. It drives, um, uh, geopolitical advancements, quality of life, longevity advancement. I mean, technology is at the root of, of just about everything. So, uh, uh, I think there are very few examples where, uh, where, where this is anything but a great, a great thing over the longer term. That doesn't mean that there aren't like temporary dislocations or where there aren't individuals that are impacted, and that's, like that- that matters too. So, it can't just be that the average is good. You've got to also think about how you take care of each individual person as best you can. So, uh, it- it's something that we think a lot about and as we, you know, work with the administration, as we work with policy, like we- we try and help where- wherever we can. We do a lot with education. Um, I, you know, one of the, one of the benefits here is that ChatGPT is also perhaps the best like re-skilling app you could possibly want. It knows a lot of things. It can teach you a lot of things if you're interested in learning new things. So, but I, these are, these are very real issues. I'm super optimistic about the long run, and we're gonna need to do everything we can as a society to ensure that we, like make this transition, you know, a- a- as graceful and as well supported as we can.

    4. LRLenny Rachitsky1:09:59

       To give people a sense of where things might be going, that's a big question on a lot of people's minds. So someone asked this question that I love, which is, uh, AI is already changing creative work in a lot of different ways, writing and design and coding. What do you, what do you think is the next big leap? What should we be thinking is the next big leap in AI-assisted creativity specifically? And then just broadly, like where do you think things are gonna be going in the next few years?

    5. KWKevin Weil1:10:23

       Yeah. I, this is also an area where I'm, I'm a big optimist. Like, eh, if you, if you look at Sora, for example. I mean, we talked about Imagen earlier and the, the, the absolute like fount of creativity that people are putting across Twitter and Instagram and other places. Uh, I'm, I am the world's worst artist. Like, the worst. Maybe the only thing I'm worse at than, than, than art is singing. And I, you know, I, like give me a pencil and a pad of paper and I can't draw better than my five, uh, than our eight-year-old. You know, it's just like it's, but give me, give me Imagen and, you know, I can think some creative thoughts and put something into the model and suddenly have output that I couldn't have possibly done myself. That's pretty cool. Even, even you look at, um, at folks that are really talented, uh, I was talking to a director recently about Sora, someone who's directed films that, that, that we would all know. And, uh, and he was saying, you know, for, for a film that he's doing, like say, say, take the example of some sort of sci-fi-ish, you know, think of like Star Wars, and you've got some scene where there's a, there's a plane zooming into some Death Star-like thing. And so you've got the plane looking at the whole planet, and then you want to cut to a scene where the p- the plane's like, you know, kind of at the ground level and all of a sudden you see the city and everything else, right? How are you gonna manage that cut scene and, and that transition? And he, he was saying, "You know, in, in the world of two years ago, I would have paid a, a, you know, a 3D effects company, uh, 100 grand and they would have taken a month and they would have produced two versions of this cut scene for me, and I would have evaluated them. We would have chosen one because what are you gonna do? Like pay another 50 grand and wait another month? And, uh, and we would have just gone with it. And, you know, it would be fine." Like, movies are great. I love them and, and, um, there have been obviously we can do great things with the technology that we've had. But you now look at what you can do with Sora and, and his point was, "Now, I can use Sora, our video model, and I can get 50 different variations of this cut scene just, you know, me brainstorming into a prompt and the model brainstorming a little bit with me. I've got 50 different versions and, and then of course I can like iterate off of those and refine them and take different ideas. And now I'm still gonna go to that, that 3D effects studio to produce the final one, but I'm gonna go having brainstormed and like g- had this, had a much more creative approach with a, with an outcome that's much better and, and like I did that assisted by AI."My personal view on, on creativity in general is that it's ... No one's gonna ... Y- you don't type into Sora, like, "Make me a great movie." It requires creativity and ingenuity and all these things. But it can help you explore more, it can help you get to a better final result. So, you know, again, I tend to be an optimist in, in most things. But I'm actually ... I, I, I think, I think there's a very good story here.

    6. LRLenny Rachitsky1:13:31

       I know Sam Altman, I think it was him who tweeted recently the creative writing piece that you guys are working on-

    7. KWKevin Weil1:13:35

       Yeah. Yeah, yeah.

    8. LRLenny Rachitsky1:13:36

       ... where He's very bad at writing creative stuff. And he shared an example where it's actually really good. I imagine that's another area of investment.

    9. KWKevin Weil1:13:43

       Yeah. There's, there's some exciting stuff happening internally, um, with some new research techniques. So, uh, we'll have more to say about that at some point. But yeah, Sam, uh, Sam sometimes, uh, likes to show off some of the stuff that's coming, um, which is fine. By the way, it's like very sort of indicative of this iterative deployment, uh, philosophy. We don't have some breakthrough and keep it to ourselves forever and then, you know, bestow it upon the world someday. We kinda just talk about the things we're working on and share when we can, and launch early and often, and then iterate in public. And I,

18. 1:14:20 – 1:17:58

    The AI model you're using today is the worst AI model you'll ever use

    1. KWKevin Weil1:14:20

       I, I really like that philosophy.

Episode duration: 1:31:40