How OpenAI Builds for 800 Million Weekly Users: Model Specialization and Fine-Tuning

1. 0:00 – 8:36

   Introduction

   1. SWSherwin Wu0:00

      We want ChatGPT as a first-party app. First-party app's a really great way to get 800 million WAU's or whatever now. Tenth of the globe, right? [laughs] Yeah, yeah, a tenth of the globe uses it weekly. Every week. Every week. Yeah. Even within OpenAI, the, the thinking was that there would be, like, one model that rules them all. It's, like, definitely completely changed. It's, like, becoming increasingly clear that there will be room for a bunch of specialized models. There will likely be a proliferation of other types of model. Companies just have giant treasure troves of data that they are sitting on. The big unlock that has happened recently is with the reinforcement fine-tuning. With that set up, we're now letting you actually run RL, which allows you to leverage your data way more. Sherwin, thanks very much for joining. So we're being joined by Sherwin Wu. Um, it'd be great actually if you provided the long form of your background as we get into this, just for those that, that may not know you. I mean, I, I've viewed Sherwin as one of the, the top AI thought leaders- [laughs] ... so I'm really looking forward to this. Yeah, yeah. Thanks for having me. I'm, I'm, I'm really excited to be, be on the podcast. Um, yeah, so a little bit more of my, my background. So, uh, maybe I can start from present day and go backwards. So I, I currently lead, um, the engineering team for the O- for OpenAI's, uh, developer platform. So the biggest product in there, of course, is the, the API. Um- Is there more for the developer platform than the API? Um- I just kind of assumed they were synonymous. Well, so I, I also think about, uh, other things that we put into our platform side. So, like, technically our government work, uh, is, is also like offering- Oh, interesting ... and deploying the- Oh, I didn't know that ... into different areas. That's cool. Yeah, like I've talked about on some of our government deployments- Oh, like, so if you have, like, a local deployment, like- Yeah, yeah. So we actually do have a local deployment- I didn't know that ... uh, at Los Alamos National Labs. It's super cool. I went to visit it. It's, like- No kidding ... very different than what I'm used to. Um, but yeah, in a, like, you know, classified, uh, supercomputer- That makes sense, yeah ... uh, with, with our, with our model running there. Oh, that's cool. Um, so there's that. Um, but, like, mostly the, the API, um, 'cause- Did you go to Los Alamos? We did. Yeah, I did go to Los Alamos. [laughs] It's great. They, uh, showed us around. They showed us some of the historic sites. Real history. Um, yeah, yeah. I used to work at Livermore, man, so I've got, like, an affinity- Oh, yeah, yeah, yeah ... my first job out of college, so. Right, right, right. Yeah. Maybe you'll sell to them next. Yeah, yeah, yeah. Well, we, we hope to. We hope to. Um, but, uh, yeah, so I work on the developer platform. I've been working on it for around, uh, three years, uh, now. So I joined in 2022. Uh, was basically hired to work on the, the API product, which at the time was the only product that- Yeah ... that OpenAI had. Yeah. Um, and I've basically just worked on it the, uh, the entire time. Um, I've always been super interested in the developer side and kind of, like, the startup story of this technology- Right ... and so it's been really, really cool to kind of see, see this evolve. And so, um, that's my time at OpenAI. Before OpenAI, um, I was at, uh, Opendoor, uh, for around six years. I was working on the pricing side. Yeah. My, my general background before- [laughs] That, like, is such a dissonant, like, you know- Yeah, yeah ... pricing at Opendoor to, like, running API at a- It, it's such a different, uh- Yeah ... it's been fascinating actually for me to see the differences between the companies. Yeah. Like, they're run so differently. Um, they both have "open" in the name, so there's, you know, some overlap- [laughs] ... but, like, that's basic- that's pretty much it. Um, but yeah, I was there for around six years, uh, working on the pricing team. So our team basically would run the ML models, um- This is actually pricing the assets on Opendoor- Yeah, yeah ... like the inventory. Exactly. So Open... So yeah, Opendoor would buy and sell homes, and- Yeah ... the, their main product was buying homes directly from people selling them- Yeah ... for, with all cash offers. Yeah, yeah, yeah. And so my team was responsible for how much we would pay for them. Wow. Uh, and so it was a really fun, like, ML challenge. Yeah. Uh, it had a huge operational element to it as well- Yeah ... 'cause not everything was automated, obviously. Yeah. But it was a really fascinating technical challenge and, um, yeah I was working on that for- Is there any sense of that on the API side, like, like GPU capacity buying or is it just totally unrelated? Oh, on the, on the API side? Um, there is, is a small bit of, like, how we price the, the models but, uh, I don't think we do anything as sophisticated- Yeah ... as, uh, uh, as Opendoor. Opendoor was- Yeah ... just, like, such a hard problem. Yeah, of course. Of course. It's, like, such a, like, expensive asset. It's like y- like, the, the holding costs are very expensive. You're, like, holding onto it for, like, months at a time. There's, like, a variability in, like, the holding time. A massive lo- long tail of, like, potential things that could go wrong. Lo- long tail, yes. Okay, sorry. And, like, you know, you have to sell, you know, like, try to think about it from a portfolio perspective- Yeah ... and, like, if one of them just, like, you're holding onto it for two years, it blows- Everything ... you know, everything, like, goes negative. Yeah, yeah, yeah. So it's, uh, it's a very, very different- Six years? ... uh, different challenge. Yeah. Yeah, six years there. Wow. Uh, lots of ups and downs. Uh, saw a lot of the booms, saw a lot of the struggles, and then- Wow ... we IPO'd, uh- Yeah, yeah ... before I, be- before I left. Yeah. Um, but yeah, just in general it was, it was a very f- great experience. I think, uh, for me, it was, uh, it was also just, like, had such a very, like, business operations- Yeah ... and, like- Yes ... like, a very, like, by the book type of culture, whereas OpenAI is, like, very different. Well, it's so interesting. Um- I was just thinking about it now, is like even for a company like that, like, you don't think about it as a tech company, but if there is a deep technology problem, it actually is the pricing, right? Yes. It's actually an ML problem. It's not- Yeah, that's what attracted me to the company ... it's not, like, the website, it's not the, [laughs] you know- Yeah, yeah, yeah ... it's not the platform. It's, like, it's not the API. Yeah. It's literally that. Yep, yep, yep. Yeah. And that's what attracted me to it. For sure. I think that was, was interesting. Um, uh, it's also a way, like, lower margin business than, uh, OpenAI, uh, 'cause you're, like, making a tiny spread on these homes. Yeah. Uh- Right ... uh, they would talk about, like, basis points, like, eating bits for breakfast and all that, so. Yeah, yeah. Um, anyways, I was at Open, uh, Opendoor for around six years. Um, and then before that, uh, was my first job out of college, which was at Quora, um, with, uh, Adam D'Angelo and the group there. No way. Yeah, so I was working on the newsfeed. Oh, yeah. Uh, so worked on newsfeed ranking for a bit, worked on the product side. Yeah. Um, but was, uh, uh, that was actually my first exposure to, like, actual ML in industry, and, uh, learned a lot from, uh, uh, from, from the engineers at Quora. We basically hired a lot of the, the early, like, feed, uh, engineers from Facebook. Was, was Charlie still there when you were there? Charlie was not there when I, when I was there. He had- Okay, so you left right after he left. Yeah, yeah, yeah. That was a really legendary team, you know? So- I, I was super- It's still- Yeah ... it's still known to be kind of this super iconic founding team. Yeah, yeah. The, the, the early founding team was really solid. I still think that even while I was there, I was, I, I still, like, am amazed at the quality of the talent that we had. Yes. Phenomenal. I think this was, like, when the company was, like, 50 to 100 people. Yeah. But, like, yeah, like, a bunch of the Perplexity team was there. Yeah. Dennis, Dennis was on the feed team with me. Yeah. Yeah. John, uh, uh, uh, Johnny Ho- Yeah ... uh, Jerry Ma. Yeah, that's right. Yeah. Uh- This is crazy ... and then, and then Alexander The Scale, now at MSL. Yes, Alexander Wang, yes. You know, like, uh- Yeah, yeah. It was crazy ... was there. He was, he was, he was there between, uh, high school and college. [laughs] Um, it was a, an incredible team. I, I don't think I... I think I kinda took it for granted when I was there. Yeah. Uh, it was a good group. But, um-Uh, yeah

   2. MCMartin Casado6:05

      How, and how did you, how did you get to Quora? What'd you study in undergrad?

   3. SWSherwin Wu6:08

      Yeah, so before that I was at MIT for undergrad. I studied computer science.

   4. MCMartin Casado6:12

      Okay.

   5. SWSherwin Wu6:12

      Um, uh, did like one of those, like, computer science and the master's degree-

   6. MCMartin Casado6:15

      Okay, yeah

   7. SWSherwin Wu6:15

      ... kinda like crammed it in.

   8. MCMartin Casado6:16

      Yeah, yeah, yeah.

   9. SWSherwin Wu6:16

      Um, uh, I ended up at Quora because I got an, uh, what we call an externship there. So, like, uh, at MIT, uh, you actually get January off, so there's like the fall semester-

   10. MCMartin Casado6:26

       Oh, wow

   11. SWSherwin Wu6:26

       ... and then January's off.

   12. MCMartin Casado6:28

       That's cool.

   13. SWSherwin Wu6:28

       And then you have, uh, the spring semester.

   14. MCMartin Casado6:29

       Yeah.

   15. SWSherwin Wu6:29

       And so it's kind of this, it's called independent activities period. So some people just, like, take classes, some people just do nothing.

   16. MCMartin Casado6:35

       Okay.

   17. SWSherwin Wu6:35

       But it's, uh, some people will do, like, month-long internships.

   18. MCMartin Casado6:38

       Okay.

   19. SWSherwin Wu6:39

       And some crazy companies will, like, offer a month-long internship to a college student.

   20. MCMartin Casado6:42

       Oh, that's cool. Yeah, yeah.

   21. SWSherwin Wu6:42

       And it really is just kinda like a way to get, uh, people interested-

   22. MCMartin Casado6:44

       Did you come out here from Boston?

   23. SWSherwin Wu6:47

       Yeah. Yeah, it was-

   24. MCMartin Casado6:47

       How did that work?

   25. SWSherwin Wu6:47

       Yeah, it was crazy. So, um, so, so, so, uh, you had to apply. I remember, uh, yeah, this is like, I think 2013 s- uh, January or something. Um, you had to apply, and I remember the Quora internship was the one that just paid the most. They paid, I think it was like 8,000, $9,000.

   26. MCMartin Casado7:02

       Nice. [laughs]

   27. SWSherwin Wu7:02

       And I was like, "Wow, this is like all for a month," and you're, you're just kinda ramping up like half the time.

   28. MCMartin Casado7:06

       I can eat for a year. [laughs]

   29. SWSherwin Wu7:07

       Yeah, yeah. As a college student I was like-

   30. MCMartin Casado7:08

       Yeah, yeah
2. 8:36 – 12:18

   Horizontal vs vertical OpenAI

   1. SWSherwin Wu8:36

      Yeah.

   2. MCMartin Casado8:36

      So one place I wanted to start, um, is, uh, is something that I find very unique about, um, OpenAI, uh, is it's both a pretty horizontal company, like it's got an API. Like I would say we've got this massive portfolio of companies, right? And I would say a good fraction of them use the, the API. And then it's also a vertical company in that you've got full-on apps, right?

   3. SWSherwin Wu9:00

      Yep.

   4. MCMartin Casado9:00

      Um, you know, like everybody uses ChatGPT, for example. And so, you know, you're responsible for like the API and kind of the dev tool side. So maybe just to begin with, like is there an internal tension between the two? Like, is that a discussion? Like, you know, like, like the API may, whatever, it may help a competitor to like the vertical version, or is it not... that things are just growing-

   5. SWSherwin Wu9:25

      Yeah

   6. MCMartin Casado9:25

      ... so fast it's not an issue. I would just love to... how you think about that. By the way, it's very unusual for companies to have both of that.

   7. SWSherwin Wu9:29

      Yeah, yeah.

   8. MCMartin Casado9:29

      These two things this early. It's very unusual.

   9. SWSherwin Wu9:31

      Yeah, yeah. I completely agree. Uh, I think there is, there's, there's some amount of tension. I, uh, I would say it's... I think one thing that really helps here is, uh, Sam and Greg, just from a, like, founder, you know, perspective-

   10. MCMartin Casado9:43

       Yeah

   11. SWSherwin Wu9:43

       ... have since day one just been very principled in, in, in, in the way in which we approach this. They've always, you know, have kinda told us, "You know, we want ChatGPT as a first party app."

   12. MCMartin Casado9:52

       Yeah.

   13. SWSherwin Wu9:52

       "Uh, we also want, want the API." And then, and the nice thing is, I think they're able to do this because, uh, at the end of the day, it kinda comes back to the, the mission of OpenAI, which is, uh, to create AGI and then to distribute the benefits as broadly as possible.

   14. MCMartin Casado10:03

       Yeah.

   15. SWSherwin Wu10:03

       And so if you interpret this, you want it in as many surfaces as, as you want, you... and the first party app's a really great way to get, you know, uh, what is it, like 800 million, uh, WAU's or whatever now. And-

   16. MCMartin Casado10:12

       Yeah

   17. SWSherwin Wu10:12

       ... um, uh, but the-

   18. MCMartin Casado10:14

       800 million WAU's?

   19. SWSherwin Wu10:15

       Uh, yeah. It's, yeah. It's pretty... It's, it's actually mind-boggling to think about. Um-

   20. MCMartin Casado10:18

       I have to say, I don't think... Many people listening to this don't understand how big that is.

   21. SWSherwin Wu10:21

       Yeah.

   22. MCMartin Casado10:21

       But that is, I mean-

   23. SWSherwin Wu10:22

       It's, it's, it's crazy, yeah. Yeah

   24. MCMartin Casado10:23

       ... that's gotta be like actually historic for the, for the, the time it's taken to get to 800 million, but like-

   25. SWSherwin Wu10:27

       It's historic. It's, um-

   26. MCMartin Casado10:29

       It's also just-

   27. SWSherwin Wu10:29

       ... also just like, yeah, the amount of time and just like how much we've had to scale up, just like-

   28. MCMartin Casado10:32

       A, a 10th of the globe, right? [laughs]

   29. SWSherwin Wu10:35

       Yeah, yeah. 10% of the globe uses it week-

   30. MCMartin Casado10:36

       Every week. Every week.
3. 12:18 – 15:11

   Why you can’t “disintermediate” the model

   1. MCMartin Casado12:18

      ... which is the, the, the problem historically with, you know, offering, um, a, a core service as an API is you can get disintermediated, right? And so I can build on top of it, but then-You know, the user doesn't know, like whatever, I build on top of the cloud-

   2. SWSherwin Wu12:33

      Mm-hmm

   3. MCMartin Casado12:33

      ... but I disintermediate from the cloud, and then I can switch to another cloud or whatever.

   4. SWSherwin Wu12:37

      Yeah.

   5. MCMartin Casado12:37

      And it occurs to me that that's kind of hard to do with these models because the models are so hard to abstract away. Like, they're just, they're just unruly, right?

   6. SWSherwin Wu12:45

      Yeah.

   7. MCMartin Casado12:46

      If you try to like have traditional software drive them, they just don't kinda manage very well.

   8. SWSherwin Wu12:51

      Mm-hmm.

   9. MCMartin Casado12:51

      So part of me thinks that it's almost like this like anti-disintermediation technology that you kind of have to expose it to the, to the user directly.

   10. SWSherwin Wu13:01

       Yep.

   11. MCMartin Casado13:01

       Does that make sense?

   12. SWSherwin Wu13:02

       Yeah.

   13. MCMartin Casado13:02

       And so I'm wondering if like, so even if I think ChatGPT is really just trying to expose the model to the user, the API's kinda just trying to expose the model to the user. So I think there's almost this argument that's like if the real value is in the models, it doesn't really matter how you get it to them, 'cause it's-

   14. SWSherwin Wu13:15

       Yep

   15. MCMartin Casado13:15

       ... gonna be very tough for someone to, to abstract it away in, in, in, in the classic sense of computer science, of like they don't know-

   16. SWSherwin Wu13:21

       Yeah

   17. MCMartin Casado13:21

       ... that they're using the model. Like you always know you're using GPT-5.

   18. SWSherwin Wu13:24

       Yeah. And, and the interesting thing is I think like the entire industry kind of has slowly changed their mind around this too. I think like in the beginning we-

   19. MCMartin Casado13:29

       That's true, yeah

   20. SWSherwin Wu13:29

       ... kind of thought like, oh, these are all gonna be interchangeable.

   21. MCMartin Casado13:31

       It's just like software.

   22. SWSherwin Wu13:32

       Yeah, yeah. Exactly.

   23. MCMartin Casado13:33

       And like API stuff.

   24. SWSherwin Wu13:33

       So it's a piece of infrastructure that you can just swap out, yeah. But I think we're learning this on the product side with like, you know, the GPT-5 launch and like 4.0 and like how, how so many people liked 03 and 4.0 and, and all of that. We're also seeing this-

   25. MCMartin Casado13:43

       I felt that. I felt that when it changed.

   26. SWSherwin Wu13:45

       [laughs]

   27. MCMartin Casado13:45

       I'm like, I'm like, "You're not as nice to me." [laughs]

   28. SWSherwin Wu13:47

       Yeah.

   29. MCMartin Casado13:48

       I'm like, "I like the validation."

   30. SWSherwin Wu13:49

       Yeah.
4. 15:11 – 17:30

   People build relationships with models

   1. MCMartin Casado15:11

      do you think that is because of a relationship between the user and the model, or do you think it's more of a technical thing, which is like my evals work for like OpenAI and it's, you know, and like the correctness maintains or-

   2. SWSherwin Wu15:25

      Yeah. Yeah

   3. MCMartin Casado15:25

      ... or?

   4. SWSherwin Wu15:25

      I think it's both. Um, so I think there's, there's definitely an end user piece here, which is what we've heard from, from some of our customers, like they just get familiar with, with the model itself. Uh, but I also think there's a technical piece, which is like the... Also, as a developer, especially with startups, you're like really going deep with these models and like really like iterating on it, trying to get, get it really good within your particular harness. You're iterating on your harness itself. You're giving it different tools here and there. Uh, and so you really do end up like building a product around the model, and so there is a technical piece where, you know, as you kind of keep building with a particular, uh, uh, product like GPT-5, um, you're actually like building more around it so that your product works uniquely-

   5. MCMartin Casado16:03

      Yeah

   6. SWSherwin Wu16:03

      ... well with that, with that model.

   7. MCMartin Casado16:05

      So, so I, I use, I use Cursor and, um, a lot, just for like a lot of stuff, like, like writing blogs and like-

   8. SWSherwin Wu16:11

      Yeah

   9. MCMartin Casado16:11

      ... you know, you know, we're investors, and I use it for sometimes for coding. And it's remarkable how many models I use in Cursor. So like literally my go-to model is GPT-5.

   10. SWSherwin Wu16:19

       Yeah.

   11. MCMartin Casado16:19

       I love GPT-5. I think it's a phenomenal like, you know... And then like I use like max mode with GPT-5 for planning, and then, but, you know, like I mean, I like the tab complete model that's in Cursor and like-

   12. SWSherwin Wu16:30

       Yep

   13. MCMartin Casado16:30

       ... you know, the, the new model they just dropped is for like some basic, you know, some stuff is, is good.

   14. SWSherwin Wu16:33

       Yeah, the Composer one.

   15. MCMartin Casado16:34

       Like yeah, the Composer one's good.

   16. SWSherwin Wu16:36

       Yep.

   17. MCMartin Casado16:37

       Um, and so like, you know...

   18. SWSherwin Wu16:40

       And I think that like kinda reflects this too, 'cause it's like-

   19. MCMartin Casado16:42

       Yeah

   20. SWSherwin Wu16:42

       ... it's a particular model for each particular use case.

   21. MCMartin Casado16:44

       Yes. Yeah, yeah, yeah.

   22. SWSherwin Wu16:44

       Like I've talked to a bunch of people who've used the new Composer model, and-

   23. MCMartin Casado16:46

       Yeah

   24. SWSherwin Wu16:46

       ... it's just really good for like fast-

   25. MCMartin Casado16:48

       It's super fast

   26. SWSherwin Wu16:49

       ... like first pass.

   27. MCMartin Casado16:50

       Exactly. That's right. Yeah.

   28. SWSherwin Wu16:50

       Like keep you in flow kinda thing.

   29. MCMartin Casado16:52

       Yeah.

   30. SWSherwin Wu16:52

       And then you kind of like bubble out to another model if you want, like, you know, deeper thinking or something like that.
5. 17:30 – 20:10

   Not one AGI model, but many

   1. MCMartin Casado17:30

      the first one is what does that mean to A- for AGI? [laughs]

   2. SWSherwin Wu17:32

      [laughs]

   3. MCMartin Casado17:32

      And the second one is what does that mean for OpenAI? Like does that mean that like you end up with a model portfolio? Do you select a subset? Do you think this all gets superseded by some God model in the future? Like how does that play out? 'Cause it's, it's, it's against what most people thought.

   4. SWSherwin Wu17:46

      Yeah.

   5. MCMartin Casado17:46

      Most people thought this is all going towards one large model that does everything.

   6. SWSherwin Wu17:49

      Yeah. I think the, the crazy thing about all this is just like how everyone's thinking has just changed over time.

   7. MCMartin Casado17:53

      Totally.

   8. SWSherwin Wu17:54

      Like the, I, I distinctly remember this like... And, and, and the crazy thing, it's not that long ago. It's just like three, like two or three years ago.

   9. MCMartin Casado17:59

      Yeah.

   10. SWSherwin Wu18:00

       I remember like even with an OpenAI, the, the thinking was that there would be like one model that rules them all.

   11. MCMartin Casado18:04

       Yep.

   12. SWSherwin Wu18:04

       And it's like, why would you... I mean, like this kinda goes to the fine-tuning API product. It's like, why would you even have a fine-tuning product? Why would you even want to like iterate on it?

   13. MCMartin Casado18:10

       Yep. Yep.

   14. SWSherwin Wu18:11

       Um, there's gonna be just one model that just subsumes everything.

   15. MCMartin Casado18:13

       Yeah, yeah, yeah.

   16. SWSherwin Wu18:13

       And that was also kind of the v- that, that is also like the most simplistic like view of what the, what the AGI will look like.

   17. MCMartin Casado18:18

       Yeah.

   18. SWSherwin Wu18:19

       Um, and, and yeah, it's like definitely completely changed since then. I think one, uh... And, and, and but then the other thing to keep in mind is like it might continue to change, like even from where we are today.

   19. MCMartin Casado18:29

       Yeah.

   20. SWSherwin Wu18:29

       But it's like becoming increasingly clear, I think that, um, uh-There, there will be room for a bunch of specialized models. There will likely be a proliferation of other types of models. I mean, you see us do this with like the Codex model-

   21. MCMartin Casado18:39

       Yeah, totally

   22. SWSherwin Wu18:40

       ... uh, uh, itself. We have like, you know, we have like GPT-4.1 and like 4.0 and like 5 and, and, and, and all of this. Um, and so I, I definitely think there's, there's room for all, for, for all this. I, I, I don't think that's bad for what it's worth. Like-

   23. MCMartin Casado18:52

       Yeah

   24. SWSherwin Wu18:52

       ... um, if anything, I think, you know, as we've tried to move towards AGI, things have just been very unexpected, and I think the market just evolves and the product portfolio evolves because of that. Um, so I, I don't think it's a bad thing at all. Um, what I do think it means-

   25. MCMartin Casado19:03

       I think you could, you could easily argue it's very good for OpenAI and very good for like the model companies to like-

   26. SWSherwin Wu19:08

       Yeah, because-

   27. MCMartin Casado19:09

       ... not have like, you know, winner take all consolidated dynamics, right? I mean, you just have a healthier ecosystem, a lot more solutions you can provide, a lot-

   28. SWSherwin Wu19:16

       Yeah

   29. MCMartin Casado19:16

       ... you know.

   30. SWSherwin Wu19:17

       Yeah. A- and as, as the ecosystem grows, it generally is helpful.
6. 20:10 – 24:44

   Fine-tuning, RFT, and customer data choices

   1. MCMartin Casado20:10

      so I, I've, I've noticed that you are moving towards kinda more sophisticated use of things like, you know-

   2. SWSherwin Wu20:14

      Yep

   3. MCMartin Casado20:14

      ... like fine-tuning. Um, which, you know, in a way you could read that as a bit of a capitulation that like, you know, there is product specific data and there's pr- product specific use cases that a general model won't do, to your point, right?

   4. SWSherwin Wu20:27

      Mm-hmm.

   5. MCMartin Casado20:28

      So like as opposed to proliferation model, you do that. Um, it seems like a lot of that data is actually very, very valuable, right?

   6. SWSherwin Wu20:33

      Yep.

   7. MCMartin Casado20:33

      And so, you know, to what extent is there, like, interest in almost a tit for tat where you can like expose, you know, the ability to get product data into fine-tuning, and then you also benefit from that data because-

   8. SWSherwin Wu20:52

      Oh, yeah

   9. MCMartin Casado20:52

      ... the, uh, the, the vendors provide it to you, um, versus like this is 100%, you know, like they keep their own data and-

   10. SWSherwin Wu20:59

       Yeah

   11. MCMartin Casado20:59

       ... there's kind of no interest in that. 'Cause it feels to me like the next level of scaling, this is kind of where-

   12. SWSherwin Wu21:03

       Yeah

   13. MCMartin Casado21:03

       ... we're at. And so I'm just kinda curious how.

   14. SWSherwin Wu21:06

       Yeah. So I mean, uh, maybe even like taking a step back, the, the main reason why we even invested in a fine-tuning API in the very beginning is, uh, one, there's been huge demand from people to be able to customize the models a bit more. It kinda goes into like prompt engineering and also like I think the industry's changed their mind on that as well.

   15. MCMartin Casado21:22

       Yeah.

   16. SWSherwin Wu21:22

       Like it, it's evolved. But the second thing is exactly what you said, which is the companies just have giant treasure troves of data that they are sitting on-

   17. MCMartin Casado21:29

       Mm

   18. SWSherwin Wu21:29

       ... that they would like to utilize in some fashion in this AI wave. And you can, you know, the simple thing is to put it in like, you know, some like vector, like do rag with it or something.

   19. MCMartin Casado21:38

       Yeah.

   20. SWSherwin Wu21:38

       But there's also, you know, if they have a more technical team, they do wanna see how they can use it-

   21. MCMartin Casado21:41

       Yeah

   22. SWSherwin Wu21:41

       ... to customize the models. And, uh, and so that is actually the main reason why we've invested in, in this. The, the interesting thing was, uh, way back, kinda back in like '22, '23, our fine-tuning offering was, I'd, I'd say like too limited, so that it was very difficult for people to, to tap into and use this data.

   23. MCMartin Casado21:57

       Right.

   24. SWSherwin Wu21:58

       So it was just like an SF, like a-

   25. MCMartin Casado21:59

       Yeah

   26. SWSherwin Wu21:59

       ... supervised fine-tuning-

   27. MCMartin Casado22:00

       Yep

   28. SWSherwin Wu22:00

       ... API. And like, we're like, "Oh, you can kind of use it," but in practice, it really is only useful for like, like, uh, it's, it's honestly just like instruction following plus plus.

   29. MCMartin Casado22:08

       Yeah, yeah, yeah.

   30. SWSherwin Wu22:08

       You like kind of change the tone, and you're just like really like instructing it.
7. 24:44 – 28:06

   Prompt engineering isn’t the point anymore

   1. SWSherwin Wu24:44

      Yeah.

   2. MCMartin Casado24:44

      Okay. You, you said that, uh, views on prompt engineering have changed.

   3. SWSherwin Wu24:47

      Yeah. [laughs]

   4. MCMartin Casado24:48

      I wasn't, actually I wasn't aware of that. [laughs] All the other things I wasn't aware of, this one I wasn't. How is the prompt engineering-

   5. SWSherwin Wu24:51

      Yeah. I mean, I think the prevailing view, this is back in 2022. I remember I was talking to so many people, and they're basically... I mean, this is similar to like the single model AGI view as well, which is like-Like prompt engineering is just not gonna be a thing, and you're just not gonna have to think about what you're putting in the, in the context window in the future.

   6. MCMartin Casado25:07

      Oh.

   7. SWSherwin Wu25:07

      Like the model will just be good enough-

   8. MCMartin Casado25:08

      Oh, yeah, yeah, yeah

   9. SWSherwin Wu25:08

      ... and it will just like know.

   10. MCMartin Casado25:09

       Yeah, yeah, yeah.

   11. SWSherwin Wu25:09

       It'll know what, what, what you need to do.

   12. MCMartin Casado25:11

       Okay. Yeah, that's definitely-

   13. SWSherwin Wu25:11

       And so like there's like-

   14. MCMartin Casado25:12

       That's definitely not a thing. [laughs]

   15. SWSherwin Wu25:13

       Yeah, but like that, that... Like I, I don't know, maybe people forget it, but like that was like a very common-

   16. MCMartin Casado25:17

       No, it was a great point

   17. SWSherwin Wu25:17

       ... use back then-

   18. MCMartin Casado25:17

       It was a great point, yeah

   19. SWSherwin Wu25:17

       ... 'cause like the scaling laws or whatever-

   20. MCMartin Casado25:19

       Yeah

   21. SWSherwin Wu25:19

       ... something about the scaling laws, and like you'll just mind meld with the model, and like-

   22. MCMartin Casado25:22

       Yeah

   23. SWSherwin Wu25:22

       ... you just like, like prompting and like instruction following will just, will be so good that you won't really need to do it.

   24. MCMartin Casado25:27

       Yeah, yeah, yeah.

   25. SWSherwin Wu25:27

       And if anything, like yeah, it's like clearly been wrong and-

   26. MCMartin Casado25:29

       Yeah, yeah, yeah.

   27. SWSherwin Wu25:29

       Um, uh, but it is interesting because it's, I think it's a slightly different world that we're in now where the models have gotten really, really good at instruction following relative to the-

   28. MCMartin Casado25:37

       Yeah

   29. SWSherwin Wu25:37

       ... you know, like GPT-3.5 or something.

   30. MCMartin Casado25:39

       Yeah.
8. 28:06 – 31:55

   What an “agent” really is

   1. SWSherwin Wu28:06

      but my, my general take on agents is it's, it's a, it's an, it's an AI that will take actions on your behalf that can work over long time horizons.

   2. MCMartin Casado28:12

      Okay.

   3. SWSherwin Wu28:13

      And I think that's the, that's the mo- pretty general-

   4. MCMartin Casado28:15

      Pretty utilitarian-

   5. SWSherwin Wu28:15

      Yeah, yeah

   6. MCMartin Casado28:16

      ... definition.

   7. SWSherwin Wu28:16

      But like if you think about it that way, yeah, I, I mean, maybe this is what you mean by modality, but it is just a like way of like using AI, and it is a... I guess it could be viewed as a modality, but we don't view it as like a separate thing, separate from a-

   8. MCMartin Casado28:29

      Well, let me, let me, let me just-

   9. SWSherwin Wu28:32

      ... AI, uh, API, and ChatGPT

   10. MCMartin Casado28:32

       ... let me just, let me just try and kind of, uh, you know, give you a sense of where this question's coming from. Like I know how to build a product. Like, and we know how to do go to market for products. We know how to do like, you know, we know the implications of turning them into platforms. Like it's just we've been doing this for a very long time, right?

   11. SWSherwin Wu28:46

       Mm-hmm.

   12. MCMartin Casado28:46

       We know how to do the same thing for APIs, right? We know how to do billing. We know like the tension of like people build on top of it and all of that stuff. And like what I've been trying to... And this is just maybe a, a personal inquiry. It's just not clear for me for an agent if you ha- if, if it, if it sits in one of those two camps. Is it more like the product camp? [laughs] Is it more like the... Or is, or-

   13. SWSherwin Wu29:07

       I see, I see

   14. MCMartin Casado29:07

       ... 'cause it's kind of both. Like, I could like literally give you Codex.

   15. SWSherwin Wu29:10

       Yeah, yeah.

   16. MCMartin Casado29:11

       And like as a user, and then you just talk to it, or I could like build in a way, kind of embed it in like my app.

   17. SWSherwin Wu29:19

       Yeah.

   18. MCMartin Casado29:20

       And so like... But then that means something to you as far as like, you know, how do you price it, and what does it mean for ecosystem? Like-

   19. SWSherwin Wu29:26

       Yeah

   20. MCMartin Casado29:26

       ... like for example, like would you be fine if I started a company and just like built it around Codex? Is that a thing?

   21. SWSherwin Wu29:31

       Starting a company and building it around Codex?

   22. MCMartin Casado29:33

       Around Codex, yeah.

   23. SWSherwin Wu29:33

       I actually think that'd be great. Like it's a-

   24. MCMartin Casado29:35

       Yeah.

   25. SWSherwin Wu29:35

       We, we like released like the Codex SDK, and we like want people-

   26. MCMartin Casado29:38

       Yeah. No, I saw that

   27. SWSherwin Wu29:38

       ... to be able to build in it-

   28. MCMartin Casado29:39

       Yeah

   29. SWSherwin Wu29:39

       ... and hack on it. Yeah. Actually, I think this might be what you're getting at, which is, um, uh, and this is like a kind of a unique thing about OpenAI and, and kind of reflects on how, how it's run, which is at the end, like at the end of the day, OpenAI is like a, an AGI company. It's like an intelligence company.

   30. MCMartin Casado29:52

       Yeah, yeah. For sure.
9. 31:55 – 36:46

   How OpenAI thinks about pricing

   1. MCMartin Casado31:55

      mean, how, how, how have you evolved your thinking and how do you price these, you know, access to intelligence where, you know, you don't know how many people are gonna use it. It's almost certainly usage-based billing-

   2. SWSherwin Wu32:07

      Yeah

   3. MCMartin Casado32:07

      ... not something else. Like, can you-

   4. SWSherwin Wu32:08

      Yeah

   5. MCMartin Casado32:09

      ... can you talk just a bit about like philosophy around pricing on these things? Is it different for product versus API or like?

   6. SWSherwin Wu32:14

      Yeah. I think, um, the, the, the, the honest truth here is like it's evolved over time as well. And, and like I actually think the simplest... Like, the reason why we've done, uh, usage-based pricing on the API honestly is because it's then, like, it, it's closest to how it's actually being used.

   7. MCMartin Casado32:27

      Yeah.

   8. SWSherwin Wu32:27

      And so that's kinda how we, how we started. I actually think usage-based pricing on the API has h- has, has, like, surprisingly held strong. And, like, I actually think this might be something that we'll keep doing for, for quite a long time. Mostly because, um-

   9. MCMartin Casado32:40

      The cogs are so high, I don't know how you don't do usage-based.

   10. SWSherwin Wu32:42

       Yeah, yeah, yeah, yeah.

   11. MCMartin Casado32:43

       I just don't know h- how that-

   12. SWSherwin Wu32:44

       Yeah

   13. MCMartin Casado32:44

       ... evolves.

   14. SWSherwin Wu32:44

       And then, and then, and then there's also the strategy of, like, how we price it. And, and, and internally o- one thing we do is, is, uh, we always make sure that we actually price our, our, our, our usage-based pricing from a, like, cost-plus perspective.

   15. MCMartin Casado32:54

       Right.

   16. SWSherwin Wu32:54

       Like, we're n- we're actually just, like-

   17. MCMartin Casado32:55

       Yeah

   18. SWSherwin Wu32:56

       ... trying to make sure that we're being responsible-

   19. MCMartin Casado32:57

       Yeah, yeah

   20. SWSherwin Wu32:57

       ... from a-

   21. MCMartin Casado32:58

       Totally makes sense

   22. SWSherwin Wu32:58

       ... from a margin perspective. Uh-

   23. MCMartin Casado33:00

       Yeah, yeah, yeah. By the way, this is a huge shift in the industry in general, just because, like, I remember the shift from on-prem to, uh, to recurring.

   24. SWSherwin Wu33:05

       Yeah.

   25. MCMartin Casado33:05

       That was a big, big deal. Like, that created Zuora. Like, it like created whole companies.

   26. SWSherwin Wu33:09

       Yep, yep, yep.

   27. MCMartin Casado33:09

       There's like whole books on it and all, like-

   28. SWSherwin Wu33:11

       Yeah, yeah, yeah

   29. MCMartin Casado33:11

       ... a bunch of consultants on how you do this. It changed, like-

   30. SWSherwin Wu33:13

       Yeah
10. 36:46 – 42:57

    Why open-weights don’t kill the API

    1. SWSherwin Wu36:46

       stay with that for a while.

    2. MCMartin Casado36:46

       So how do you think about open source? I mean, uh, you know, I think you're the only big lab that's releasing open source. Is that-

    3. SWSherwin Wu36:53

       No, Google has, uh, some of theirs.

    4. MCMartin Casado36:55

       Okay.

    5. SWSherwin Wu36:55

       Um, yeah, mostly smaller models on their side.

    6. MCMartin Casado36:57

       Yeah, yeah, yeah.

    7. SWSherwin Wu36:57

       Yep.

    8. MCMartin Casado36:57

       That's right.

    9. SWSherwin Wu36:58

       Yep, yep.

    10. MCMartin Casado36:58

        Yeah. So how do you think about open source vis-a-vis, you know, competition, cannibalization, you know? Like, what's the, uh, what's the strategical... What's the complexity?

    11. SWSherwin Wu37:09

        Yeah, yeah. Um, so, uh, I personally love open source. Like, I, I think, I think it's great that there's a, there's a-

    12. MCMartin Casado37:15

        All of us grew up with it, right? [laughs]

    13. SWSherwin Wu37:17

        Yeah, all of us grew up with it. Like, the internet wouldn't exist without it. Like, you know, so much of the world, world, world is built, built on top of it.

    14. MCMartin Casado37:21

        Cloud wouldn't exist without of it, basically.

    15. SWSherwin Wu37:23

        Yeah, yeah.

    16. MCMartin Casado37:23

        Nothing would exist without of it-

    17. SWSherwin Wu37:25

        Yeah

    18. MCMartin Casado37:25

        ... except for maybe Windows.

    19. SWSherwin Wu37:26

        And so it, it was interesting 'cause, like, I felt like over the last... There was before we launched the open source model, I know Sam feels this way as well.

    20. MCMartin Casado37:31

        Yeah.

    21. SWSherwin Wu37:32

        It's like there's this, like, weird, like, you know, uh, mindset where because OpenAI hadn't launched anything, it just seemed like it was super, like, anti... Like, OpenAI was, like, super anti-open source.

    22. MCMartin Casado37:42

        Yeah, yeah.

    23. SWSherwin Wu37:43

        But I'd actually been having conversations with Sam ever since I joined about open sourcing a model.

    24. MCMartin Casado37:46

        Mm-hmm.

    25. SWSherwin Wu37:46

        We were trying to think about, like, well, how can we sequence it?

    26. MCMartin Casado37:49

        Yeah.

    27. SWSherwin Wu37:49

        What compute is always the hard thing. It's like, do we have the compute to kinda like train, train this thing?

    28. MCMartin Casado37:53

        Yeah.

    29. SWSherwin Wu37:53

        So we've always wanted to, to-

    30. MCMartin Casado37:54

        Yeah
11. 42:57 – 45:47

    Different stacks for text, images, video

    1. MCMartin Casado42:57

       Like-

    2. SWSherwin Wu42:57

       Yeah

    3. MCMartin Casado42:57

       ... how, how do you operationalize?

    4. SWSherwin Wu42:59

       Yeah. I think, uh, I, I think you're totally right. It's an anti-pattern. It's pretty tough to pull off. Um, uh, I think honestly, like props to Mark on our research team for like, you know, s- structuring things in a way where we're, we're able to do it. Um, from, from my perspective, I think the biggest thing is I think our, like, image, uh, like our, our, um, I think called like the world simulation team or like the team that builds Sora and all that under Aditya-

    5. MCMartin Casado43:21

       Yeah

    6. SWSherwin Wu43:21

       ... uh, is just extremely solid. Like they are pro- it's like the highest concentration of, like, talent that I've seen-

    7. MCMartin Casado43:27

       Wait, but no-

    8. SWSherwin Wu43:27

       ... in a while

    9. MCMartin Casado43:28

       ... so but, but is it the same, like, is it the sa- is it like, are they like totally separate infrastructure? Do they use the same infrastructure?

    10. SWSherwin Wu43:33

        Yeah, yeah, yeah. So it's, it's, it's actually like pretty separate. So and I think that's part of the-

    11. MCMartin Casado43:36

        Oh, okay. Yeah

    12. SWSherwin Wu43:36

        ... the reason why we're able to-

    13. MCMartin Casado43:37

        That makes sense

    14. SWSherwin Wu43:37

        ... to kind of do this.

    15. MCMartin Casado43:38

        Yeah, yeah, yeah.

    16. SWSherwin Wu43:38

        Well, it's like one is like the team needs to be extremely strong, which, which, which they are.

    17. MCMartin Casado43:40

        Yeah. That's fair.

    18. SWSherwin Wu43:40

        And then two is, um, they're, they're, they're, they're run very separately. They're kind of like thinking about their own, uh, par- particular roadmap. They think about productization very separately as well, right? Which is how like the Sora app kind of, uh-

    19. MCMartin Casado43:50

        Yeah, yeah, yeah

    20. SWSherwin Wu43:51

        ... came, came, came out of that as well. Um-And then, uh, yeah, even like the inference stacks are, are, are slightly different-- are, are, are kind of like different. Um, they, they, they own a lot more around their-

    21. MCMartin Casado44:00

        Right

    22. SWSherwin Wu44:00

        ... inference stack and they optimize their inference stack pretty, pretty, uh, separately. And so, um-

    23. MCMartin Casado44:05

        Maybe you could-

    24. SWSherwin Wu44:05

        I think that, that contributes to, to helping us run things in parallel-

    25. MCMartin Casado44:08

        I see

    26. SWSherwin Wu44:08

        ... but, uh, uh, it's, it's pretty hard to pull off, for sure.

    27. MCMartin Casado44:11

        May-may-maybe you can educate, uh, this on me. Like, so I think about APIs as mostly text-based for an opening. Do you guys do actual... Do you do actual-

    28. SWSherwin Wu44:18

        Yeah

    29. MCMartin Casado44:18

        ... pixel-based stuff?

    30. SWSherwin Wu44:19

        Yeah, yeah. We do. We, we have a bunch. Uh, so DALL-E, DALL-E-
12. 45:47 – 53:21

    How the agent builder actually works

    1. SWSherwin Wu45:47

       Yep.

    2. MCMartin Casado45:47

       It seems to me that like actually how you build agents and expose them has evolved too.

    3. SWSherwin Wu45:51

       Yeah.

    4. MCMartin Casado45:51

       So maybe you can talk a bit about that.

    5. SWSherwin Wu45:52

       Yeah. Yeah. I think, um... So at, at Dev Day this year when we launched our agent builder, I got a bunch of questions around this because-

    6. MCMartin Casado45:59

       Yeah, that's what I was gonna figure about exactly. Yeah, yeah, yeah

    7. SWSherwin Wu46:00

       ... the agent builder is like, yeah, is like the bunch of different nodes-

    8. MCMartin Casado46:01

       It's different. Totally

    9. SWSherwin Wu46:01

       ... and it's like a deterministic thing.

    10. MCMartin Casado46:03

        Yeah.

    11. SWSherwin Wu46:03

        And everyone's like, "Oh, is this really like the future, future of agents?" And, um, we o-obviously put a lot of thought into this when we were thinking about building that product, but the way I think about it is-

    12. MCMartin Casado46:10

        Do you think, do you think they came from a point of being constrained, by the way? They're like, "Oh, this is too constraining," and like...

    13. SWSherwin Wu46:15

        Yeah. I think people are like, "It, it's too constraining. It's not like AGI forward."

    14. MCMartin Casado46:18

        No, that's right. That, yeah

    15. SWSherwin Wu46:18

        ... you know, like at the end of the... Again, at the, at the end of the day, the AGI will be able to do everything.

    16. MCMartin Casado46:21

        [laughs]

    17. SWSherwin Wu46:21

        And so like why not-

    18. MCMartin Casado46:22

        So why, why, why-

    19. SWSherwin Wu46:22

        Why have nodes in this like node builder thing?

    20. MCMartin Casado46:24

        ... make nodes do everything? Just tell it what to do. [laughs]

    21. SWSherwin Wu46:25

        Um, yeah. And so I think there's like two things at play here. One of them is like there is a, like practicality component.

    22. MCMartin Casado46:31

        Yeah.

    23. SWSherwin Wu46:31

        And then the other thing is I think there are actually like different types of work that exist out there that could be automated into agents.

    24. MCMartin Casado46:36

        Okay.

    25. SWSherwin Wu46:36

        And so on the practicality side is, yeah, like the models today just like maybe in some future world, instruction following would be so good that you just like ask it to do this four-step process and it like always does the four-step process exactly.

    26. MCMartin Casado46:48

        Yeah.

    27. SWSherwin Wu46:48

        Um, we're still not there yet, and in the meantime, you know, this entire industry being born and a lot of, you know, people still wanna use these models, so like what, what can you build for them? So there's a practicality component of it.

    28. MCMartin Casado46:58

        When, when, when did you launch that?

    29. SWSherwin Wu47:00

        Uh, Dev Day. So [laughs] it feels like forever ago. Uh, earlier this month, uh, October, uh-

    30. MCMartin Casado47:06

        [laughs] It feels so funny

Episode duration: 53:22