Inside OpenAI Enterprise: Forward Deployed Engineering, GPT-5, and More | BG2 Guest Interview

Sherwin Wu 0:00 We literally had to bring the weights of the, the model, uh, physically into their supercomputer.

Apoorv Agrawal 0:04 In San Francisco, you could take a car from one part of SF to the other fully autonomously. As opposed to the digital world, I can't book a ticket online right now. Physical autonomy is ahead of digital autonomy in 2025.

Sherwin Wu 0:16 I think AI agents are, like, really in day one here.

Apoorv Agrawal 0:18 Mm.

Sherwin Wu 0:18 Like, ChatGPT only came out in 2022, and the slope, I think, is incredibly steep. I actually do think self-driving cars have s- a good amount of scaffolding in the world. You have roads. Roads exist. They're pretty standardized.

Apoorv Agrawal 0:29 Yeah.

Sherwin Wu 0:29 You have stoplights. AI agents are just kind of dropped in the middle of nowhere.

Apoorv Agrawal 0:33 We'll start with long-short game.

Sherwin Wu 0:35 I'm short on, uh, the entire category of, like, tooling evals products.

Apoorv Agrawal 0:40 Healthcare is probably the industry that will benefit the most from AI. I think I'm AGIPL.

Sherwin Wu 0:46 You're definitely AGIPL.

Apoorv Agrawal 0:47 The first one was the realization in 2023 that I would never need to code manually, like, ever, ever again.

Sherwin Wu 0:53 [laughs]

Apoorv Agrawal 1:01 Hey, folks. I'm Apoorv Agrawal, and today at the OpenAI office, we had a wide-ranging conversation about OpenAI's work in enterprise. I have with me the head of engineering and head of product of the OpenAI platform, Sherwin Wu and Olivier Godement. OpenAI is well known as the creator of ChatGPT, which is a product that billions across the world have come to love and enjoy. But today, we dive into the other side of the business, which is OpenAI's work in enterprise. We go deep into their work with specific customers and how OpenAI is transforming large and important industries like healthcare, telecommunications, and national security research. We also talk about Sherwin and Olivier's outlook on the next-- what's next in AI, what's next in technology, and their picks both on the long and short side. This was a lot of fun to do. I hope you really enjoy it. Well, two world-class builders, two people who make look building easy. Sherwin,

Apoorv Agrawal 1:50 uh, my Palantir 2013 classmate-

Sherwin Wu 1:52 Yeah

Apoorv Agrawal 1:52 ... tennis buddy, with two stops at Quora and Opendoor through the IPO before joining OpenAI. Before ChatGPT, you've now been here for three years and lead engineering for all OpenAI platform. Olivier, former entrepreneur, winner of the Golden Llama-

Sherwin Wu 2:08 [laughs]

Apoorv Agrawal 2:09 ... at Stripe, where you were for just under a decade, and now lead all of the product at, uh, OpenAI Platform.

Sherwin Wu 2:15 That's right.

Apoorv Agrawal 2:16 Thanks for doing it.

Sherwin Wu 2:17 Thank you.

Apoorv Agrawal 2:17 Thanks for having us. You know, as a shareholder, as a thought partner, kicking ideas back and forth, I always learn a lot from you guys, and so it's a treat. It's a real treat to be do this for everybody. You know, I'll open with people know OpenAI as the firm that built ChatGPT-

Sherwin Wu 2:33 Mm-hmm

Apoorv Agrawal 2:33 ... the product that they have in their pocket that comes with them every day to work, to personal lives. But the focus for today is OpenAI for Enterprise. You guys lead OpenAI Platform. Tell us about it. What's underneath the OpenAI Platform for B2B, for enterprise?

Sherwin Wu 2:49 Yeah. So this is actually a really interesting question too because I, I said when I joined OpenAI, uh, around three years ago to work on the API, it was actually the only product that we had.

Apoorv Agrawal 2:58 Mm-hmm.

Sherwin Wu 2:58 So I think a lot of people actually forget this, where the original product for, uh, from OpenAI actually was not ChatGPT. It was a B2B product. It was, uh, the API. We were catering towards developers.

Apoorv Agrawal 3:07 Mm-hmm.

Sherwin Wu 3:07 Um, and so I've actually seen, you know, the launch of ChatGPT and, and all of, you know, everything downstream from that. Um, but at its core, I actually think the, the reason why we have a platform and why we started with an API is, uh, kind of comes back to the OpenAI mission. So our, our mission obviously is to build AGI, uh, which is pretty hard, uh, in and of itself, but also to distribute the benefits of it to everyone, uh, in the world, uh, to all of humanity. Um, and you know, it's pretty clear right now to see ChatGPT doing that because, you know-

Apoorv Agrawal 3:34 Mm-hmm

Sherwin Wu 3:34 ... my mom, you know, maybe even your parents-

Apoorv Agrawal 3:36 Mm-hmm

Sherwin Wu 3:36 ... are using ChatGPT. But, um, we actually view our, uh, platform and especially-

Apoorv Agrawal 3:41 Mm

Sherwin Wu 3:41 ... our API and how we work with our customers-

Apoorv Agrawal 3:43 Mm

Sherwin Wu 3:43 ... our enterprise customers, as our way of getting the benefits of AGI, of, of AI-

Apoorv Agrawal 3:47 Mm

Sherwin Wu 3:47 ... to as many people as possible, to everyone in every corner of the world.

Apoorv Agrawal 3:50 Mm-hmm.

Sherwin Wu 3:51 Uh, ChatGPT obviously is really, really, really big now. It's, I think, like the fifth largest website in the world.

Apoorv Agrawal 3:55 Mm-hmm.

Sherwin Wu 3:56 Um, but we actually, by working through developers using our API, we're actually able to reach even more, uh, people in, you know, every corner of the world and every different use case that you might have. Uh, and especially with some of our enterprise customers, we're able to reach even, uh, use cases within businesses-

Apoorv Agrawal 4:10 Mm

Sherwin Wu 4:10 ... and reach, uh, end users of those businesses as well. And so we actually view the, the platform as kind of our way of fully expressing our, our mission-

Apoorv Agrawal 4:18 Mm-hmm, mm-hmm

Sherwin Wu 4:18 ... uh, of getting the, the benefits of AGI to, to, to everyone. And so, um, concretely though, what the platform actually includes today-

Apoorv Agrawal 4:25 Mm-hmm

Sherwin Wu 4:25 ... um, the biggest product that we have is, is obviously, um, our developer platform, which is our, our API.

Apoorv Agrawal 4:30 Mm.

Sherwin Wu 4:30 Um, you know, many developers, the, you know, the majority of, of the startup ecosystem builds on top of this-

Apoorv Agrawal 4:35 Mm-hmm

Sherwin Wu 4:35 ... as well as, uh, a lot of start, uh, digital natives, uh, Fortune 500 enterprises at this point.

Apoorv Agrawal 4:40 Mm-hmm.

Sherwin Wu 4:41 Um, we also have a product that we sell to governments, um, as well in the public sector.

Apoorv Agrawal 4:44 Mm.

Sherwin Wu 4:44 So that's all part of this as well. Um, and also an emerging product line for us in the platform is our, uh, enterprise products-

Apoorv Agrawal 4:50 Mm

Sherwin Wu 4:50 ... so what we actually might sell directly to, uh, enterprises, uh, beyond just a, a core API offering.

Apoorv Agrawal 4:56 Fascinating.

Sherwin Wu 4:56 And maybe to, to, to double down, like, um, I think B2B is actually quite core to the OpenAI mission.

Apoorv Agrawal 5:03 Mm.

Sherwin Wu 5:04 What we mean by distributing AGI benefits is, you know, I wanna live in a world where, you know, there are like 10x more medicines like, you know, going out every year.

Apoorv Agrawal 5:13 Mm-hmm.

Sherwin Wu 5:13 I wanna live in a world where, you know, education like, you know, public service, civil service, you know, are like, you know, increasingly like optimized, you know, to everyone. And you know, there are like a large like category of use cases that only go through B2B, frankly-

Apoorv Agrawal 5:27 Right

Sherwin Wu 5:27 ... unless you enable like the enterprises.

Apoorv Agrawal 5:29 Right.

Sherwin Wu 5:29 And you know, we talked about Palantir. I think that's probably the same fees at Palantir.

Apoorv Agrawal 5:32 Yeah.

Sherwin Wu 5:32 It's like, hey, like those are the businesses who are actually like, you know, making stuff happen in the real world. And so if you do enable them, if you do accelerate them-

Apoorv Agrawal 5:39 Mm-hmm

Sherwin Wu 5:40 ... like that's how essentially you benefit, you know, distribute AGI.

Apoorv Agrawal 5:43 Yeah. Well, maybe we can double-click into that, Olivier. You know, the reach for chat is obviously wide.

Sherwin Wu 5:50 Yep.

Apoorv Agrawal 5:50 Billions of users. But for enterprise, it's-- Maybe tell us about it. Maybe we go deep into a customer example or two, and what is an organization that we have helped-

Olivier Godement 6:01 Uh, transform maybe, uh, and at what layers. So if I were to step back, like, we started our B2B efforts with the API, like, a few years ago.

Apoorv Agrawal 6:08 Mm.

Olivier Godement 6:09 Initially, the customers were startups, developers-

Apoorv Agrawal 6:12 Mm

Olivier Godement 6:12 ... indie hackers, extremely technically sophisticated people, like, you know-

Apoorv Agrawal 6:17 Mm

Olivier Godement 6:17 ... who are building, like, you know, cool new stuff essentially-

Apoorv Agrawal 6:19 Mm

Olivier Godement 6:19 ... and taking massive, like, you know, market and technical risk. Um, so we still have a bunch of customers in that category-

Apoorv Agrawal 6:25 Mm

Olivier Godement 6:25 ... and we love them, um, and we keep building with them.

Apoorv Agrawal 6:28 Mm.

Olivier Godement 6:28 On top of that, you know, over the past couple of years, we've been working one more with traditional, like, enterprises, and also, like, digital natives.

Apoorv Agrawal 6:34 Mm.

Olivier Godement 6:35 Um, essentially, I think basically everyone woke up, like, with ChatGPT and, like, um, those models are working. There is a ton of value, and they could see essentially many use cases in enterprise. Um, couple of examples which I like the most, um, one which is very both fresh and, you know, is quite cool, um, we've been working a lot with T-Mobile.

Apoorv Agrawal 6:53 T-Mobile.

Olivier Godement 6:54 Um, so T-Mobile, leading, like, US, uh, telco operator.

Apoorv Agrawal 6:57 Right.

Olivier Godement 6:58 Um, T-Mobile has, like, you know, a massive customer support load.

Apoorv Agrawal 7:01 Mm.

Olivier Godement 7:01 Like, you know, people asking, like, you know, "Hey, I was charged-"

Apoorv Agrawal 7:04 Yeah

Olivier Godement 7:04 ... "like, that amount of money. What's going on?"

Apoorv Agrawal 7:05 Yeah.

Olivier Godement 7:05 Or, you know, "My, my cell phone, like, is not working anymore." Um, a massive, like, you know, um, share of that load is, like, you know, voice calls. Like, people-

Apoorv Agrawal 7:13 Mm

Olivier Godement 7:13 ... wanna talk to someone. Um, and so for them, like, you know, to be able to essentially automate, like, more and more and, you know, to help, like, people, like, self-serve in a way-

Apoorv Agrawal 7:20 Mm

Olivier Godement 7:21 ... like, you know, um, debug their, their subscription was pretty big.

Apoorv Agrawal 7:24 Mm.

Olivier Godement 7:24 And so we've been working with T-Mobile pretty much for the past year at that point-

Apoorv Agrawal 7:28 Mm

Olivier Godement 7:28 ... to basically automate, like, not only, like, text support, but also voice support.

Apoorv Agrawal 7:32 I see.

Olivier Godement 7:32 And so today, like, you know, there are, like, features, like, in the T-Mobile app that if you call are actually handled by OpenAI, you know, models behind the scenes. And, you know, it does sound, like, super natural, like, you know, human-sounding-

Apoorv Agrawal 7:43 Yeah

Olivier Godement 7:43 ... latency quality-wise. So that one was, um, really fun. A second one, uh, which is very-

Apoorv Agrawal 7:48 Olivier, just on that-

Olivier Godement 7:49 Yeah

Apoorv Agrawal 7:49 ... can I, can I ask you a follow-up question?

Olivier Godement 7:51 Yeah.

Apoorv Agrawal 7:51 So we've got text models. We've got voice models, maybe even video models someday-

Olivier Godement 7:55 Yeah

Apoorv Agrawal 7:55 ... that are deployed at T-Mobile.

Olivier Godement 7:57 Yeah.

Apoorv Agrawal 7:57 But what above the models or, or, or, or adjacent to the models might we have helped T-Mobile with, for example?

Olivier Godement 8:03 Yeah. There is a ton we are doing. Um, the first one is, you know, you have to put yourself in the shoes of an enterprise buyer.

Apoorv Agrawal 8:09 Mm.

Olivier Godement 8:09 Like, their goal is to automate, you know, or reduce, like, you know, optimize customer support. And, you know, going from, like, a model, like, tokens in, tokens out-

Apoorv Agrawal 8:17 Right. Right, right

Olivier Godement 8:17 ... to that use case, it's hard.

Apoorv Agrawal 8:19 Yeah.

Olivier Godement 8:19 And so, you know, first-

Apoorv Agrawal 8:20 Yeah, yeah

Olivier Godement 8:20 ... like, there's a lot of design, like, you know, system design.

Apoorv Agrawal 8:22 Mm.

Olivier Godement 8:23 Um, we do have actually now forward-deployed engineers who are helping us quite a bit. So actually-

Apoorv Agrawal 8:26 Forward-deployed engineers.

Olivier Godement 8:27 Yeah, I mean-

Apoorv Agrawal 8:28 Well, that's familiar to the-

Sherwin Wu 8:29 Borrow the term from Palantir.

Apoorv Agrawal 8:30 [laughs]

Sherwin Wu 8:30 Um, yeah, it's a great term.

Olivier Godement 8:31 Were you FDEs at, uh, Palantir?

Sherwin Wu 8:33 I was not an FD. I was on... I think they called it the dev side, right-

Olivier Godement 8:36 Right

Sherwin Wu 8:36 ... just, like, software engineering.

Olivier Godement 8:37 Right, right, right.

Sherwin Wu 8:37 Uh, I was al- also only an intern at, at Palantir. Um, but yeah, it's a great term. I think it ac- accurately describes what we're asking folks to do, which is, like, embed very deeply with customers-

Olivier Godement 8:45 Yeah, yeah

Sherwin Wu 8:46 ... and, and honestly, like, build things specific to their systems-

Olivier Godement 8:48 Yeah

Sherwin Wu 8:48 ... that are deployed, uh-

Olivier Godement 8:50 Yeah

Sherwin Wu 8:50 ... onto these customers. Um, but yeah, we, we are obviously growing and hiring that team quite a bit 'cause they've been very effective-

Olivier Godement 8:54 Yeah

Sherwin Wu 8:55 ... like, at T-Mobile.

Olivier Godement 8:55 Yeah.

Sherwin Wu 8:55 Yeah.

Olivier Godement 8:55 Four years of my life.

Sherwin Wu 8:57 Yeah.

Apoorv Agrawal 8:57 Oh, yeah.

Olivier Godement 8:58 Yeah, forward deployed.

Apoorv Agrawal 8:59 Yeah.

Olivier Godement 8:59 But, but go ahead. So forward deployed engineering or-

Sherwin Wu 9:01 Forward deployed engineers and the sort of, like, systems and, like, integrations they're doing is, you know, first, like, you know, you have to orchestrate those models. Like, those models are not just, you know... Those models, like, know nothing about, like, you know, the CRM, like, you know, and, like, what's going on, and so you have to plug the model to, like, many, many different internal tools. Many of those, like, tools, like, in the enterprise do not even have, like, APIs or, like, clean interfaces, right?

Olivier Godement 9:23 Mm.

Sherwin Wu 9:23 It's the first time they're being exposed, like, you know, to a third-party, uh, system. And so there is a lot of, you know, standing up, like, you know, API gateways, like, tools, connecting. Um, then you have to essentially, like, define what good looks like.

Olivier Godement 9:35 Mm.

Sherwin Wu 9:35 You know? Uh, again, like, it's a pretty new exercise for everyone. Like, you know, defining, like, a golden set of evals-

Olivier Godement 9:41 Mm

Sherwin Wu 9:41 ... is, you know, easier than it sound.

Olivier Godement 9:43 Mm.

Sherwin Wu 9:43 Um, harder than it sounds.

Olivier Godement 9:45 Yeah.

Sherwin Wu 9:45 Um, and so we have been spending, like, a bunch of time with them.

Olivier Godement 9:48 Evals are important.

Sherwin Wu 9:49 Evals are super important.

Apoorv Agrawal 9:49 Especially, like, audio evals. I know audio evals are, like, extra hard to grade-

Olivier Godement 9:53 Yeah

Apoorv Agrawal 9:53 ... and, and get right and-

Olivier Godement 9:54 Right

Apoorv Agrawal 9:54 ... but, like, the bulk of the use case here is actually audio-

Olivier Godement 9:56 Right, right

Apoorv Agrawal 9:56 ... and when you have, like, I don't know, a five-minute, like, call transfer, how do you actually know that the right thing happened? Uh, it's a pretty tough problem.

Olivier Godement 10:02 Yeah, it's pretty tough.

Apoorv Agrawal 10:03 Mm.

Olivier Godement 10:04 Um, and then, you know, actually nailing down, like, the quality of the customer experience, like, you know, until it feels natural, and here latency and interruptions-

Apoorv Agrawal 10:14 Mm

Olivier Godement 10:14 ... play a really, like, you know, important part. We shipped, um, in GA, um, an API, Realtime API.

Apoorv Agrawal 10:20 Mm.

Olivier Godement 10:20 I think it was last week.

Apoorv Agrawal 10:21 That's right. Couple of weeks ago, yeah.

Olivier Godement 10:22 Uh, yeah.

Apoorv Agrawal 10:23 It was just last week, I think.

Olivier Godement 10:24 Um, which is, like, a beautiful work of engineering. Uh, you know, that was a really cracked team behind the scenes-

Apoorv Agrawal 10:28 Yeah, yeah

Olivier Godement 10:29 ... um, which basically allows us, like, to get, like, the most, like, natural sounding, like, you know, voice experience-

Apoorv Agrawal 10:35 Mm

Olivier Godement 10:35 ... without having, like, these weird interruptions or, you know, lag where you can feel that essentially the thing is off.

Apoorv Agrawal 10:40 Mm-hmm.

Olivier Godement 10:40 Um, so yeah, cobbling all that together, you know, and you get, like, you know-

Apoorv Agrawal 10:45 Yeah

Olivier Godement 10:45 ... a really good experience.

Apoorv Agrawal 10:46 Yeah, that's a lot more than just models.

Olivier Godement 10:48 Yeah.

Sherwin Wu 10:48 Yeah, I was gonna say, um, uh, one actually really great thing that I think we've gotten from the T-Mobile experience is actually working with them to improve our models themselves. So for example-

Olivier Godement 10:56 Mm

Sherwin Wu 10:56 ... the last, uh, real time, uh, the Realtime GA last, last week, we obviously released a new snapshot-

Olivier Godement 11:01 Mm-hmm

Sherwin Wu 11:01 ... of the GA snapshot.

Olivier Godement 11:02 Mm.

Sherwin Wu 11:02 Uh, and a lot of what, the improvements that we actually got into the model came out of, you know, the learnings that we had from T-Mobile.

Olivier Godement 11:07 Mm.

Sherwin Wu 11:07 It brings in a lot of other change from other customers-

Olivier Godement 11:09 Mm

Sherwin Wu 11:10 ... but because we were so deeply embedded into T-Mobile and, and-

Olivier Godement 11:12 Mm

Sherwin Wu 11:12 ... we were able to understand what good looks like for them-

Olivier Godement 11:14 Mm

Sherwin Wu 11:14 ... we were able to bring that to some of our models.

Apoorv Agrawal 11:16 That makes sense. So this is a large customer with tens of millions of users, if not hundreds of millions, and the before and after is on the support side, both tech support internally and then their customer support.

Olivier Godement 11:26 Yeah.

Apoorv Agrawal 11:27 Makes sense.

Olivier Godement 11:27 Yeah.

Apoorv Agrawal 11:28 Is there another one that you guys can share?

Olivier Godement 11:30 I like a lot, uh, Amgen. Um-

Apoorv Agrawal 11:33 Amgen, the healthcare business.

Olivier Godement 11:34 Amgen, yeah.

Apoorv Agrawal 11:35 Mm.

Olivier Godement 11:35 So we're working quite a bit with, uh, healthcare companies.

Apoorv Agrawal 11:39 Mm-hmm.

Olivier Godement 11:39 Amgen is one of the leading, like, healthcare companies who specialized into drugs for, um, uh, cancer or, like, you know, inflammatory diseases. Uh, they're based out of LA, out of LA. Um, and we've been working essentially with Amgen to essentially speed up, like, the drug, like, development and, like-

Apoorv Agrawal 11:55 Wow

Olivier Godement 11:55 ... consolidation process. So, you know, the, the sort of the, the North Star is, like, pretty bold. Um, and it's really interesting, like, when you- Similarly, like, you know, we embedded, like, pretty deeply with Amgen to understand what are their needs.

Apoorv Agrawal 12:07 Mm.

Olivier Godement 12:08 And it's really interesting, like when I look at those healthcare companies, I feel like there are two big buckets of needs.

Apoorv Agrawal 12:13 Mm.

Olivier Godement 12:14 One is like pure R&D. It's like, you know, you're seeing, like, a massive amount of data, and like you have super smart-

Apoorv Agrawal 12:19 Mm

Olivier Godement 12:19 ... scientists who are trying to, you know, combine, test out things, you know. So that's one bucket. A second bucket is like, you know, much more like, you know, common across other industries. It's like pure like, you know, admin, document authoring, documents reviewing work, which is, you know, by the time, like, your R&D team has essentially locked the recipe-

Apoorv Agrawal 12:37 Mm

Olivier Godement 12:37 ... of a medication, getting that medication to market-

Apoorv Agrawal 12:41 Mm

Olivier Godement 12:41 ... is a ton of work.

Apoorv Agrawal 12:43 Mm.

Olivier Godement 12:43 Like, you have to submit to, like, various regulatory bodies, get a ton of reviews.

Apoorv Agrawal 12:47 Mm.

Olivier Godement 12:47 And you know, when we looked at essentially those problems, what we knew, what, what models were capable of, we saw like, you know, a ton of benefits-

Apoorv Agrawal 12:54 Mm

Olivier Godement 12:54 ... a ton of, like, opportunities to automate and, you know, augment essentially the work of those teams. Um, and so yeah, Amgen has been like a top customer of GPT-5, for instance.

Apoorv Agrawal 13:02 Wow. I mean, this could be hundreds of millions of lives if a new drug is developed faster.

Olivier Godement 13:08 Yeah, exactly.

Apoorv Agrawal 13:09 Huge impact.

Olivier Godement 13:10 So that's, you know... That's, I think, one good example of like a kind of impact on which you need to enable enterprises like to do it-

Apoorv Agrawal 13:17 Right

Olivier Godement 13:17 ... you know. Uh, and so I think we're going to do more and more of those. Uh, and yeah, frankly, like, you know, on a personal level, like it's a delight, you know?

Apoorv Agrawal 13:24 Yeah.

Olivier Godement 13:24 If I can play like, you know, a tiny role essentially-

Apoorv Agrawal 13:26 Yeah

Olivier Godement 13:26 ... like doubling like, you know, the kind of, um, medication that people, you know, get in the real world, that feels like, you know, a pretty good like, you know, achievement.

Apoorv Agrawal 13:32 Huge. Huge, huge. I know you had one as well.

Sherwin Wu 13:35 Yeah. So one of my favorite deployments that we've done, uh, more recently actually is with the, uh, Los Alamos National Labs. So this is the, the, the, like, government national research lab-

Apoorv Agrawal 13:44 Mm

Sherwin Wu 13:44 ... that,

Sherwin Wu 13:45 uh, the US government is running in Los Alamos, New Mexico.

Apoorv Agrawal 13:47 Mm-hmm, mm-hmm.

Sherwin Wu 13:48 It's also where, you know, the Manhattan Project happened-

Apoorv Agrawal 13:50 Yeah

Sherwin Wu 13:50 ... uh, back in the '40s and '50s-

Apoorv Agrawal 13:52 Yeah

Sherwin Wu 13:52 ... um, back when it was a secret, secret project. So, you know, after that, they ended up formalizing it as a city and a program, and then now it's a, a pretty sizable national-

Apoorv Agrawal 13:59 Mm

Sherwin Wu 13:59 ... uh, laboratory. Um, this one is, is very interesting because one, just the depth of impact here is-

Apoorv Agrawal 14:04 Oh

Sherwin Wu 14:04 ... is, like, unimaginable. For me, it's like on the scale of, of, of Amgen and, and some-

Apoorv Agrawal 14:08 Interesting

Sherwin Wu 14:08 ... of these other, uh, uh, larger companies. But, you know, obviously, they're doing a lot of actual n-new research there, so a lot of new science. They're doing a lot of stuff with, uh, our, uh, defense department, uh, and defense use cases as well.

Apoorv Agrawal 14:20 Wow.

Sherwin Wu 14:20 So very intense, you know-

Apoorv Agrawal 14:21 Yeah

Sherwin Wu 14:21 ... very intense stuff. Um, but the other thing that's ac-actually very interesting about this one was that it-it's, it's a, it's also a, a story of a very, like, bespoke and, like, new type of, uh, deployment that we've done.

Apoorv Agrawal 14:31 Yeah.

Sherwin Wu 14:31 Um, so because they are so... They're a government lab, they're so, you know, uh, restrictive and high security and high clearance with a lot of their things, uh, we couldn't just do a normal deployment with them.

Apoorv Agrawal 14:40 Mm.

Sherwin Wu 14:40 They couldn't... You know, you can't have people doing national security research-

Apoorv Agrawal 14:43 Yeah

Sherwin Wu 14:43 ... just hitting our, uh, our APIs. And so, um, we actually did a custom on-prem deployment with them-

Apoorv Agrawal 14:49 Mm-hmm

Sherwin Wu 14:49 ... onto one of their supercomputers called, uh, Venado.

Apoorv Agrawal 14:52 Mm.

Sherwin Wu 14:52 Um, and so this actually involves a bunch of, you know, uh, very bespoke work with some FDEs, uh, also with a lot of our developer team-

Apoorv Agrawal 14:58 Nice

Sherwin Wu 14:59 ... um, to actually bring one of our reasoning models, o3, um, into their laboratory, into an air-gapped, you know, supercomputer Venado-

Apoorv Agrawal 15:06 Yeah

Sherwin Wu 15:06 ... and actually deploy it, uh, and get it installed to work on their hardware, on their networking stack-

Apoorv Agrawal 15:11 Mm

Sherwin Wu 15:11 ... um, uh, and, uh, and actually run it in this, in this particular environment.

Apoorv Agrawal 15:15 Mm.

Sherwin Wu 15:15 And so it was, it was actually very interesting because we literally had to bring the weights of the, the model, uh, physically into their, uh-

Apoorv Agrawal 15:21 A briefcase

Sherwin Wu 15:21 ... into their supercomputer, um, in an environment, by the way, where you're not allowed to have... You know, it's very, it's very locked down for a good reason.

Apoorv Agrawal 15:28 Yeah.

Sherwin Wu 15:28 Um, you're not allowed to have, like, cell phones or, like, any-

Apoorv Agrawal 15:30 Yeah

Sherwin Wu 15:30 ... electronics with you, um, as well. And so that, I think that was a very, very unique challenge. Um-

Apoorv Agrawal 15:35 Yeah

Sherwin Wu 15:35 ... and then, uh, the other interesting thing about this deployment is just how it's being used, right?

Apoorv Agrawal 15:38 Mm.

Sherwin Wu 15:38 So, um, the interesting thing is because it's so, uh, locked down and on-prem, we actually do not have much visibility into exactly what they're-

Apoorv Agrawal 15:45 Mm-hmm

Sherwin Wu 15:45 ... what they're doing with it, but we do have-

Apoorv Agrawal 15:46 Right

Sherwin Wu 15:47 ... you know, they give us feedback.

Apoorv Agrawal 15:48 No telemetry. Yeah.

Sherwin Wu 15:48 Yeah, yeah. Um, they actually do have some telemetry, but it's, you know, within their own-

Apoorv Agrawal 15:52 Okay

Sherwin Wu 15:52 ... their own systems.

Apoorv Agrawal 15:53 Yeah.

Sherwin Wu 15:53 But we do know that it's, it's, it's, you know, being used for, uh, a bunch of different things. It's being used for, um, aiding them in, in terms of speeding up their experiments. They have a lot of data analysis use cases-

Apoorv Agrawal 16:02 Mm

Sherwin Wu 16:03 ... a lot of notebooks that they're running with reams of data that they're trying to process.

Apoorv Agrawal 16:06 Mm.

Sherwin Wu 16:06 Um, they're actually using it as a thought partner, which is something that's pretty interesting to me. Um, o3 is like-

Apoorv Agrawal 16:10 Interesting

Sherwin Wu 16:11 ... pretty smart, uh, as a-

Apoorv Agrawal 16:12 Yeah

Sherwin Wu 16:13 ... as a model, and a lot of these people are tackling really tough, you know, novel research problems.

Apoorv Agrawal 16:16 Mm.

Sherwin Wu 16:16 And a lot of times they're kind of using o3, um, and, uh, going back and forth with it on their experiment design-

Apoorv Agrawal 16:22 Mm

Sherwin Wu 16:22 ... on, like, what they actually should be using it for-

Apoorv Agrawal 16:24 Mm

Sherwin Wu 16:24 ... um, uh, which is, you know, something that we couldn't really say about our, our-

Apoorv Agrawal 16:28 Yeah

Sherwin Wu 16:28 ... our older models. Um, and so yeah, it's, it's just being used by, by, um, for a lot of different use cases for the, for the-

Apoorv Agrawal 16:34 Yeah

Sherwin Wu 16:34 ... national lab. And, um, the other cool thing is it's actually being shared, uh, between Los Alamos and some of the other labs, uh, Lawrence Livermore, uh, Sandia-

Apoorv Agrawal 16:42 Mm

Sherwin Wu 16:42 ... uh, as well, uh, because it, it's the supercomputer setup where they can all kind of, uh, connect-

Apoorv Agrawal 16:45 Right

Sherwin Wu 16:45 ... with it remotely.

Apoorv Agrawal 16:46 Fascinating. I mean, we've just gone through three pretty large scale enterprise deployments, right? Which might done, uh, touch tens, if not hundreds of millions of people. But there's this... On the other side of this is the MIT report that

Apoorv Agrawal 17:00 came out a couple of weeks ago. [laughs] 95% of AI deployments don't work. A bunch of, you know, scary headlines that even shook the markets for a couple of days. Like, you know, put this in perspective, like, like, for every deployment that works, there is presumably a bunch that don't work. So maybe we can, you know, may-maybe talk about that. Like, what does it take to build a successful enterprise deployment, a successful customer deployment, um, and, and, and the counterfactual, uh, based on all your experience serving all these large enterprises?

Olivier Godement 17:29 I think at that point, I may have worked with like a couple of hundreds, I think, enterprises. [laughs]

Apoorv Agrawal 17:35 A couple of hundreds? Yeah.

Olivier Godement 17:35 So, okay, I'm going to pattern match, um, what I've seen being like clear leading indicator of success. Uh, number one is like the, the interesting combination of like top-down, like, buy-in and like enabling-

Apoorv Agrawal 17:48 Mm

Olivier Godement 17:48 ... like, you know, very clear group of like a tiger team essentially, like, you know, at the enterprise, which is sometime a mix of like OpenAI, like, you know, enterprise employee.

Apoorv Agrawal 17:56 Mm.

Olivier Godement 17:56 So, you know, typically like, you know, you take like T-Mobile, like the top leadership was like extremely bought in-

Apoorv Agrawal 18:01 Right

Olivier Godement 18:01 ... like, you know, it's a priority.

Apoorv Agrawal 18:02 Right. Right.

Olivier Godement 18:02 But then letting the team like, you know, organize and be like, okay-

Apoorv Agrawal 18:04 Right

Olivier Godement 18:05 ... if you want to start small, start small.

Apoorv Agrawal 18:07 Right.

Olivier Godement 18:07 You know, and then we can scale it up essentially. So that would be part number one.

Apoorv Agrawal 18:11 So top-down buy-in.

Sherwin Wu 18:12 And a bottom call a tiger team. Tiger team, you know, of people like, you know, a mix of like technical skills and like people- Mm ... who just have like the organizational knowledge, like- Yeah ... institutional knowledge, you know? Yeah, yeah. It's really funny, like in the enterprise, like customer support is a good example, like what we found is that the vast majority of the knowledge- Mm ... is in people's heads. Right. Right? [laughs] Which is probably like a thing that- Yeah ... you know, with FD's like in general, but like, you know, you take a customer support, you would think that, you know, everything is like perfectly documented like in a- Yeah ... GI, et cetera. The reality is like the, the standard like operating procedures, like the SOPs- Mm ... are largely in people's head. Oh. And so unless you have that tiger team, like mix of like technical and like, you know, subject matter expert- Mm ... really hard, like to get something off the ground. That would be one. Um, two would be evals first. Evals. Like whenever we define the good evals, like that gives like a clearly clear common goal- Mm ... for people to hit. Mm. Whenever like, you know, the customer like fails to come up with good evals- Mm ... it's a moving target. You never know ex- Yeah ... essentially, you know, if you've made it or not. Um, and you know, evals are much harder than what it looks, uh- Mm ... to get done. And evals also oftentimes need to come up bottom up, right? Mm. Because all of these things are kind of in people's heads, in the actual operators' heads. Like it's actually very hard to have a top-down mandate of like- Mm ... you got like this is how the evals should look. Mm. A lot of it needs the bottom-up adoption. Right. Yeah, yeah, yeah. And so we've been building quite a bit of tooling on evals. We have like an evals product- Mm ... and you know, we are working on more to essentially solve like, you know, that problem or, you know, make it easy as we can. Um, the last thing is, you know, you want to hill climb, essentially. You have your evals. The goal is to get to 99%. Yeah. You start at like, you know, 46. Uh, you know- Yeah ... how do you get there? And here, frankly, I think oftentimes like, you know, a mix of like, like I will say like almost wisdom from people who've done it before. Yeah. Like, you know, a lot of that is like, you know, like art sometime more than science. Yeah, yeah, yeah. And like, you know, knowing like the quirks of the model, the behavior. Mm-hmm. Sometime we even need to fine-tune ourselves the models, you know, when there are some clear limitation. Mm. Uh, and you know, being patient, getting your way, you know, up there- Mm ... and then, you know, ship. Can we go under the hood a little bit? You know, one of the things that we think about a lot is autonomy more broadly, right? What is the makeup of autonomy? On the one side, you know, in San Francisco, you could take a car from one part of SF to the

Sherwin Wu 20:30 other fully autonomously. No humans involved, no... You press a button. Yeah, we love the Waymos. Right? Yeah. They've done billions of rides. I think it was like, what? Three and a half billion rides, um, on, on the... That's just on the Tesla FSD. I think Waymo's done like million, tens of millions of rides. That's a lot of autonomy. Yeah. In the physical world as opposed to the digital world, I can't book a ticket online right now. Mm-hmm. It's, there's all sorts of problems that happen if I have my operator try to book a ticket, and it's very counterintuitive because the bar for physical safety is so much higher. The bar for physical safety is higher than the human's capability because lives are at stake. Yeah. The bar for digital safety, not that high because all you're gonna lose is money. Nobody's life is at stake, but yet physical autonomy is ahead of digital autonomy in 2025, which seems counterintuitive. Like why is that the case at, you know, at a technical level? Why is it that what should sound easier is actually a lot harder? Yeah. So I, I think there are, there are kind of two things at play here, and I really like the analogy with, with self-driving cars- Mm-hmm ... 'cause you know, they, they, they've been, they've actually been like one of the best applications of AI, I think, that I've- Yeah ... I've used recently. Um, but I think there are two things in play. One of them is honestly just the timelines. Like we've been working on self-driving cars for so long. Longer. That's right. Like I remember when I, you know, back in like 2014, it was kind of like the, the advent of this, and everyone was like, "Oh, this is happening in like five years." Mm. Mm. It turns out it took like, I don't know, 10, 15 years or so for this to happen. That's right. So there's been a long time for this- Yeah ... technology to, to really mature, and I think- Yeah ... there's probably like dark ages, you know, back in like 2015- [laughs] ... or 2018 or something where it felt like- Yeah ... it wasn't gonna happen. The trough of disillusionment. Yes, yes. Yeah. And then now we're, you know, um- Yeah ... um, finally seeing it get deployed, um, which is really exciting, but it has been like, I don't know, 10 years, maybe even 20 years from the very beginning, uh, of the research. Mm-hmm, mm-hmm, mm-hmm. Uh, whereas, uh, I think AI agents are like really in day one here. Mm. Like ChatGPT only came out in 2022. Yeah. Uh, so like around three... Like less than three years ago. Yeah. Um, but I actually think the, uh, what we think about with, uh, AI agents and all that- Mm ... really I think started, uh, with the reasoning paradigm that- Mm ... uh, when we released the, the o1-preview model back in, uh, la-late last year, I think. Mm. Uh, and so I actually think this whole reasoning paradigm with AI agents and the robustness that those bring- Mm ... has only really unfolded for like a year. Mm. Uh, less than a year really. Mm, mm. And so, um, I, I know you had a chart in your blog post which I really like- Yeah ... which, you know, the, the slope is very meaningfully different now. Yeah, yeah, yeah. Like self-driving started very, very early. Yeah. Slope seems to be a little bit slower, but now it's reaching the promised land. Yeah. But man, like we, we started super recently with AI agents, and the slope I think is incredibly steep, and we'll probably see a crossover at some point. Yeah. Um, but uh, uh, uh, we really have only had like a year really to explore these, uh, these things. Do you think we haven't crossed over already when you look at like the coding work in particular? Yeah, it's, it's a, it's a good point. It's like, you know, um, your chart actually shows, uh, AI agents as below self-driving. Yeah. But like, you know, it's like what is the Y-axis? Yeah. Like by some measures, like I would not be surprised actually if, you know, AI products or AI agent products is making more revenue than Waymo at this point. Yeah. Like Waymo's making a lot- Mm ... but like just look at all the startups coming up. That's a good point. Look at, you know, ChatGPT and how many subscriptions are happening there and all of that. And so maybe we have actually crossed- Yeah ... and you know- Yeah, yeah ... a couple years from now it's gonna look very, very different. Yeah. The Y-axis is tangible felt autonomy. [laughs] Yeah. Yeah, yeah. Perfectly objective. [laughs] Right, right, right, right. How do I feel about- Exactly. Vibes more than, uh, revenue. [laughs] But revenue's a good one. We should probably redo that with revenue. The um... There's a second thing I wanted to, uh, mention on, on this as well, which is, uh, the scaffolding and the environment- Yeah ... in which these things operate in. So, um, I actually remember in the early days of self-driving, a lot of the like researchers around self-driving were saying that the roads themselves will have to like change to accommodate self-driving, right? There might be like sensors everywhere so that the self-driving cars can interact with it, um, which I think is like, you know, retrospect overkill. Yeah. Um, but I actually do think self-driving cars have s- a good amount of scaffolding in the world for them- Yeah ... to operate in. It's like not completely like unlimited. Yeah. Um, you have roads. Roads exist. They're pretty standardized. Yeah. You have stoplights. Yeah. People generally, uh, operate in like pretty normal ways, um, and there are all these traffic laws- Yeah ... that, that you can learn. Yeah, yeah. Um, whereas, um, uh, AI agents are just kind of dropped in the middle of nowhere- Mm ... and they kind of have to feel, feel around for them. Mm. And I actually think, you know, going off of what Olivier just said too, my hunch is some of the, um- Uh, enterprise deployments that don't actually work out likely don't have the scaffolding or-

Apoorv Agrawal 24:49 Mm

Sherwin Wu 24:49 ... infrastructure for these agents to interact with as well.

Apoorv Agrawal 24:52 Mm.

Sherwin Wu 24:52 Um, a lot of the, like, really successful deployments that we've made, a lot of what our FDs end up doing with some of these customers is to create, um, almost like a platform or some type of scaffolding, connectors, organizing the data-

Apoorv Agrawal 25:03 Mm

Sherwin Wu 25:03 ... so that the models have something that they can interact with in a more standardized way.

Apoorv Agrawal 25:06 Mm.

Sherwin Wu 25:07 And so my sense is self-driving cars actually have had this-

Apoorv Agrawal 25:09 Yeah

Sherwin Wu 25:09 ... uh, in, in, in some degree with, with roads over the last-

Apoorv Agrawal 25:12 Yeah

Sherwin Wu 25:12 ... uh, uh, you know, over the course of their deployment. But I actually think it's still very early in the AI agents', uh, space. And I would not be surprised if a lot of these, uh, uh, a lot of enterprises, a lot of companies just don't really have the scaffolding ready. So if you drop an AI agent in there-

Apoorv Agrawal 25:24 Yeah

Sherwin Wu 25:24 ... it kind of doesn't really know what to do.

Apoorv Agrawal 25:26 Yeah.

Sherwin Wu 25:26 Uh, and, and its impact will be limited. And so I think once the s- scaffolding gets built out, uh, across some of these companies, I think the deployment will also speed up.

Apoorv Agrawal 25:32 Mm.

Sherwin Wu 25:33 Um, but again, to our point earlier, I, I think there's no slowdown. Uh, there's no-

Apoorv Agrawal 25:36 Yeah

Sherwin Wu 25:36 ... you know, things are still moving very fast.

Apoorv Agrawal 25:38 That's great. Well, you know, it... I, I, I've thought about autonomy as a three-part structure. You've got perception, you've got the, the reasoning, the brain, and then you've got the, call it the scaffolding, the last mile of making things work. Uh, maybe we can dive into the second part, which is the reasoning, which is the, the, the juice that you guys are building with GPT-5 most recently.

Apoorv Agrawal 26:00 Huge endeavor. Congrats. The first time you guys have launched a full system, not a model or a set of models, but a full system. Talk about that. I mean, the full arc of that development, what was your focus? I mean, honestly, the benchmarks all seem so saturated. Like, clearly it was more than just benchmarks that you were focused on. And so what was the North Star like? Tell us about GPT-5, soup to nuts.

Olivier Godement 26:22 It's been the work of love of many people for a long time.

Apoorv Agrawal 26:26 Mm.

Olivier Godement 26:26 Um, and to your point, I think GPT-5 is amazingly intelligent. You look at the benchmark, like, you know, the SWE-bench and the likes, you know, it is going pretty high. But I think to me, equally important and impactful was, I would say, the craft, like the style, the tone, the behavior of the model.

Apoorv Agrawal 26:44 Mm.

Olivier Godement 26:44 So, you know, capabilities intelligence and, you know, behavior of the model.

Apoorv Agrawal 26:47 Mm.

Olivier Godement 26:47 Um, on the behavior of the model, I think it's the first model, like large model release, for which we have worked so closely with a bunch of customers for like month and month-

Apoorv Agrawal 26:59 Yeah

Olivier Godement 26:59 ... essentially to better understand, like what are the concrete like blocks? Like what are the concrete blockers of the model? And often, like, you know, it's not about like, you know, um, like having a model which is way more intelligent, a model which is faster, a model that better-

Apoorv Agrawal 27:12 Yeah

Olivier Godement 27:12 ... like follows instruction, a model like which is more likely to say no, you know, when, you know, it doesn't know about something. Um, and so that like super close, like, you know, customer feedback loop on GPT-5 was pretty impressive to, uh, to, to see. And I think like all the love that GPT-5 has been getting, like, you know, in the past like-

Apoorv Agrawal 27:29 Yeah

Olivier Godement 27:29 ... couple of weeks-

Apoorv Agrawal 27:30 Mm

Olivier Godement 27:30 ... I think people are starting to feel that essentially.

Apoorv Agrawal 27:32 Yeah.

Olivier Godement 27:32 The builders. And once you see it, like it's really hard essentially to come back to a model which is like extremely intelligent but more, you know, an exclusively like academic essentially way.

Apoorv Agrawal 27:43 Sure.

Olivier Godement 27:43 Yeah.

Apoorv Agrawal 27:44 Yeah. Are there trade-offs that you made as you were going through it? Like maybe what are the hardest trade-offs you made as you were building GPT-5?

Sherwin Wu 27:51 I actually think a very clear trade-off, which I, I think... I honestly think we are still iterating on, is, uh, the trade-off between the reasoning tokens and how long it thinks versus, uh, performance.

Apoorv Agrawal 28:00 Yeah.

Sherwin Wu 28:00 Um, because... And I honestly-

Apoorv Agrawal 28:01 Yeah. Yeah

Sherwin Wu 28:01 ... this is something that I think we've been, um, working on with our customers since the launch of the reasoning models, which is these models are so, so smart.

Apoorv Agrawal 28:08 Mm.

Sherwin Wu 28:09 Uh, uh, especially if you give it all this like thinking time.

Apoorv Agrawal 28:12 Mm.

Sherwin Wu 28:12 Um, I, I, I think the, the feedback I've been seeing around GPT-5 Pro has been pretty crazy too.

Apoorv Agrawal 28:17 Yeah.

Sherwin Wu 28:17 It's just like, like, you know, these unsolved-

Apoorv Agrawal 28:18 Andre had a great tweet last night.

Sherwin Wu 28:20 Yeah, yeah, I saw that, Sam. Retweeted it. [laughs] Um.

Apoorv Agrawal 28:22 Yeah.

Sherwin Wu 28:22 Uh, but like these, like unsolved problems that none of the other models could handle, you throw to GPT-5 Pro and-

Apoorv Agrawal 28:28 Mm

Sherwin Wu 28:28 ... uh, it just like one-shots it. Is-

Apoorv Agrawal 28:29 Mm

Sherwin Wu 28:29 ... it's pretty crazy. But the trade-off here is you're, you're, you're waiting for 10 minutes.

Apoorv Agrawal 28:33 Yeah. Yeah, yeah.

Sherwin Wu 28:33 Uh, it's quite a long time. And so these things just get like so smart with, with-

Apoorv Agrawal 28:37 Mm

Sherwin Wu 28:37 ... um, uh, with more inference time.

Apoorv Agrawal 28:38 Mm.

Sherwin Wu 28:39 But on the like product builder, on the API side for some of these like business use cases-

Apoorv Agrawal 28:43 Mm

Sherwin Wu 28:43 ... I think it's pretty tough to like, you know, manage that trade-off.

Apoorv Agrawal 28:46 Yeah.

Sherwin Wu 28:46 And for us, it's been difficult to figure out where to... where we want to fall on that spectrum. So we've had to make some trade-offs on like how much should the model think versus like how intelligent should it get.

Apoorv Agrawal 28:54 Yeah.

Sherwin Wu 28:54 Because as a product builder, there's a latency, there's a real latency trade-off that you have to deal with-

Apoorv Agrawal 28:58 Yeah

Sherwin Wu 28:58 ... where, you know, your user might not be happy waiting 10 minutes for like the best answer in the world.

Apoorv Agrawal 29:02 Yeah.

Sherwin Wu 29:02 They might be more okay with the substandard answer and like no wait at all.

Apoorv Agrawal 29:05 Yeah. Yeah. I mean, even between GPT-5 and GPT-5 Thinking, I have to toggle it now because sometimes I'm so impatient I just want it ASAP.

Sherwin Wu 29:13 Yeah. I think there's a-

Apoorv Agrawal 29:14 But other times-

Sherwin Wu 29:14 ... ability to skip, right, in, in ChatGPT-

Apoorv Agrawal 29:16 Yeah, that's right. That's right

Sherwin Wu 29:16 ... where it's like I'm, I'm impatient, I just want the-

Apoorv Agrawal 29:18 That's right

Sherwin Wu 29:18 ... the more simple answer.

Apoorv Agrawal 29:19 That's right. That's right. Well, four weeks in, GPT-5, how's the feedback?

Sherwin Wu 29:24 Yeah. I, I think the feedback has been very positive, uh, especially on the platform side, which, uh, has been really great to see. I think, um, a lot of the, the things that Olivier mentioned have been... you know, come up in feedback from customers.

Apoorv Agrawal 29:35 Mm.

Sherwin Wu 29:35 Um, the model, uh, is extremely good at coding.

Apoorv Agrawal 29:38 Mm.

Sherwin Wu 29:38 Um, extremely good at, at kind of like reasoning through, through different tasks.

Apoorv Agrawal 29:41 Mm. Mm.

Sherwin Wu 29:42 Um, but especially for like coding use cases-

Apoorv Agrawal 29:44 Mm

Sherwin Wu 29:44 ... it, uh, especially at the, uh, you know, at the, uh... When it thinks for a while, it'll usually solve problems that no other models can solve.

Apoorv Agrawal 29:50 Mm.

Sherwin Wu 29:50 So I think that's been a big, uh, positive, uh, point of feedback.

Apoorv Agrawal 29:53 Mm.

Sherwin Wu 29:53 Um, the kind of robustness and the reduction in hallucinations has been a really big-

Apoorv Agrawal 29:57 Yeah

Sherwin Wu 29:57 ... uh, positive feedback.

Apoorv Agrawal 29:58 Yeah, yeah, yeah.

Sherwin Wu 29:59 Um, uh,

Sherwin Wu 30:00 I, I think there was an eval that showed that the hallucinations basically went to zero for a lot of this. It's not perfect.

Apoorv Agrawal 30:04 Wow.

Sherwin Wu 30:04 There's still a lot of work to be done, but-

Apoorv Agrawal 30:05 That's a big one

Sherwin Wu 30:06 ... um, I think because of the reasoning in there too, it just makes, uh, the model more likely to say no, less likely to hallucinate, um, answers. So that's been something that people have really liked as well. Other bit of feedback, uh, has been around instruction following, so it's, uh, really good at instruction following. This almost bleeds into like the constructive feedback that we're working on, where for that it's so good at construction following that... instruction following, that people need to tweak their prompts or-

Apoorv Agrawal 30:28 Mm

Sherwin Wu 30:28 ... it's almost like too literal, uh, in, in kind of-

Olivier Godement 30:30 That's why it is an interesting trade-off actually, 'cause, you know, when you ask people, developers, like what do you want? Like you want the model to follow instructions, of course, you know.

Sherwin Wu 30:36 Yeah. Yeah, yeah. [laughs]

Olivier Godement 30:37 But once you have a model which, who is like... that is like, you know, extremely literal essentially-

Sherwin Wu 30:42 Mm

Olivier Godement 30:42 ... that essentially forces you to express extremely clearly what you want.

Sherwin Wu 30:46 Yeah.

Olivier Godement 30:46 Otherwise the model, you know, may go sideways.

Sherwin Wu 30:48 Yeah.

Olivier Godement 30:48 And so that one-

Sherwin Wu 30:48 Yeah

Olivier Godement 30:48 ... was interesting feedback.

Sherwin Wu 30:49 It's almost like the monkey paw where it's like developers and, and platform customers ask for better instruction following, so we're like, "Yes, we'll give you really good instruction following," but it was like, you know, it follows it almost to a, to a T.

Apoorv Agrawal 30:59 Mm.

Sherwin Wu 30:59 And so it's obviously something that the, the, the team is actually working, working through. I think a good example of this, by the way, is, um, some customers would have these prompts... I remember when we were testing GPT-5, one of the, uh, negative feedback that we got was the model was too concise. We were like, "What's going on?"

Apoorv Agrawal 31:12 Mm.

Sherwin Wu 31:12 "Why is the model so concise?"

Apoorv Agrawal 31:13 Interesting.

Sherwin Wu 31:14 And then we realized it was because they were reusing their old prompts from other models.

Apoorv Agrawal 31:17 Mm.

Sherwin Wu 31:18 And with the other models they have to like, you have to like really beg the model to be prec- uh, concise, so there are like 10 lines of like, "Be concise, really be concise."

Apoorv Agrawal 31:26 Mm.

Sherwin Wu 31:26 "Also, keep your answer short." Um, and it turns out when you give that to GPT-5, it's like, "Oh my gosh, this person really wants it to be concise," and so the response would be like one sentence, which is too terse. And so just by removing the extra prompts, uh, around being concise-

Apoorv Agrawal 31:39 Mm

Sherwin Wu 31:39 ... the model behaved in a much better way-

Apoorv Agrawal 31:41 Mm. Mm

Sherwin Wu 31:41 ... uh, and much closer to what they actually, you know, wanted.

Apoorv Agrawal 31:43 Yeah. Turns out writing the right prompt is still important.

Sherwin Wu 31:46 Yes. Yes. Yeah. Prompt engineering is still very, very important. Yeah.

Apoorv Agrawal 31:49 Yeah.

Sherwin Wu 31:49 Um, on constructive feedback for GPT-5, um, there's actually been a good amount as well, which, uh-

Apoorv Agrawal 31:54 Yeah

Sherwin Wu 31:54 ... we're all, we're all working through.

Apoorv Agrawal 31:55 Yeah.

Sherwin Wu 31:55 Uh, one of them that I think is, uh, I'm, I'm really excited, um, to, for, for the next, uh, you know, snapshot to come out to, to fix some of this is code quality, and, like, small, like, code, uh, like paradigms or, like, uh, uh, idioms that they might use. I think there were, like, feedback around, um, the types of code and, and, and the patterns in which it was using, which I think we're working through-

Apoorv Agrawal 32:14 Mm

Sherwin Wu 32:14 ... uh, as well. Um, and then, uh, the, the other bit of feedback which, uh, I think we've already made good progress on internally, uh, is around the trade-off of the reasoning tokens and thinking and latency around intelligence.

Apoorv Agrawal 32:24 Right.

Sherwin Wu 32:24 I think especially for these simpler problems, you don't usually need a lot of thinking. The thinking should ideally be a little bit more dynamic.

Apoorv Agrawal 32:31 Mm.

Sherwin Wu 32:31 Um, and of course, we're always trying to squeeze as much reasoning and performance into as little reasoning tokens as possible.

Apoorv Agrawal 32:36 Yeah.

Sherwin Wu 32:36 So I'd mention that, that curve-

Apoorv Agrawal 32:37 Yeah

Sherwin Wu 32:37 ... kind of going down as well.

Apoorv Agrawal 32:39 Yeah. Well, huge congrats. I mean, it's been... I know it's a work in motion for a bunch of our companies. They've had incredible outcomes with GPT-5. One of them is Expo, uh, cybersecurity business.

Sherwin Wu 32:49 Mm-hmm.

Apoorv Agrawal 32:49 Just like a huge-

Sherwin Wu 32:49 Yeah, I saw the chart from that. It was pretty crazy.

Apoorv Agrawal 32:51 Huge-

Sherwin Wu 32:51 Yeah

Apoorv Agrawal 32:52 ... huge upgrade from whatever they were using prior to that, and, uh-

Sherwin Wu 32:55 I think they're gonna need a new eval soon.

Apoorv Agrawal 32:56 [laughs] That's right.

Sherwin Wu 32:57 Yeah.

Apoorv Agrawal 32:57 They're gonna need a new eval.

Sherwin Wu 32:59 Yeah.

Apoorv Agrawal 32:59 It's all about

Apoorv Agrawal 33:00 evals. On, on the multimodality side of it, uh, obviously you guys announced the real-time API last week.

Sherwin Wu 33:05 Yeah.

Apoorv Agrawal 33:05 Uh, I saw T-Mobile was one of the featured customers on there.

Sherwin Wu 33:08 Yeah.

Apoorv Agrawal 33:08 Uh, talk about that, like, how... Obviously, the text models are qui- leading the pack.

Sherwin Wu 33:13 Yeah.

Apoorv Agrawal 33:13 But then we got audio and we got video.

Sherwin Wu 33:15 Yeah.

Apoorv Agrawal 33:15 Talk about the progress on the multimodal, uh, models. Uh, when should we expect to have, like, the next big unlock, and what would that look like?

Olivier Godement 33:23 Uh, it's a good question. Um, the teams have been making amazing progress on multimodality, on voice, image, video. Frankly, the last generation models have been unlocking, like, quite a few cool use cases.

Apoorv Agrawal 33:33 Mm.

Olivier Godement 33:34 Um, one of the feedback that we've received is, you know, because, like, text was so much leading the pack on intelligence-

Apoorv Agrawal 33:40 Mm

Olivier Godement 33:40 ... like, people felt like inputting your own voice at the moment was somewhat a little less intelligent. And, you know-

Apoorv Agrawal 33:45 Yeah

Olivier Godement 33:45 ... until you actually see it, like, it does feel weird, like-

Apoorv Agrawal 33:47 Yeah, yeah

Olivier Godement 33:48 ... you need to have, like, you know, to have a better answer, like, on text versus voice. And so that's pretty much a focus that we have at the moment.

Apoorv Agrawal 33:54 Mm.

Olivier Godement 33:54 Um, I think we, like, filled, like, part of that gap, but not the full gap, for sure. So I think, you know, catching up, I would say, you know-

Apoorv Agrawal 34:02 Yeah

Olivier Godement 34:02 ... with the text, like, you know, would be one.

Apoorv Agrawal 34:04 Yeah.

Olivier Godement 34:04 Um, a second one, you know, which is absolutely fascinating is, uh, the model is, like, excellent at the moment on, like, you know, easy, like, casual conversation.

Apoorv Agrawal 34:16 Mm.

Olivier Godement 34:16 Like, talk to your coach, your therapist.

Apoorv Agrawal 34:18 Mm.

Olivier Godement 34:18 And we basically had to teach the model, like, to, um, speak essentially better, like in actual, like, you know, work, economically valuable setups. Give an example, like, the model has to be able to understand what an SSN is and, you know, what does it mean to spell a SSN.

Apoorv Agrawal 34:32 Mm.

Olivier Godement 34:33 And if one digit is actually, like, you know, fuzzy, it should actually has to repeat versus, you know, guess.

Apoorv Agrawal 34:37 Mm, mm.

Olivier Godement 34:37 You know, there are lots of, like, you know, intuitions like that that someone, you know-

Apoorv Agrawal 34:40 Right

Olivier Godement 34:40 ... of course, has, you know, of our voice-

Apoorv Agrawal 34:42 Right

Olivier Godement 34:42 ... that we are currently, like, teaching the model.

Apoorv Agrawal 34:44 Mm.

Olivier Godement 34:44 And that's, like, an ongoing work actually with our customers-

Apoorv Agrawal 34:47 Yeah

Olivier Godement 34:47 ... you know, until we actually confront the model to, like, actual, like, customer support calls, actual sales call-

Apoorv Agrawal 34:53 Right

Olivier Godement 34:53 ... it's really hard, like, to get a feel-

Apoorv Agrawal 34:55 Right

Olivier Godement 34:55 ... you know, for those gaps. So that's, um, a top, like, you know, priority as well.

Apoorv Agrawal 34:59 This is, uh, completely off script, but an interesting question that comes up in voice models, particularly the real-time API, is, you know, previously people were doing-- They were taking a speech input, convert that to text-

Olivier Godement 35:12 Yeah

Apoorv Agrawal 35:12 ... then have some layer of intelligence.

Olivier Godement 35:14 Yeah.

Apoorv Agrawal 35:14 Then you would have a text-to-speech model that would sort of play it back.

Olivier Godement 35:17 Yeah.

Apoorv Agrawal 35:18 And this would be... It would be a stitch of these three parts. But the real-time API, you guys have integrated all of that.

Olivier Godement 35:24 Yes.

Sherwin Wu 35:24 Yeah.

Apoorv Agrawal 35:25 And h- you know, h- how does it happen? Because a lot of the logic is written in, in text.

Apoorv Agrawal 35:30 Be- a lot of the Boolean logic or you call it any function calling is written in text. How, how does it work with the real-time API? Is that...

Olivier Godement 35:38 That's an excellent question. Um, so the reason why we shipped the real-time API is that we saw that for the stitch model-

Apoorv Agrawal 35:44 Mm. The stitch model?

Olivier Godement 35:46 Yeah.

Apoorv Agrawal 35:46 Oh, the real-time, yeah.

Olivier Godement 35:47 The stitched. The stitched.

Sherwin Wu 35:48 We call it, like, a stitch together.

Olivier Godement 35:49 Stitched together-

Apoorv Agrawal 35:49 Okay

Olivier Godement 35:50 ... like speech to text, thinking-

Apoorv Agrawal 35:51 I see

Olivier Godement 35:51 ... text to speech.

Apoorv Agrawal 35:52 Yeah.

Olivier Godement 35:53 Like, we saw essentially a couple of issues. One, slowness, like, you know-

Apoorv Agrawal 35:56 Yeah

Olivier Godement 35:57 ... more ops essentially.

Apoorv Agrawal 35:57 Yeah, yeah, yeah.

Olivier Godement 35:58 Two, loss of signal, like, you know-

Apoorv Agrawal 36:01 Mm

Olivier Godement 36:01 ... across each model. Like, the speech-text model-

Apoorv Agrawal 36:04 Mm

Olivier Godement 36:04 ... is less intelligent.

Apoorv Agrawal 36:05 Yeah, you lose-

Olivier Godement 36:05 Essentially

Apoorv Agrawal 36:06 ... emotion.

Olivier Godement 36:06 Exactly.

Apoorv Agrawal 36:06 You lose accent, tone.

Olivier Godement 36:08 Exactly.

Apoorv Agrawal 36:09 Right.

Olivier Godement 36:09 Things like that.

Apoorv Agrawal 36:09 Pauses. Yeah.

Olivier Godement 36:10 And you know, when you, when you are doing, like, actual voice, like, phone, like, calls, essentially, like, th- those signals are, like, so important-

Apoorv Agrawal 36:18 Yeah

Olivier Godement 36:18 ... like, you know-

Apoorv Agrawal 36:18 Yeah, yeah

Olivier Godement 36:18 ... for the sensitivity stem.

Apoorv Agrawal 36:20 Yeah.

Olivier Godement 36:20 Um, one of the challenge that we have is what you mentioned-

Apoorv Agrawal 36:23 Mm

Olivier Godement 36:23 ... which is, you know, it means, like, a slightly different architecture essentially for text versus voice.

Apoorv Agrawal 36:27 Mm.

Olivier Godement 36:27 And so that's something that we are actively working on. Uh, but I think it was the right call to start essentially with let's make, like, the voice experience, like, natural sounding to a point where essentially you're feeling comfortable, like, putting in production, and then working backward, like, to unify, like, the sort of the orchestration logic essentially across modalities.

Sherwin Wu 36:46 And, and to be clear, like, a lot of customers still stitch these together. It's, like, kind of what worked in the last generation.

Apoorv Agrawal 36:52 Yeah, yeah.

Sherwin Wu 36:52 But what we're in- interestingly seeing is more and more customers moving towards the real-time approach because of how natural it sounds-

Apoorv Agrawal 36:57 Yeah

Sherwin Wu 36:57 ... how much lower latency it is-

Apoorv Agrawal 36:59 Yeah

Sherwin Wu 36:59 ... especially as we uplevel the intelligence-

Apoorv Agrawal 37:01 Yeah

Sherwin Wu 37:01 ... uh, of the model. Uh, but also even, like, taking a step back, I will say it's, like, pretty mind-blowing to me that it works. Like, the fact that-

Olivier Godement 37:06 [laughs]

Sherwin Wu 37:07 Like, I think it's mind-blowing that these LMS work at all-

Apoorv Agrawal 37:09 Yeah

Sherwin Wu 37:09 ... where you just train it on a bunch of text, and it's just-

Apoorv Agrawal 37:11 Yeah

Sherwin Wu 37:11 ... you know, autoregressively coming up with the next token, and it sounds super intelligent.

Apoorv Agrawal 37:14 Yeah.

Sherwin Wu 37:14 That's, like, mind-blowing in and of itself, but I think it's actually even more mind-blowing that this speech-to-speech setup actually works correctly-

Apoorv Agrawal 37:19 Yeah

Sherwin Wu 37:19 ... 'cause you're literally taking the audio bits-

Apoorv Agrawal 37:21 Yeah

Sherwin Wu 37:22 ... uh, from it, from someone speaking-

Apoorv Agrawal 37:24 Yeah

Sherwin Wu 37:24 ... streaming or putting it into the, the model, and then it's generating audio bits back.

Apoorv Agrawal 37:28 Yeah.

Sherwin Wu 37:28 And so, uh, to me, it's actually crazy this works at all-

Apoorv Agrawal 37:30 It's pretty crazy

Sherwin Wu 37:31 ... uh, let alone the fact that it can understand accents and tone-

Apoorv Agrawal 37:34 Yeah, yeah, yeah

Sherwin Wu 37:34 ... and pauses and things like that, and then also be intelligent enough to-

Apoorv Agrawal 37:37 Yeah

Sherwin Wu 37:37 ... handle a support call or something like that.

Apoorv Agrawal 37:38 I mean, you've gone from text in, text out to voice in, voice out.

Sherwin Wu 37:42 Yep.

Apoorv Agrawal 37:42 That's pretty crazy.

Olivier Godement 37:43 Yeah.

Sherwin Wu 37:43 Yeah.

Apoorv Agrawal 37:44 We have a bunch of companies i- in our portfolio that are using these models, you know, Parlo on the customer support side-

Sherwin Wu 37:49 Yeah

Apoorv Agrawal 37:49 ... Livekit on the infra side, and-

Sherwin Wu 37:51 Yeah

Apoorv Agrawal 37:52 ... you know, there's a bunch of use cases we are starting to see that, that, that a speech-to-speech model could, could address. Obviously, a lot of the c- Harder ones still, still running

Apoorv Agrawal 38:00 on what you're calling the stitched model.

Olivier Godement 38:01 Yeah, yeah.

Apoorv Agrawal 38:02 Uh, but I, I hope the day is not far when it's all on Realtime API.

Olivier Godement 38:06 It's gonna happen-

Apoorv Agrawal 38:07 Yeah

Olivier Godement 38:07 ... at some point. [laughs]

Apoorv Agrawal 38:08 Right. Right, right, right. And actually, maybe that's a good segue into talking about model customization because I suspect that you have such a wide variety of enterprise customers. I think you mentioned, what, hundreds of customers or, or, or maybe more. Um, each of them has a different use case, a different problem set, a different colored envelope of, of, of parameters that they're working in, maybe latency, maybe, maybe power, maybe others. How do you, how do you handle that? Talk about what OpenAI offers enterprises who need a customized version of, of, of a great model to make it great for them.

Sherwin Wu 38:38 Yeah, so model customization's actually been something that we've invested very deeply in on the API platform since the very beginning.

Apoorv Agrawal 38:44 Mm.

Sherwin Wu 38:44 So even, you know, uh, pre-ChatGPT days, we actually had a supervised fine-tuning, uh, API available-

Apoorv Agrawal 38:50 Mm-hmm

Sherwin Wu 38:50 ... and people were actually using it to great effect.

Apoorv Agrawal 38:52 Mm-hmm.

Sherwin Wu 38:52 Um, the most exciting thing actually, I'd say, around model customization, it obviously resonates quite well with customers-

Apoorv Agrawal 38:57 Mm

Sherwin Wu 38:57 ... because they want to be able to, uh, bring in your own custom data and create your own custom version of a, you know, o3 or o4 mini or something-

Apoorv Agrawal 39:04 Mm.

Sherwin Wu 39:04 Um, or GPT-5 even, um, uh, suited to their own needs.

Apoorv Agrawal 39:09 Mm.

Sherwin Wu 39:09 Um, it's very attractive, but the, the most recent development I think is very exciting is, has been the introduction of reinforcement fine-tuning.

Apoorv Agrawal 39:15 Mm.

Sherwin Wu 39:16 Which is something we announced, uh, late last year, I think in the 12 Days of Christmas. We've GA'd it since.

Apoorv Agrawal 39:20 Mm.

Sherwin Wu 39:20 Uh, and we're continuing to iterate on it.

Apoorv Agrawal 39:22 What, what is it? Break it down for us.

Sherwin Wu 39:24 Yeah, so it's called... Uh, it's actually funny, I, I, I think we, we made up the term reinforcement fine-tuning. It's like not a real, uh, thing until, until we announced it.

Olivier Godement 39:31 It's stuck now.

Sherwin Wu 39:32 It's stuck. [laughs]

Olivier Godement 39:32 I see it on Twitter all the time, yeah.

Sherwin Wu 39:33 I remember we were discussing it, and I was like, "I, I don't know about RFT, guys."

Olivier Godement 39:35 Good marketing. Good marketing. [laughs]

Sherwin Wu 39:37 Yeah, yeah. So reinforcement fine-tuning, um, so it really, uh, it, it's introducing, uh, reinforcement learning into the fine-tuning process.

Apoorv Agrawal 39:44 Mm.

Sherwin Wu 39:44 So, um, the original, um, fine-tuning API, uh, does something called supervised fine-tuning. We call it SFT.

Apoorv Agrawal 39:51 Mm.

Sherwin Wu 39:51 Um, it is not using reinforcement learning.

Apoorv Agrawal 39:53 Yeah.

Sherwin Wu 39:53 It is, um, you know, it's, it's, it's using supervised learning.

Apoorv Agrawal 39:56 Supervised learning, yeah.

Sherwin Wu 39:57 Um, and so what that usually means is you need a bunch of data, a bunch of prompt completion pairs. You need to really supervise-

Apoorv Agrawal 40:02 Labels

Sherwin Wu 40:02 ... and tell exactly the model how to, how it should be acting, and then when you train it on, on our fine-tuning API, it moves it closer in that direction. Reinforcement fine-tuning introduces re- like RL or reinforcement learning to this loop.

Apoorv Agrawal 40:13 Mm.

Sherwin Wu 40:13 Way more complex, way more finicky, but an order of magnitude more powerful.

Apoorv Agrawal 40:18 Mm.

Sherwin Wu 40:18 And so, um, that's actually what's really resonated with a lot of our customers. It allows you to, if you use RFT, the, the discussion is less of, like, creating a s- custom model that's specific to your own use case. It is, you can actually use your own data-

Apoorv Agrawal 40:30 Mm

Sherwin Wu 40:30 ... and actually crank the RL, uh, uh, yeah, turn the crank on RL to actually create a, like, best-in-class model for your own particular use case.

Apoorv Agrawal 40:39 Mm.

Sherwin Wu 40:39 And so that's kind of the, the main difference here.

Apoorv Agrawal 40:41 Mm.

Sherwin Wu 40:41 Um, with RFT, the data set looks a little bit different. Instead of, you know, prompt completion pairs-

Apoorv Agrawal 40:45 Gotcha

Sherwin Wu 40:45 ... you really need a set of tasks that are very gradable. You need a grader, uh, that is very objective that you can use here as well.

Apoorv Agrawal 40:51 Mm.

Sherwin Wu 40:52 Uh, and so that's actually been something that we've invested a lot in over the last year, and, um, we've actually seen a couple, a good number of customers get really good results on this.

Apoorv Agrawal 40:59 Mm.

Sherwin Wu 40:59 We've talked about a couple of the, couple of them across different verticals. So Rogo, um, which is a startup in the, um-

Apoorv Agrawal 41:04 Yeah

Sherwin Wu 41:05 ... uh, financial services space-

Apoorv Agrawal 41:06 Financial.

Sherwin Wu 41:06 Um, they, uh, have a very, you know, sophisticated AI team. I think they hired some folks from DeepMind, um-

Apoorv Agrawal 41:11 Mm

Sherwin Wu 41:11 ... to, to run their AI program. Um, and they've used, been using RFT, uh, to get best-in-class results on-

Apoorv Agrawal 41:16 Mm

Sherwin Wu 41:17 ... you know, parsing through financial documents, answering questions around it, and, and doing tasks around that as well. Um, there's another, uh, uh, startup called Accordance that's doing this in the tax space. I think they, they've been targeting a, um, an eval called Tax Bench-

Apoorv Agrawal 41:30 Mm

Sherwin Wu 41:30 ... uh, which looks at, you know, CPA-style tasks as well.

Apoorv Agrawal 41:33 Mm.

Sherwin Wu 41:33 And because they, because, you know, they're able to turn it into a very gradable setup-

Apoorv Agrawal 41:36 Mm

Sherwin Wu 41:37 ... um, they're actually able to turn the RFT crank and also get, I think, like, SOTA results on, uh, Tax Bench just using our RFT-

Apoorv Agrawal 41:43 Nice

Sherwin Wu 41:44 ... uh, uh, product as well. And so, um, it has kind of shifted the, the discussion away from just customizing something for your own use case to, like-

Apoorv Agrawal 41:50 Mm

Sherwin Wu 41:50 ... really leveraging your own data to create a best-in-class-

Apoorv Agrawal 41:53 Mm

Sherwin Wu 41:53 ... maybe best-in-the-world model for, uh, something that you care about for your business.

Olivier Godement 41:56 Yeah, I feel like the-

Apoorv Agrawal 41:57 Fascinating

Olivier Godement 41:57 ... the base models are getting so good at instruction following-

Apoorv Agrawal 42:01 Mm

Olivier Godement 42:01 ... that for, you know, behavior like steering, like, you know, you don't need to fine-tune at that point. Like, you know, you can describe what you want, and the model is pretty good at it.

Apoorv Agrawal 42:09 Mm.

Olivier Godement 42:09 But pushing the frontier on, like, actual capabilities-

Apoorv Agrawal 42:13 Mm

Olivier Godement 42:13 ... my hunch is that RFT will pretty much become the norm.

Apoorv Agrawal 42:16 Interesting.

Olivier Godement 42:16 Like, you know, if you are actually pushing in your field, like, you know-

Apoorv Agrawal 42:19 Mm

Olivier Godement 42:20 ... intelligence, like, you know, to a, a pretty high point-

Apoorv Agrawal 42:22 Mm

Olivier Godement 42:23 ... like, at some point, like, you know, you, you need to RL, RL essentially with custom environments.

Apoorv Agrawal 42:27 Yeah.

Sherwin Wu 42:27 Fascinating. And even going back to the point earlier around, like, top-down versus bottoms-up for some of these enterprises, a lot of the data that you end up needing for RFT require, like, very intricate knowledge about the exact task that you're doing-

Apoorv Agrawal 42:39 Mm

Sherwin Wu 42:39 ... and understanding how to grade it.

Apoorv Agrawal 42:41 Mm.

Sherwin Wu 42:41 Uh, and so a lot of that actually comes from bottoms-up. Like, a lot, I know a lot of these startups will work with experts in their field-

Apoorv Agrawal 42:46 Mm

Sherwin Wu 42:46 ... to try and get the right tasks and, um, um, get the right feedback to craft some of these data sets.

Apoorv Agrawal 42:51 Without further ado, we're gonna jump into my favorite section, which is a rapid-fire question. We had a lot of great, uh, friends of ours send in some questions for you guys.

Sherwin Wu 43:00 Okay.

Apoorv Agrawal 43:00 But we'll start with, uh, Altimeter's favorite game, which is a long-short game.

Sherwin Wu 43:04 Mm-hmm.

Apoorv Agrawal 43:04 Pick a business, an idea, a startup that you're long, and the same short that you would bet against, that there's more hype than there is reality. Uh, whoever's ready to go first, long-short.

Sherwin Wu 43:17 My long is actually not in the AI space, so this is gonna be slightly different.

Apoorv Agrawal 43:20 Wow. Here we go.

Sherwin Wu 43:21 Um, uh, my short is, though, in the, in the AI, uh, space. So, um, I'm actually extremely long, uh, esports. And so, uh, what I mean by esports is the entire, like, professional gaming industry-

Apoorv Agrawal 43:31 Interesting

Sherwin Wu 43:31 ... that's emerging around video games.

Apoorv Agrawal 43:33 Mm-hmm.

Sherwin Wu 43:33 Um, very near and dear to my heart. I play a lot of video games, and so I watch a lot of this.

Apoorv Agrawal 43:36 [laughs]

Sherwin Wu 43:37 So, um, obviously I'm, I'm pretty in the weeds on this.

Apoorv Agrawal 43:39 Mm.

Sherwin Wu 43:39 Um, but I actually think there's incredible untapped potential in esports and incredible growth to be had in this area.

Apoorv Agrawal 43:44 Mm.

Sherwin Wu 43:45 Um, so concretely what I mean are, like, you know, a really big, uh, one is, uh, League of Legends, all of the games that Riot Games puts out.

Apoorv Agrawal 43:51 Mm.

Sherwin Wu 43:51 They actually have their own professional leagues. Uh, they actually have professional, uh, tournaments.

Apoorv Agrawal 43:56 Mm.

Sherwin Wu 43:56 Uh, believe it or not, they rent out stadiums-

Apoorv Agrawal 43:58 Mm

Sherwin Wu 43:58 ... actually now. Um, but I just think it's, like, if you look at kind of what the youth and, like, what younger kids are looking-

Apoorv Agrawal 44:04 Mm

Sherwin Wu 44:04 ... and where their time is going, it's-

Apoorv Agrawal 44:05 Mm

Sherwin Wu 44:05 ... predominantly going towards these things.

Apoorv Agrawal 44:07 Mm.

Sherwin Wu 44:07 They spend a lot of time on video games.

Olivier Godement 44:08 They watch more esports than, like, soccer or basketball, et cetera.

Sherwin Wu 44:11 Yeah, yeah, yeah, yeah. A growing number of these too. Um, I've actually been to some of these, uh, events, uh, and it's very interesting.

Olivier Godement 44:18 He's very committed to his long. [laughs]

Sherwin Wu 44:20 Yeah, yeah, I'm extremely long on this stuff. Um, uh, and-

Apoorv Agrawal 44:22 And so they're booking out stadiums for people to go watch Electronic sports?

Sherwin Wu 44:28 Yeah. Yeah, yeah. It's, uh... I literally went to Oracle Arena, the old Warriors stadium-

Apoorv Agrawal 44:32 Mm-hmm, mm-hmm

Sherwin Wu 44:32 ... to watch one of these, I think, uh, before COVID. Um-

Apoorv Agrawal 44:35 Wow

Sherwin Wu 44:35 ... and then the... So it, it's just-

Apoorv Agrawal 44:37 Before COVID? Wow

Sherwin Wu 44:37 ... and all of the-

Apoorv Agrawal 44:38 That's five years ago, six years ago

Sherwin Wu 44:39 ... it was a while ago. So I actually- I've been following this for a while, and I actually think it had a really big moment in COVID, like everyone was playing video games.

Apoorv Agrawal 44:44 Yeah, of course, of course.

Sherwin Wu 44:45 And I think it's kind of, like, come back down. So I think it's, like, undervalued.

Apoorv Agrawal 44:47 Hmm.

Sherwin Wu 44:47 You know, it's like I think no one's really appreciating it now.

Apoorv Agrawal 44:50 Hmm.

Sherwin Wu 44:50 But it has all the elements to, like, really, really take off.

Apoorv Agrawal 44:52 Hmm.

Sherwin Wu 44:53 Um, and so the youth are, are doing it. The other thing I'd say is it is huge in Asia. Like, absolutely-

Apoorv Agrawal 44:58 Right

Sherwin Wu 44:58 ... massive in Asia. Um, it is absolutely big in, uh, Korea and China as well.

Apoorv Agrawal 45:03 Hmm.

Sherwin Wu 45:03 Like, you know, we rented out Oracle Arena, I think. Or, like, the, the, the event I went to was in Oracle Arena. My sense is in Asia they rent out, like, the entire stadiums, like the soccer stadiums, and the players are already like celebrities.

Apoorv Agrawal 45:14 Hmm.

Sherwin Wu 45:14 So anyways, um, you know, as, like, you know, the... I know, like, Korean culture is really making its way into the US as well. I think that's another tailwind for this whole thing.

Apoorv Agrawal 45:20 Hmm.

Sherwin Wu 45:20 But anyways, esports I think is something you should keep an eye out on-

Apoorv Agrawal 45:23 All right

Sherwin Wu 45:23 ... 'cause, uh, uh, there's a lot of room for growth.

Apoorv Agrawal 45:25 Very unexpected.

Sherwin Wu 45:26 Yeah.

Apoorv Agrawal 45:27 Good to hear. Short?

Sherwin Wu 45:28 Uh, my short.

Apoorv Agrawal 45:28 My short.

Sherwin Wu 45:29 Short's a little spicy, um, which is I'm short on, uh, the entire category of, like, tooling, uh, uh, around AI, uh, AI products. Uh, and so this, this encapsulates a lot of different things.

Apoorv Agrawal 45:41 Hmm.

Sherwin Wu 45:41 Kind of cheating 'cause some of these, you know, I think are starting to play out already, but-

Apoorv Agrawal 45:44 Hmm

Sherwin Wu 45:44 ... um, I think, like, two years ago it was maybe, like, evals products or, like-

Apoorv Agrawal 45:49 Hmm

Sherwin Wu 45:49 ... uh, frameworks, uh, or vector stores.

Apoorv Agrawal 45:51 Hmm.

Sherwin Wu 45:51 I'm pretty short those.

Apoorv Agrawal 45:52 Hmm.

Sherwin Wu 45:53 I think nowadays, uh, there's a lot of, um, additional, uh, excitement around other, uh, tooling around AI models. So, uh, RL environments I think are really-

Apoorv Agrawal 46:02 [laughs]

Sherwin Wu 46:02 ... big right now as well. Unfortunately, I'm very short on those.

Apoorv Agrawal 46:04 Okay.

Sherwin Wu 46:04 I'm not really-- uh, I don't really see a lot of, uh, potential there.

Apoorv Agrawal 46:08 Hmm.

Sherwin Wu 46:08 See a lot of potential in, in reinforcement learning and applying it-

Apoorv Agrawal 46:11 Hmm

Sherwin Wu 46:11 ... but I think the startup, uh, uh-

Apoorv Agrawal 46:13 Hmm

Sherwin Wu 46:14 ... space around RL environments-

Apoorv Agrawal 46:15 Hmm

Sherwin Wu 46:15 ... I think is really tough. Um, main, main thing is, is, one, it's just a very competitive space.

Apoorv Agrawal 46:19 Yeah.

Sherwin Wu 46:19 There's just a lot of people kind of-

Apoorv Agrawal 46:21 Yeah

Sherwin Wu 46:21 ... operating in. And then, two, um, if the last two years have shown us anything, the space is evolving so quickly-

Apoorv Agrawal 46:26 Hmm

Sherwin Wu 46:26 ... and it's so difficult to try and, like, adapt and make s- and understand what the exact stack is that will-

Apoorv Agrawal 46:31 Hmm

Sherwin Wu 46:31 ... really, you know, uh, uh, carry through to the next generation of models. I think that just makes it very difficult when you're in, when you're in the tooling space because-

Apoorv Agrawal 46:38 Hmm

Sherwin Wu 46:38 ... you know, today's really hot framework or really har-

Apoorv Agrawal 46:40 Hmm

Sherwin Wu 46:41 ... hot tool might just not get used in the next generation of models.

Apoorv Agrawal 46:44 Fascinating.

Olivier Godement 46:44 So I, I've been noticing, like, the same pattern, which is-

Apoorv Agrawal 46:47 Hmm

Olivier Godement 46:47 ... the, the teams that build, like, breakout, like, startups in AI are extremely pragmatic.

Apoorv Agrawal 46:53 Pragmatic.

Olivier Godement 46:54 Like, they are not-

Apoorv Agrawal 46:54 Yep

Olivier Godement 46:54 ... super, like, you know, intellectual, but like the perfect world, et cetera.

Apoorv Agrawal 46:58 Mm-hmm. Mm-hmm.

Olivier Godement 46:58 And it's funny 'cause I feel like, you know, our generation has basically started in tech in a very, like, stable, like, you know, moment. While, you know, technology had been building up, like, you know, for years and years with, like, SaaS, like cloud, et cetera. And so we were, in a way, like, raised, like, you know, in that very stable moment-

Apoorv Agrawal 47:13 Yeah

Olivier Godement 47:13 ... where it makes sense at that point-

Apoorv Agrawal 47:14 Hmm

Olivier Godement 47:14 ... to, you know, design, like, very, like, you know, good, like, abstractions and toolings-

Apoorv Agrawal 47:18 Hmm

Olivier Godement 47:18 ... 'cause, you know, you have a sense where it's going. But it's so different today. Like-

Apoorv Agrawal 47:22 Hmm

Olivier Godement 47:22 ... no one quite knows what's gonna happen next year or two.

Apoorv Agrawal 47:24 Hmm.

Olivier Godement 47:24 So it's almost impossible, like, to define, like, the perfect tooling and platform.

Apoorv Agrawal 47:27 Right. Right, right, right. Well, that's, uh, there's a lot of that going around right now. [laughs]

Olivier Godement 47:32 Yes.

Apoorv Agrawal 47:32 Um, spicy. Lot of homework there. Uh, Olivier, over to you, sir. Long/short.

Olivier Godement 47:37 Long/short, I've been thinking a lot about education-

Apoorv Agrawal 47:41 Hmm

Olivier Godement 47:41 ... for the past month, uh, in the context of kids.

Apoorv Agrawal 47:43 Mm-hmm, mm-hmm.

Olivier Godement 47:45 Um, I'm pretty short on any education which basically emphasizes, uh, human memorization at that point. And I say that having mostly been through that education myself.

Apoorv Agrawal 47:56 [laughs]

Olivier Godement 47:56 But, you know, like, I learned so much on, like, you know, history facts, like, you know, legal things, et cetera.

Apoorv Agrawal 48:01 Yeah, yeah.

Olivier Godement 48:01 Um, you know, some of it, like, does shape your way of thinking. A lot of it, frankly, is just, like, you know, knowledge tokens, essentially.

Apoorv Agrawal 48:08 Yeah, yeah, yeah.

Olivier Godement 48:08 And those knowledge tokens, you know, it turns out, like, you know, LLMs are pretty good at it.

Apoorv Agrawal 48:12 Yeah.

Olivier Godement 48:12 Um, so I'm, I'm quite short on that. Uh-

Apoorv Agrawal 48:15 That's right. You won't need memory once ChatGPT is bionic. You can just-

Olivier Godement 48:18 Yeah. [laughs]

Apoorv Agrawal 48:18 ... think about it-

Olivier Godement 48:18 There we go

Apoorv Agrawal 48:19 ... straight into your head, yeah.

Olivier Godement 48:20 Exactly.

Apoorv Agrawal 48:20 That's right. [laughs]

Olivier Godement 48:20 Exactly. What am I long at? Um, frankly, I think healthcare is probably the industry that will benefit the most from AI-

Apoorv Agrawal 48:32 Interesting

Olivier Godement 48:32 ... in the next, like, year or two.

Apoorv Agrawal 48:33 Oh, say more.

Olivier Godement 48:34 I think, like, all the ingredients are here for a perfect storm.

Apoorv Agrawal 48:38 Mm-hmm.

Olivier Godement 48:38 Um, a huge amount of, like, structure, the structured data. You know-

Apoorv Agrawal 48:42 Hmm

Olivier Godement 48:42 ... it's basically the heart of, you know-

Apoorv Agrawal 48:44 Mm-hmm

Olivier Godement 48:44 ... like the, the pharma companies.

Apoorv Agrawal 48:46 Hmm.

Olivier Godement 48:46 Uh, the models are excellent at digesting-

Apoorv Agrawal 48:49 Hmm

Olivier Godement 48:49 ... processing that kind of data. Uh, a huge amount of, like, admin, like, you know, heavy-

Apoorv Agrawal 48:53 Hmm

Olivier Godement 48:53 ... like, documents heavy, like, you know, culture.

Apoorv Agrawal 48:56 Interesting.

Olivier Godement 48:56 But at the same time, like, companies which are very technical, very R&D friendly-

Apoorv Agrawal 49:01 Hmm

Olivier Godement 49:01 ... like, you know, companies like, you know, who's, like, sort of technology in a way is at the heart-

Apoorv Agrawal 49:05 Hmm

Olivier Godement 49:05 ... of what they do. And so yeah, I'm pretty bullish on the healthcare.

Apoorv Agrawal 49:08 This is like life sciences. So you mean life sciences-

Olivier Godement 49:10 Yes

Apoorv Agrawal 49:10 ... research organizations that are producing-

Olivier Godement 49:12 Exactly

Apoorv Agrawal 49:12 ... drugs. Gotcha.

Olivier Godement 49:13 Exactly. Yeah.

Apoorv Agrawal 49:14 Gotcha.

Sherwin Wu 49:14 Yeah, it's, it's almost like, you know, over the last 20, 30 years, these, these, like, pharma or, like, biotech companies have, have basically-- If you look at the work that they're doing, like, only a small amount of it is, is actual research, and so much of it ends up being admin and, like-

Apoorv Agrawal 49:29 Hmm

Sherwin Wu 49:29 ... you know, documents and, and things like that. And that area-

Apoorv Agrawal 49:31 Hmm

Sherwin Wu 49:31 ... is just so ripe for-

Apoorv Agrawal 49:33 Yeah

Sherwin Wu 49:33 ... you know, something to happen with AI, and I think that's what we're seeing with Amgen and some of these other customers.

Olivier Godement 49:36 Exactly.

Sherwin Wu 49:37 And it's also, like, not what they want to do. It's-- I think it's good that we have some regulations there, obviously.

Apoorv Agrawal 49:41 Hmm.

Sherwin Wu 49:41 But, like, just means that they have, like, reams and reams of things they kind of go through.

Apoorv Agrawal 49:45 Right.

Sherwin Wu 49:45 And so, you know, w- like, when you have a technology that's able to really help s- like, bring down the, the cost of something like that, I think it'll just, you know, tear right through it.

Olivier Godement 49:53 And I think once governments and, like, you know, institutions are going to realize that... Like, if you step back, like, it is probably one of the biggest bottleneck to, like, human progress, right?

Apoorv Agrawal 50:01 Hmm.

Olivier Godement 50:02 You step back in the past decade, like, you know, how many, like, true, like, breakthrough drugs have there been?

Apoorv Agrawal 50:07 Yeah.

Olivier Godement 50:07 Like, you know, not that many. Like, you know, how life would be different, like, if you double that rate, essentially. So once you realize what are the stakes, yeah, my hunch is that we're going to see quite a bit of momentum in that space.

Apoorv Agrawal 50:18 Wow. All right. Lots of homework there as well.

Olivier Godement 50:21 Yeah.

Apoorv Agrawal 50:22 Next one, favorite underrated AI tool, other than ChatGPT, maybe.

Olivier Godement 50:27 I love Granola.

Sherwin Wu 50:28 Oh man, you stole my-

Apoorv Agrawal 50:29 Granola

Sherwin Wu 50:29 ... you stole my answer. [laughs]

Olivier Godement 50:31 I love so much Granola. Like, um-

Apoorv Agrawal 50:32 Two votes for Granola

Olivier Godement 50:33 ... there is something like, yeah-

Sherwin Wu 50:35 Hey, what about ChatGPT record?

Olivier Godement 50:37 I like ChatGPT record as well-

Sherwin Wu 50:38 [laughs]

Olivier Godement 50:38 ... but, um, there are some features of Granola which I think are really done well.

Sherwin Wu 50:42 Mm.

Olivier Godement 50:42 Like, the whole, like, you know, integration with your Google Calendar is excellent.

Sherwin Wu 50:45 Yeah.

Olivier Godement 50:46 Um, and just, you know, the quality of, like, the transcription and, like, the summary is pretty good.

Sherwin Wu 50:50 Mm.

Apoorv Agrawal 50:51 Do you just have it on? 'Cause I know your calendar is back to back. You just have Granola on [laughs] .

Olivier Godement 50:55 So the funny thing is that I don't use Granola internally. Uh-

Apoorv Agrawal 50:57 Oh, okay

Olivier Godement 50:57 ... I use Granola for my personal life mostly.

Apoorv Agrawal 50:59 I see.

Olivier Godement 50:59 Yeah, yeah.

Apoorv Agrawal 51:00 I see. On dates. I'm joking. [laughs]

Sherwin Wu 51:02 [laughs] Uh, I was gonna say, yeah, Granola's actually gonna be mine, so, uh-

Apoorv Agrawal 51:06 Yeah

Sherwin Wu 51:06 ... two votes for Granola. Um, I was gonna say the easy answer for me is Codex. Uh, as a-

Apoorv Agrawal 51:10 Yeah

Sherwin Wu 51:10 ... software engineer, it's just, like-

Apoorv Agrawal 51:11 Yeah

Sherwin Wu 51:12 ... so... it's gotten so good recently.

Apoorv Agrawal 51:13 Mm.

Sherwin Wu 51:13 Codex CLI, especially with GPT-5.

Apoorv Agrawal 51:16 Mm.

Sherwin Wu 51:16 Uh, especially for me, I tend to be less time sensitive about, like, uh, you know, the iteration loop with, with coding, and so-

Apoorv Agrawal 51:22 Mm-hmm

Sherwin Wu 51:23 ... uh, leaning into GPT-5 on Codex I think has been-

Apoorv Agrawal 51:24 Interesting

Sherwin Wu 51:25 ... really, really-

Apoorv Agrawal 51:25 He- what about Codex has changed? 'Cause, you know, Codex has also been through a journey. Codex has been around for a bit. I remember, like, it's been launched for, like, a y- more than over a year ago. So, like, what's, what's changed about Codex?

Sherwin Wu 51:35 I- yeah, I was actually gonna say-

Apoorv Agrawal 51:36 Codex and CLI has been around for a bit.

Sherwin Wu 51:38 So feel like it's been less than a year for Codex?

Olivier Godement 51:40 A few months I would say.

Sherwin Wu 51:41 Oh, okay.

Apoorv Agrawal 51:41 The time dilation is so crazy-

Sherwin Wu 51:42 Okay, okay, yeah

Apoorv Agrawal 51:42 ... in this field. It feels like it's been around for a very long time.

Olivier Godement 51:44 And a year ago was GPT-4o. Like, you remember that demo? [laughs]

Apoorv Agrawal 51:47 Yeah, yeah, yeah.

Olivier Godement 51:47 Like that feels like ages ago at this point.

Sherwin Wu 51:49 O1 hadn't even come out yet.

Olivier Godement 51:51 Yeah. [laughs]

Sherwin Wu 51:51 12 days of Christmas hadn't happened yet.

Olivier Godement 51:52 The voice demos.

Apoorv Agrawal 51:53 Maybe there's a naming thing. Okay, but anyway, yeah.

Olivier Godement 51:55 Oh, there was a Codex model.

Apoorv Agrawal 51:56 That's what I'm thinking about.

Sherwin Wu 51:57 There was, there was a Codex model.

Olivier Godement 51:57 We are, we are, yeah. We are.

Sherwin Wu 51:59 We had-

Olivier Godement 51:59 You're not, you're not to blame for that confusion. [laughs]

Sherwin Wu 52:01 Also, I think the-

Olivier Godement 52:02 We are

Sherwin Wu 52:02 ... the GitHub thing was called Codex for a bit-

Apoorv Agrawal 52:04 That's right

Sherwin Wu 52:04 ... as well. Yes, yes.

Apoorv Agrawal 52:05 That's right.

Sherwin Wu 52:05 Uh, but I'm talking about our, uh, coding product i- within ChatGPT-

Apoorv Agrawal 52:09 Yeah

Sherwin Wu 52:09 ... which is the Codex Cloud offering-

Apoorv Agrawal 52:11 Yeah

Sherwin Wu 52:11 ... and then also Codex CLI. So actually maybe if I were to narrow my answer a little bit more, it's Codex CLI, which I've, I've really, really liked.

Apoorv Agrawal 52:16 Yeah. Mm.

Sherwin Wu 52:16 I like the local environment set up.

Apoorv Agrawal 52:18 Mm.

Sherwin Wu 52:18 Um, the thing that's actually made it really useful in the last, I'd say, like, month or so is, uh, one, I think the team has done a really good job of just, like-

Apoorv Agrawal 52:25 Mm

Sherwin Wu 52:25 ... getting rid of all the paper cuts, like the small-

Apoorv Agrawal 52:27 Mm

Sherwin Wu 52:27 ... product polish-

Apoorv Agrawal 52:27 Mm

Sherwin Wu 52:27 ... and, like, paper cut things.

Apoorv Agrawal 52:28 Mm. Mm.

Sherwin Wu 52:28 It just, it's a... it kind of feels like a joy to use now.

Apoorv Agrawal 52:31 Nice.

Sherwin Wu 52:31 It feels more reactive. Uh, and then, uh, the second thing honestly is, is GPT-5. Like, I- I just think-

Apoorv Agrawal 52:36 Yeah

Sherwin Wu 52:36 ... GPT-5 really allows the product to shine.

Apoorv Agrawal 52:39 Yeah.

Sherwin Wu 52:39 Um, it's, you know, at the end of the day, this is kind of a, this is a product that really is dependent on the mo- underlying model.

Apoorv Agrawal 52:45 Mm-hmm. Mm-hmm.

Sherwin Wu 52:45 And when you have to, you know, like, iterate and go back and forth with the model, like, four or five times to get it right-

Apoorv Agrawal 52:50 Mm

Sherwin Wu 52:50 ... to get it to, like, you know, do the change that you want-

Apoorv Agrawal 52:53 Mm

Sherwin Wu 52:53 ... versus, uh, having it think a little bit longer and it just, like, one shots and does exactly what you want to do-

Apoorv Agrawal 52:57 Yeah

Sherwin Wu 52:58 ... you get this, like, weird, like, bionic feeling where you're like-

Apoorv Agrawal 53:00 Yeah

Sherwin Wu 53:00 ... I feel so mind melded with the model right now and-

Apoorv Agrawal 53:02 Yeah

Sherwin Wu 53:02 ... like, perfectly understands what I'm doing.

Apoorv Agrawal 53:04 Yeah.

Sherwin Wu 53:04 And so getting, like, that kinda, like, dopamine hit and, like, feedback loop constantly with Codex has made it a, a kind of, like, an indispensable thing that I really, really like.

Apoorv Agrawal 53:11 Nice.

Sherwin Wu 53:12 Um, and the other thing I'd say Codex is just really good for me is, um, uh, so I use it in my, uh, for, like, personal projects. Um, I also use it to, like, help me understand code bases, like, as a, like, as a, um, uh, engineering manager now I'm not as in the weeds on-

Apoorv Agrawal 53:25 Mm-hmm

Sherwin Wu 53:26 ... uh, uh, on, on the actual code. Uh, and so you're actually able to use Codex to really understand what's happening with the code base, um, have it, like, an- ask questions and have it answer about, uh-

Apoorv Agrawal 53:34 Mm-hmm

Sherwin Wu 53:34 ... things and really catch up to speed on things as well. So, like, even the non-coding use cases are really useful with Codex CLI.

Apoorv Agrawal 53:39 Fascinating. Sam had this tweet about Codex, um, usage, uh, ripping, I think, like, like, yesterday, so I wonder, uh, what's going on there. But you're, uh, you're not alone.

Sherwin Wu 53:51 Yeah. I, I think, I think I'm not alone just judging from the Twitter feedback. I think people are really realizing how great of a combination-

Apoorv Agrawal 53:56 Yeah

Sherwin Wu 53:56 ... Codex CLI and GPT-5 are. Um, yeah, I know that team is undergoing a lot of scaling challenges, but I mean, it, the system hasn't gotten down for me, so props to them.

Apoorv Agrawal 54:04 [laughs] Yeah.

Sherwin Wu 54:05 Um, but we are in a GPU crunch, so we'll-

Apoorv Agrawal 54:07 Yeah

Sherwin Wu 54:07 ... see how, you know, how long that goes.

Apoorv Agrawal 54:08 Awesome, awesome.

Sherwin Wu 54:10 Yeah.

Apoorv Agrawal 54:10 All right, the next one. Um, will there be more software engineers in 10 years or less?

Sherwin Wu 54:17 There's about, what, 40, 50 million-

Olivier Godement 54:18 Full-time professional software, software engineers? That's what you mean, like, full-time, like-

Sherwin Wu 54:23 Yeah, full-time

Olivier Godement 54:23 ... actual jobs?

Sherwin Wu 54:24 Yeah. Yeah, 'cause it's a hard one 'cause, like, I think without a doubt there's gonna be a lot more software engineering going on.

Olivier Godement 54:28 Yes. Of course.

Sherwin Wu 54:29 Um-

Olivier Godement 54:29 Of course

Sherwin Wu 54:29 ... there's actually a really great post that was shared I think in our internal Slack. It was, like, a Reddit post recently.

Olivier Godement 54:34 Yeah.

Sherwin Wu 54:34 I actually think that highlights this.

Olivier Godement 54:36 Mm.

Sherwin Wu 54:36 Um, it was a really touching story. It was a Reddit post about someone who has a brother who's non-verbal, and I actually don't know if you saw this. It was just posted. Um, it's a person on Reddit posted they have a non-verbal brother who they have to take care of.

Apoorv Agrawal 54:47 Mm. Mm. Mm.

Sherwin Wu 54:47 The brother, like, they tried all these types of things to help the brother interact with the world, use computers, but, like, vision tracking didn't work 'cause I think his vision wasn't good. Um, all the tools didn't work, and then, uh, this brother, um, uh, ended up using ChatGPT. I don't think he used Codex, but he used ChatGPT-

Apoorv Agrawal 55:02 Mm

Sherwin Wu 55:02 ... and basically taught himself how to create a set of tools that were tailor-made to his non-verbal brother.

Apoorv Agrawal 55:07 Mm.

Sherwin Wu 55:08 Basically a custom software application-

Apoorv Agrawal 55:09 Wow

Sherwin Wu 55:09 ... just for them, and because of that he, he now has this, like, custom setup that was written by his brother-

Apoorv Agrawal 55:15 Wow

Sherwin Wu 55:15 ... uh, and allows him to, like, browse the internet. I think the video was, like, him watching The Simpsons or something like that, which was really touching.

Apoorv Agrawal 55:21 Mm.

Sherwin Wu 55:21 But I think that's actually what we'll see a lot more of. Like, this guy's not a professional software engineer. His title is not software engineer, but he, like, did a lot of software engineering-

Apoorv Agrawal 55:29 Mm

Sherwin Wu 55:29 ... probably pretty good. Um, good enough definitely for his brother to use.

Apoorv Agrawal 55:32 Mm.

Sherwin Wu 55:32 So the amount of code, the amount of, like, building that'll, that'll happen I think is just gonna go through an in- incredible transformation.

Apoorv Agrawal 55:38 Right.

Sherwin Wu 55:38 I'm not sure what that means for software engineers like myself.

Apoorv Agrawal 55:41 Mm.

Sherwin Wu 55:41 Maybe there's, you know, equivalent or maybe there's even more-

Olivier Godement 55:43 Of course more Sherwin.

Sherwin Wu 55:44 Yeah, more of me.

Apoorv Agrawal 55:44 [laughs]

Sherwin Wu 55:45 More, more me specifically.

Olivier Godement 55:46 We need way more of you.

Apoorv Agrawal 55:46 That's right. [laughs]

Sherwin Wu 55:48 Yeah. But definitely a lot more software engineering and a lot of coding.

Apoorv Agrawal 55:50 Yeah.

Olivier Godement 55:50 I buy that completely. Like, I buy completely a thesis that there is a massive software shortage-

Apoorv Agrawal 55:55 Yeah

Olivier Godement 55:56 ... like, in the world. Like, we've been sort of, uh, accepting it-

Apoorv Agrawal 55:59 Saw it, yeah

Olivier Godement 55:59 ... you know, for the past 20 years, but, like, the goal of software was never to be that super rigid, super hard to build, you know, uh, artifact. It was to be, like, you know, customized, like, malleable.

Apoorv Agrawal 56:10 Yeah.

Olivier Godement 56:10 Uh, and so I expect that we'll see, like, way more... a sort of a, a reconfiguration of, like, people's, like, you know, job and skill set where way more people code. Like, you know, I expect that product managers are going to code, like, more and more for instance. Uh, but yeah-

Sherwin Wu 56:23 Yeah, you made, you made your PMs code recently if you remember.

Olivier Godement 56:25 Oh, yeah, we did that.

Apoorv Agrawal 56:26 [laughs]

Olivier Godement 56:26 That was very fun. Uh, we started, like, essentially not doing, like, PRDs, like product, uh-

Apoorv Agrawal 56:31 Wow

Olivier Godement 56:31 ... requirements documents. You know, classic PM thing.

Apoorv Agrawal 56:34 Yeah, yeah.

Olivier Godement 56:34 You write, like, five pages, like my product does X, et cetera, and you know, PMs have been basically write coding prototypes.

Apoorv Agrawal 56:39 Mm.

Olivier Godement 56:40 And one, it's pretty fast with GPT-5-

Apoorv Agrawal 56:43 Nice

Olivier Godement 56:43 ... and the Codex.

Sherwin Wu 56:44 Yeah, just a couple hours I think.

Olivier Godement 56:45 Freaking fast.

Sherwin Wu 56:46 Yeah.

Olivier Godement 56:46 And second, like, it sort of conveys, like, so much more information-

Apoorv Agrawal 56:50 Yeah

Olivier Godement 56:50 ... in a document.

Apoorv Agrawal 56:51 Yeah, yeah, yeah.

Olivier Godement 56:51 Like, you get a feel essentially for the feature, like, is it written right or not?

Apoorv Agrawal 56:55 Yeah.

Olivier Godement 56:55 So yeah, I expect that sort of, you know, behavior we're going to see more and more.

Apoorv Agrawal 56:59 Yeah. Instead of writing English, you can actually now write the actual thing you want.

Sherwin Wu 57:03 Yeah.

Olivier Godement 57:03 Yeah.

Apoorv Agrawal 57:03 Yeah. And the... Yeah, that's amazing. Advice for high school students who are just starting out their career.

Olivier Godement 57:09 My advice is-

Apoorv Agrawal 57:10 What's your advice?

Olivier Godement 57:11 I don't know, maybe it's evergreen. Like, prioritize critical thinking above anything else.

Apoorv Agrawal 57:18 Mm.

Olivier Godement 57:19 If you go in a field which requires, like, extremely high critical thinking, like, you know, skills, I don't know, math, physics, or, you know, maybe philosophy is in that bucket, you will be fine regardless.

Apoorv Agrawal 57:30 Mm.

Olivier Godement 57:30 If you go in a field that sort of turns down that thing, and again, gets back to, like, memorization, like, you know, pattern matching, I think you will probably be less future-proof-

Apoorv Agrawal 57:38 Yeah

Olivier Godement 57:38 ... as a, you know, uh-

Apoorv Agrawal 57:40 What's a good way to sharpen critical thinking?

Sherwin Wu 57:43 Um, use ChatGPT and-

Olivier Godement 57:45 [laughs]

Sherwin Wu 57:45 ... have it test you.

Olivier Godement 57:48 That's actually-

Apoorv Agrawal 57:48 Very good

Olivier Godement 57:49 ... having, like, you know-

Sherwin Wu 57:50 There you go

Olivier Godement 57:50 ... a world-class tutor who essentially knows how to put the bar, like, 20% above what you can do all the time-

Apoorv Agrawal 57:57 Yeah, yeah

Olivier Godement 57:58 ... you know, uh, is actually probably, like, a really good way to do it.

Apoorv Agrawal 58:02 Mm.

Sherwin Wu 58:02 Yeah.

Apoorv Agrawal 58:03 Nice. Anything from you, sir?

Sherwin Wu 58:05 Um, mine is, I, I think it's just... I think we're actually in such a interesting, like, unique time period where, um, the, like, younger... So, like, maybe this is more general advice for not just, like, high school students, but just, like, the younger generation, maybe even, like, college students.

Apoorv Agrawal 58:21 Mm-hmm.

Sherwin Wu 58:21 It's like, uh, I, I think the advice would be don't underestimate how much of an advantage you have relative to the rest of the world right now because-

Apoorv Agrawal 58:28 Mm

Sherwin Wu 58:28 ... of how AI native you might be or how-

Apoorv Agrawal 58:30 Interesting

Sherwin Wu 58:30 ... like, you know, in the, in the weeds of the tools you are.

Apoorv Agrawal 58:33 Mm.

Sherwin Wu 58:33 My hunch is, like, high schoolers, college students, when they come into the workplace-

Apoorv Agrawal 58:37 Mm, mm

Sherwin Wu 58:37 ... they're gonna have actually a huge leg up-

Apoorv Agrawal 58:39 Mm

Sherwin Wu 58:39 ... on, uh, how to use AI tools-

Apoorv Agrawal 58:41 Mm

Sherwin Wu 58:41 ... how to actually transform the workplace.

Apoorv Agrawal 58:43 Mm.

Sherwin Wu 58:43 And my push for, like, some of the younger, I guess, high school students is, like, one, like, just really immerse yourself in this thing.

Apoorv Agrawal 58:48 Yeah.

Sherwin Wu 58:48 And then two-

Apoorv Agrawal 58:49 Yeah

Sherwin Wu 58:49 ... just, like, really take advantage of the fact that you're, you're in a unique time where, like, no one else in the workforce really understands these tools as deeply probably as you do.

Apoorv Agrawal 58:56 Mm.

Sherwin Wu 58:56 Um, a good example of this, actually, we had our first intern class, uh, uh, recently at, at, at, at OpenAI.

Apoorv Agrawal 59:02 Mm.

Sherwin Wu 59:02 Um, a lot of, uh, software interns, uh, and some of them were just, like, the most incredible cursor power users I've, like, ever seen.

Apoorv Agrawal 59:08 Interesting.

Olivier Godement 59:09 They were so productive.

Sherwin Wu 59:10 Yeah.

Olivier Godement 59:10 I was, I was shocked-

Sherwin Wu 59:12 Yep

Olivier Godement 59:12 ... with the way.

Sherwin Wu 59:12 Yep.

Apoorv Agrawal 59:12 Yeah.

Sherwin Wu 59:12 Yep. I was like, "Yeah, I know, I know we can get good interns," but, like, I don't know they'd be, like, this good.

Apoorv Agrawal 59:16 Yeah.

Olivier Godement 59:16 Yeah.

Sherwin Wu 59:17 Um, and I think part of it is just, like, they've grown up, um, using these tools, for better or worse, in college.

Apoorv Agrawal 59:22 Yeah.

Sherwin Wu 59:22 Um, but I think the meta level point is they- they're, they're so, like, AI native and even, like, I don't know, me and Olivier, we're, like, kind of AI native.

Apoorv Agrawal 59:29 Yeah.

Olivier Godement 59:29 Yeah.

Sherwin Wu 59:29 We work at OpenAI, but, like, we haven't, like, been steeped in this-

Apoorv Agrawal 59:32 Yeah, yeah, yeah

Sherwin Wu 59:32 ... uh, and kinda grown up in this. And so, uh, the, the advice here would just be like, yeah, leverage that. Like-

Apoorv Agrawal 59:37 Yeah, yeah

Sherwin Wu 59:37 ... you know, don't be afraid to kinda, like, go in and spread this knowledge and take advantage of it in the workplace because it is a pretty big advantage for them.

Apoorv Agrawal 59:43 Yeah. I can't remember who said this to us at Palantir, but every intern class was just getting faster, smarter, like laptops, like smarter every generation.

Sherwin Wu 59:52 You sure it didn't peak in 2013 with-

Olivier Godement 59:53 [laughs]

Sherwin Wu 59:53 ... you know, when I was an intern?

Apoorv Agrawal 59:54 That's right. [laughs] That's right.

Olivier Godement 59:56 It's a weird spike.

Sherwin Wu 59:57 [laughs]

Apoorv Agrawal 59:57 [laughs]

Olivier Godement 59:57 Yeah.

Sherwin Wu 59:58 That summer of 2013, yeah.

Olivier Godement 59:59 Two guys like you.

Apoorv Agrawal 1:00:00 That's right. [laughs]

Olivier Godement 1:00:00 Two outliers. [laughs]

Apoorv Agrawal 1:00:01 That's right. That's right. That's right.

Sherwin Wu 1:00:04 Yeah.

Apoorv Agrawal 1:00:04 Well, lots happened, you know, lots happened since you guys joined OpenAI, right? What, three years in- almost three years. Um, in your OpenAI journey, what has been the, the rose moment, your favorite moment?

Olivier Godement 1:00:15 Mm.

Apoorv Agrawal 1:00:15 The bud moment, where you're, like, most excited about something but, but still opportunity ahead, and the thorn, toughest moment of your, of your, of your, uh, three-year journey.

Olivier Godement 1:00:24 The thorn is easy for me.

Apoorv Agrawal 1:00:25 Mm.

Olivier Godement 1:00:25 What we call the blip-

Apoorv Agrawal 1:00:26 [laughs]

Olivier Godement 1:00:27 ... which is, you know, the coup of the board.

Apoorv Agrawal 1:00:29 Yeah.

Olivier Godement 1:00:29 Like, that was a really tough moment.

Apoorv Agrawal 1:00:31 Yeah.

Olivier Godement 1:00:31 It's funny 'cause, you know, after the fact, it actually reunited quite a bit the company.

Apoorv Agrawal 1:00:34 Yeah.

Olivier Godement 1:00:34 Like, there was a feeling... OpenAI had a pretty strong culture before.

Apoorv Agrawal 1:00:38 Mm.

Olivier Godement 1:00:39 But, you know, there was a feeling of, like, camaraderie, essentially-

Apoorv Agrawal 1:00:41 Yeah

Olivier Godement 1:00:41 ... that was even stronger.

Apoorv Agrawal 1:00:42 Yeah.

Olivier Godement 1:00:42 But, you know, sure, like, tough on the day of. Uh-

Apoorv Agrawal 1:00:45 It's very rare to see that anti-fragility. Most orgs after something like that break, break apart, but I feel like OpenAI got stronger. OpenAI came back.

Olivier Godement 1:00:53 It's a good point. I feel it made OpenAI stronger for real now-

Apoorv Agrawal 1:00:57 Yeah

Olivier Godement 1:00:57 ... essentially when I look at other fact.

Apoorv Agrawal 1:00:59 Yeah, yeah.

Olivier Godement 1:00:59 When I look at, you know, other, like, you know, news, like departures or, you know, whatever, like, you know, bad news essentially, I feel the company has built, like, you know, a thicker skin and, you know-

Apoorv Agrawal 1:01:09 Yeah

Olivier Godement 1:01:09 ... an ability to, like, recover, like-

Apoorv Agrawal 1:01:11 Yeah

Olivier Godement 1:01:11 ... way quicker.

Apoorv Agrawal 1:01:12 Yeah, yeah.

Sherwin Wu 1:01:12 I actually think part... I think that's definitely right. Uh, part of it, too, I think, is also just the culture. I also think this is why it was such a, a low point for a lot of people. So many people just at OpenAI care so deeply about what we're doing, which is why they work so hard.

Apoorv Agrawal 1:01:24 Yeah.

Sherwin Wu 1:01:24 Um, you just care a lot about the work. It almost feels like your life's work.

Apoorv Agrawal 1:01:27 Yeah.

Sherwin Wu 1:01:27 Like, it's a very-

Apoorv Agrawal 1:01:28 Yeah

Sherwin Wu 1:01:28 ... audacious mission-

Apoorv Agrawal 1:01:28 Yeah, yeah

Sherwin Wu 1:01:29 ... and thing that-

Apoorv Agrawal 1:01:29 Yeah

Sherwin Wu 1:01:29 ... you're doing, which is why I think the blip was, like, so tough on a lot of people but also is what I think helped bring people back together-

Apoorv Agrawal 1:01:35 Yeah

Sherwin Wu 1:01:36 ... and why we were able to hold together and, and get that, that thick skin as well.

Apoorv Agrawal 1:01:39 Yeah.

Olivier Godement 1:01:39 Yeah.

Sherwin Wu 1:01:40 I have, I have a separate, uh, worse moment-

Apoorv Agrawal 1:01:42 Mm

Sherwin Wu 1:01:42 ... uh, which was, uh, the big outage that we had in, uh, December of last year if you remem- yeah, you remember.

Apoorv Agrawal 1:01:47 I do. [laughs]

Sherwin Wu 1:01:48 Uh, it was like a multi-hour, uh, outage. Um, really highlights to, to, uh, to us how, uh, essential of almost like a utility the API was.

Apoorv Agrawal 1:01:57 Mm.

Sherwin Wu 1:01:57 Uh, so, so the background is I think we had, like, a, like a three, four-hour outage, uh-

Apoorv Agrawal 1:02:00 Mm

Sherwin Wu 1:02:00 ... sometime in, uh, November or December of last year.

Apoorv Agrawal 1:02:02 Mm.

Sherwin Wu 1:02:03 Um, really brutal. Pure Sev zero. No one could hit ChatGPT and no one could hit the APIs. It was, it was really, really rough.

Apoorv Agrawal 1:02:09 Mm.

Sherwin Wu 1:02:09 Um, that was just really tough just from a, like, you know, customer trust perspective. I remember we, we, like, talked to a lot of our customers to kinda, like, um, uh, postmortem them on what happened-

Apoorv Agrawal 1:02:18 Yeah

Sherwin Wu 1:02:18 ... and kind of our plan moving forward. Thankfully, we haven't had anything [laughs] close to that-

Apoorv Agrawal 1:02:22 Yeah, yeah

Sherwin Wu 1:02:22 ... uh, since then, and I, I've been actually really happy with, um, all the investments we've made in reliability over the last, uh, six months. Um, uh, but, uh, in that moment I think it was really, it was really tough.

Apoorv Agrawal 1:02:34 Yeah.

Olivier Godement 1:02:34 On the happy side, like on the roses, um, I think I have two of them. The first one would be GPT-5 was really good. Like, the-

Apoorv Agrawal 1:02:44 Mm

Olivier Godement 1:02:44 ... sprint up to GPT-5 I think really, like, you know, showed, like, the best of OpenAI.

Apoorv Agrawal 1:02:50 Mm.

Olivier Godement 1:02:50 Like, you know, having, like, cutting edge, like, science research, like, you know, extreme, like, customer focus, like, you know, extreme, like, you know, infrastructure and inference, like, you know, talent. Uh,

Olivier Godement 1:03:00 and the fact that we were able, like, to ship, like, such a big model and, like, scale it to, like, you know- Many, many, many, many tokens, you know-

Sherwin Wu 1:03:08 Mm

Olivier Godement 1:03:08 ... per minute, like almost like immediately I think speaks to it.

Sherwin Wu 1:03:12 Mm.

Olivier Godement 1:03:12 So that one I really-

Sherwin Wu 1:03:13 With no, with no outages and-

Olivier Godement 1:03:14 With no outages.

Sherwin Wu 1:03:15 Yeah, really good reliability. Yep.

Olivier Godement 1:03:17 Like I remember when we shipped like GPT-4 Turbo like a year ago-

Sherwin Wu 1:03:20 [laughs]

Olivier Godement 1:03:20 ... a year and a half ago, we were terrified by, you know, like the-

Sherwin Wu 1:03:23 Yeah, the scale

Olivier Godement 1:03:24 ... the influx of, uh, traffic.

Sherwin Wu 1:03:25 Yeah.

Olivier Godement 1:03:25 And I feel we've really gotten like much better at, you know-

Sherwin Wu 1:03:27 Mm

Olivier Godement 1:03:27 ... shipping like those, you know, massive updates. Um, the second, uh, rose, like you know, happy moment for me would be the first DevDay was really fun.

Sherwin Wu 1:03:38 Yeah.

Olivier Godement 1:03:38 It felt like a coming of age, like OpenAI, like you know, we are embracing that, you know, we have like a huge community developers. You know, we are going to ship models, new products. And I remember basically seeing like, you know, all my favorite people, OpenAI or not, you know, like in a room essentially nerding out on, you know, what are you building? Like, you know, what's coming up next?

Sherwin Wu 1:03:56 Yeah.

Olivier Godement 1:03:56 It felt really like, you know, a special moment, uh, in time.

Sherwin Wu 1:03:59 No, that was actually gonna be mine as well.

Olivier Godement 1:04:01 Yeah.

Sherwin Wu 1:04:01 So I'll just piggyback off of that, which is the very first DevDay 2023-

Olivier Godement 1:04:04 Yeah

Sherwin Wu 1:04:05 ... uh, November. Uh, I actually, I remember it... So, I mean, obviously a lot of good things have happened since then. There's just a very... I don't know why, for, for me, it was a very memorable moment, which was-

Olivier Godement 1:04:14 Mm

Sherwin Wu 1:04:14 ... uh, one, uh, it was actually quite a rush up to DevDay. We were, we shipped a lot.

Olivier Godement 1:04:18 Mm.

Sherwin Wu 1:04:18 So our team was just really, really sprinting.

Olivier Godement 1:04:20 Mm.

Sherwin Wu 1:04:20 So it was like this high, you know-

Olivier Godement 1:04:22 Mm

Sherwin Wu 1:04:22 ... uh, uh, high-stress environment kind of going up.

Olivier Godement 1:04:24 Mm.

Sherwin Wu 1:04:24 To add to that, um, you know, of course, because we're OpenAI, we did a live demo on Sam's, uh, in Sam's keynote of all-

Olivier Godement 1:04:30 Mm

Sherwin Wu 1:04:30 ... the stuff that we shipped.

Olivier Godement 1:04:31 Mm.

Sherwin Wu 1:04:31 And I just remember being in the back of the audience sitting with like the team-

Olivier Godement 1:04:35 Mm

Sherwin Wu 1:04:35 ... and like waiting for the demo to happen. Once it finished happening, we all just like let out a huge sigh of relief.

Olivier Godement 1:04:39 [laughs]

Sherwin Wu 1:04:39 We were like, "Oh my God, thank you." Um, uh, and so there's, I think there's just like a lot of like, you know-

Olivier Godement 1:04:44 Yeah

Sherwin Wu 1:04:44 ... buildup to it. For me, the most memorable thing was I remember right after DevDay, all the demos worked well, all the talks worked well. We had the after party, and then I was just in a Waymo driving home at night-

Olivier Godement 1:04:54 Mm

Sherwin Wu 1:04:54 ... with the music playing.

Olivier Godement 1:04:55 Mm.

Sherwin Wu 1:04:55 And it was just like such a great-

Olivier Godement 1:04:56 [laughs] Yes

Sherwin Wu 1:04:57 ... end to the DevDay. That was, that was what I remember. That was my rose for the last, uh, last couple years.

Olivier Godement 1:05:00 Love it.

Sherwin Wu 1:05:01 Yeah.

Olivier Godement 1:05:01 Love it.

Apoorv Agrawal 1:05:01 That's awesome. I assume you guys are, but please tell me if you're AGI-pilled, yes or no. And if so, what was the moment that got you there? What was your aha moment? When did you feel the AGI?

Olivier Godement 1:05:12 I think I'm AGI-pilled. I think I'm AGI-pilled.

Sherwin Wu 1:05:14 You're definitely AGI-pilled.

Olivier Godement 1:05:15 I am? Okay. [laughs]

Sherwin Wu 1:05:18 [laughs]

Olivier Godement 1:05:19 Uh, I've had a couple of them. The first one was the realization in 2023 that I would never need to code manually like ever, ever again.

Sherwin Wu 1:05:28 [laughs]

Olivier Godement 1:05:28 Like I'm not, I'm not the best coder, frankly, you know. I chose my job like, you know, for a reason.

Sherwin Wu 1:05:32 Yeah.

Olivier Godement 1:05:32 Um, but realizing that, you know, what I thought was a given that we humans would have like to write like basically machine language like forever-

Sherwin Wu 1:05:41 Yeah

Olivier Godement 1:05:41 ... is actually not a given.

Sherwin Wu 1:05:43 Yeah.

Olivier Godement 1:05:43 And that, you know, the pace of price is huge feeling the AGI. Um, the second like feel the AGI moment for me was, uh, maybe the progress on voice and multimodality.

Sherwin Wu 1:05:55 Mm.

Olivier Godement 1:05:55 Like, you know, text, like at some point you get used to it.

Sherwin Wu 1:05:57 Yeah.

Olivier Godement 1:05:57 Like, okay, you know, the machine can write pretty good text.

Sherwin Wu 1:06:00 Yeah. Voice makes it real.

Olivier Godement 1:06:01 But once you start actually talking like-

Sherwin Wu 1:06:03 Yeah

Olivier Godement 1:06:03 ... you know, to something that actually understands your tone, like, you know, understand my accent like in French-

Sherwin Wu 1:06:07 [laughs]

Olivier Godement 1:06:09 ... um, it felt like sort of, um, a Twitter moment. Like, okay, machines are going beyond like cold, mechanical-

Sherwin Wu 1:06:18 Mm

Olivier Godement 1:06:18 ... deterministic, like, you know-

Sherwin Wu 1:06:20 Mm

Olivier Godement 1:06:20 ... like logic to something-

Sherwin Wu 1:06:21 Mm

Olivier Godement 1:06:21 ... like much more like emotional and like, you know, tangible.

Apoorv Agrawal 1:06:25 Yeah. That's a great one.

Sherwin Wu 1:06:27 Yeah, mine, mine are, uh, uh... So I, I do think I'm [laughs] I, I am AGI-pilled.

Apoorv Agrawal 1:06:32 Yes.

Sherwin Wu 1:06:32 I probably gradually became-

Apoorv Agrawal 1:06:33 Definitely

Sherwin Wu 1:06:33 ... AGI-pilled over the last couple years. I think there are two, and, and for me, um, yeah, I think I, I, I actually get more, get more shocked from the text models. I know the multimodal ones are, are really great as well. For me, I think they actually line up with, with two like general breakthroughs.

Apoorv Agrawal 1:06:47 Mm.

Sherwin Wu 1:06:47 So, um, the first one was right when I joined the company in September 2022.

Apoorv Agrawal 1:06:52 Mm.

Sherwin Wu 1:06:52 We... It was pre-ChatGPT.

Apoorv Agrawal 1:06:53 Yeah, two months later, yeah.

Sherwin Wu 1:06:55 Um, but at the time, GPT-4 already existed internally.

Apoorv Agrawal 1:06:57 Mm.

Sherwin Wu 1:06:58 And I think we were trying to figure out how to deploy. I think Nick Turley's talked about this a lot, um, early days of ChatGPT. But it was the first time I talked to GPT-4, and it was, it was like going from nothing to GPT-4 was just-

Apoorv Agrawal 1:07:09 Mm-hmm

Sherwin Wu 1:07:09 ... the most mind-blowing experience for me.

Apoorv Agrawal 1:07:11 Yeah.

Sherwin Wu 1:07:11 I think for the rest of the world, maybe going from nothing to GPT-3.5 in chat was-

Apoorv Agrawal 1:07:15 Mm

Sherwin Wu 1:07:15 ... maybe the big one, and then going from 3.5 to 4. But for me, and I think for a lot of maybe some other people who joined around that time, going from nothing to... or, or not nothing, but like what was publicly available at the time-

Apoorv Agrawal 1:07:26 Mm

Sherwin Wu 1:07:26 ... going from that to GPT-4 was just incredible. Like I just remember asking, throwing so many things at it. I was like, "There's no way this thing is gonna be able to give an intelligible answer."

Apoorv Agrawal 1:07:34 Mm.

Sherwin Wu 1:07:34 And it just like knocks it out of the park. It was, it was absolutely incredible.

Olivier Godement 1:07:37 GPT-4 was insane. I remember GPT-4 came out when I was interviewing with OpenAI, and I was still like on the fence-

Sherwin Wu 1:07:42 Yeah

Olivier Godement 1:07:43 ... should I join, should I join? I saw that thing, and I was like, "Okay."

Sherwin Wu 1:07:45 [laughs]

Olivier Godement 1:07:46 "I'm in."

Apoorv Agrawal 1:07:46 Let's go.

Olivier Godement 1:07:46 "I'm in, guys." [laughs]

Apoorv Agrawal 1:07:47 Let's go.

Olivier Godement 1:07:47 "There is no way I can work on anything else at that point."

Apoorv Agrawal 1:07:49 Let's go.

Sherwin Wu 1:07:50 Yep, yep, yeah.

Olivier Godement 1:07:50 [laughs]

Sherwin Wu 1:07:51 Yeah, so GPT, yeah, GPT-4 was just crazy. Um, and then, uh, the other one was, uh, like is the other breakthrough, which is like the reasoning paradigm. I actually think the, the purest represent- representation of that for, for me was deep research.

Olivier Godement 1:08:03 Mm.

Sherwin Wu 1:08:03 Uh, and throwing like, like asking it to like really look up things that-

Olivier Godement 1:08:08 Mm

Sherwin Wu 1:08:08 ... I didn't think it would be able to know, and seeing it like think through all of it, like be really-

Olivier Godement 1:08:12 Yeah

Sherwin Wu 1:08:12 ... persistent with the search-

Olivier Godement 1:08:13 Mm

Sherwin Wu 1:08:13 ... uh, get really detailed with the write-up and all of that.

Olivier Godement 1:08:16 Mm.

Sherwin Wu 1:08:16 That was pretty, pretty crazy. I don't remember the exact query that I threw it, but I just remember like I feel like the feel the AGI moments for me are like I'll throw something at the model that I, I was like, "There's no way this thing will be able to get."

Apoorv Agrawal 1:08:25 Yeah. Yeah, yeah.

Sherwin Wu 1:08:25 And then it just like knocks it out of the park.

Apoorv Agrawal 1:08:27 Yeah.

Sherwin Wu 1:08:27 Like that is kind of the feel the AGI moment. I definitely had that with deep research with some of the things I was asking it.

Apoorv Agrawal 1:08:31 Yeah. Well, this has been great. Thank you so much, folks. You guys are building the future. You guys are inspiring us every day, and appreciate the conversation.

Olivier Godement 1:08:42 Yeah.

Sherwin Wu 1:08:42 Thank you so much.

Olivier Godement 1:08:43 Thank you.

Sherwin Wu 1:08:43 Thanks for having us.

Olivier Godement 1:08:52 Thank you. [outro music]

Apoorv Agrawal 1:08:53 As a reminder to everybody, just our opinions, not investment advice.