ChatGPT – The Super Assistant Era | BG2 Guest Interview

Nick Turley 0:00 ChatGPT originally was entirely free, and the reason for that was that it was intended to be a demo.

Apoorv Agrawal 0:06 Yeah.

Nick Turley 0:06 And we were gonna wind it down after a month.

Apoorv Agrawal 0:08 Mm-hmm.

Nick Turley 0:08 We then realized that the demo went viral, and people loved the demo, and it was actually a product. And but we realized for it to be a product, you can't take the product down every time you're at capacity. So we, you know, shipped subscriptions simply 'cause it could shape the demand. It was a way of gracefully turning users away when we had to turn away someone.

Apoorv Agrawal 0:26 You know, you guys are at 900 million weekly active users now, and that growth has been incredible. The next billion users, where are they, where are they gonna come from?

Nick Turley 0:35 We've got about 10% of the world coming to us now. 90% left to go, right? There's so much more opportunity.

Apoorv Agrawal 0:48 Well, Nick, so excited to have you here.

Nick Turley 0:50 Thank you for having me, Apoorv.

Apoorv Agrawal 0:51 You've had, uh, quite the journey from Germany to the US for Brown.

Nick Turley 0:56 That's true.

Apoorv Agrawal 0:57 Most recently at Instacart delivering, uh, groceries in

Apoorv Agrawal 1:00 30 minutes. You're now delivering AGI to billions.

Nick Turley 1:02 [laughs]

Apoorv Agrawal 1:02 I'm sure that was the plan all along. Uh-

Nick Turley 1:04 Yeah, clearly total master plan.

Apoorv Agrawal 1:06 [laughs] Well, well tell us about your journey. What-- How did you get to OpenAI? I know it's a fun, fun story, and you're, you're three and a half years or so at OpenAI. How have they gone?

Nick Turley 1:16 The only through line in how I've made any sort of employment decision has been entirely people-based. So I, I don't claim any credit for, for, uh, joining OpenAI, um, or predicting ChatGPT or anything like it. But, um, I, uh, um, hit up someone who I admire a lot who I got to, got to know at Dropbox, um, Joanne, who, uh, worked, worked here at the time. And she, uh-- I asked her to get off the DALL-E 2 wait list, and she told me I had to interview if I wanted to get off the wait list. So, um, I took the bait and got totally nerd sniped in the process, and, and here I am.

Apoorv Agrawal 1:52 There you go. The DALL-E 2 wait list will get you.

Nick Turley 1:55 It's a great recruiting tool.

Apoorv Agrawal 1:56 Nice. Nice, nice, nice.

Nick Turley 1:57 We should do more wait lists probably.

Apoorv Agrawal 1:58 Yeah. Yeah, yeah. Well, you know, the big, uh, the big super cycle we're in is, is, is ChatGPT. Now, I assume over a billion users on the monthly side, 900 million weekly active users recently reported up from zero

Apoorv Agrawal 2:15 three and a half years ago. Uh, you could have-- If I imagine what the dashboard of Nick Turley looks like, it could have users, it could have paying subscribers, it could have daily active users, it could have retention engagement. I mean, there, there's like 15 things, maybe all of them. What, what is your North Star? How do you, how do you op-- What are you optimizing for? Uh, what is Nick looking at on, on his daily dashboard?

Nick Turley 2:37 It's, it's funny, right? Because it's such a young product. It's been, to your point, three and a half years. And, um, this kind of question, it, it kind of changes as, as, as you evolve and you, and, and, and you grow up and you ask yourself, you know, "What are we really building here?" And, uh, to this day, right, I wanna build a super assistant that can actually help people achieve their goals.

Apoorv Agrawal 2:59 Hmm.

Nick Turley 2:59 And ultimately, the thing we care about is like, is our product doing that? Is it actually helping you do the thing-

Apoorv Agrawal 3:04 Hmm

Nick Turley 3:04 ... that you're, you know, um, coming to the product to do? And it's so different for different people, right? Some people, um, are, are trying to get healthy. Other people are trying to start a company.

Apoorv Agrawal 3:13 Hmm.

Nick Turley 3:13 Um, um, learn a new topic, um, do their taxes. You know, there's all these different things that you might be doing. Um, and the true measure of success is whether or not we're helping you do that. And, um, obviously we look at WAU in particular because, you know, we wanna know if you're coming back to the product. We look at retention. Um, but we, you know, we, we, we look at, um, all kinds of stuff in aggregate because really there isn't like this one single thing that you can optimize for.

Apoorv Agrawal 3:39 If you were to allocate 100 units of points-

Nick Turley 3:41 Mm-hmm

Apoorv Agrawal 3:41 ... to these metrics, which metric-- Can you like distribute the 100 units across these metrics in order of importance for you right this second?

Nick Turley 3:49 It's a good question. I, I care a lot about long-term retention, and I would put all my points there-

Apoorv Agrawal 3:54 Yeah

Nick Turley 3:54 ... um, because, um, I'm, I'm really proud of the retention stats we have.

Apoorv Agrawal 3:58 Huge.

Nick Turley 3:58 But ultimately the sign of durable value is whether or not people are coming back in three months because that means you're really solving their problems. And-

Apoorv Agrawal 4:04 Yeah

Nick Turley 4:04 ... I think things like revenue, they follow from that, um-

Apoorv Agrawal 4:07 Yeah

Nick Turley 4:07 ... um, versus like, you know, trying to go on those things directly. And we've, we've had a lot of success making very principled decisions,

Nick Turley 4:15 um, on this stuff. Like one good example is, you know, GPT-4 used to be behind a pay, paywall because we couldn't serve it to everyone. And then we had GPT-4o, which was a total breakthrough-

Apoorv Agrawal 4:24 Hmm

Nick Turley 4:24 ... in, um, in, in our ability to inference it. And, um, so we just gave it away for free, and that ended up being totally revenue positive-

Apoorv Agrawal 4:33 Hmm

Nick Turley 4:33 ... and retention positive because it just provided access to the tech. And-

Apoorv Agrawal 4:36 Hmm

Nick Turley 4:36 ... I think when you make your decisions that way and you focus on the customer-

Apoorv Agrawal 4:39 Hmm

Nick Turley 4:39 ... you end up with a great product and revenue obviously follows too.

Apoorv Agrawal 4:43 Phenomenal.

Nick Turley 4:43 Yeah.

Apoorv Agrawal 4:44 Well, it, uh, it shows up in the numbers. You know, I, I posted this chart, uh, yesterday on the data that we have, you know, from a third party. The retention curves for ChatGPT are smiling. Look at that. Just like that. And that is a rare-- That is a very rare occurrence, you know, as, as we know. And, um, why, why do you think-- Like if you were to give us a narrative on that smile curve, what is the-- Why do these smile curves exist? What are you seeing in ChatGPT that has people who have, who have maybe churned off for a couple of weeks or months coming back, a-and why are they coming back?

Nick Turley 5:17 Look, there isn't one single thing.

Apoorv Agrawal 5:19 Yeah.

Nick Turley 5:19 You know, the way that you build a retentive product is lots and lots of little things and really trying to make it better systematically. I will say that, you know, with AI and in particular ChatGPT, I've found that it takes people some time to really understand all the parts of their life they can delegate, right? And I think many users for that, it, it's a multi-month process for them to understand-

Apoorv Agrawal 5:40 Hmm

Nick Turley 5:40 ... how can this thing help me and what are all the different ways-

Apoorv Agrawal 5:43 Mm-hmm

Nick Turley 5:43 ... that, um, I could plug ChatGPT into my life. Um, and, um, but, you know, when I think about some of the breakthroughs and levers we've had, um, things like search and personalization-

Apoorv Agrawal 5:55 Hmm

Nick Turley 5:55 ... they, they have helped solve those user problems because, um, you know, search provides way more daily value to you.

Apoorv Agrawal 6:00 Hmm.

Nick Turley 6:01 Um, it used to be that ChatGPT was a pretty worky product. You know, we'd-

Apoorv Agrawal 6:04 Right

Nick Turley 6:04 ... see usage go down on the weekend. We'd see usage, you know, go down during the summer months when a lot of people were off from work.

Apoorv Agrawal 6:10 Hmm.

Nick Turley 6:10 And, um- Today, you know, we're, we're mobile first. Uh, the vast majority of us is just mo-mobile.

Apoorv Agrawal 6:16 Mm.

Nick Turley 6:16 And we see all these personal use cases, and I think search was a big investment that got us there.

Apoorv Agrawal 6:20 Mm.

Nick Turley 6:20 And personalization makes ChatGPT so much more relevant for you, right? Because it gets to know you over time, you get to know it. Um, and um, those are two things that have materially moved, um, the way that, you know, people come back to the product. But, um, there's lots more to do.

Apoorv Agrawal 6:35 Yeah.

Nick Turley 6:36 Um, and you know, as I mentioned, I don't-- I'm, I'm not resting on our, um, retention stats even though, um, we're obviously very proud.

Apoorv Agrawal 6:42 Nice. Nice. Nice. Um,

Apoorv Agrawal 6:45 and you know, the other thing that I, I got wrong about ChatGPT was this is two and-- two, two and a half years ago, and I was like, "Well, you know, let's look at who's gonna win this consumer AI race."

Nick Turley 6:56 Mm-hmm.

Apoorv Agrawal 6:56 You-- typically these consumer markets are winner take most, winner take all.

Nick Turley 6:59 Mm.

Apoorv Agrawal 6:59 Look at search. Google has near ninety percent plus market share, three and a half, four trillion in market cap. Mobile, same thing with Apple. Social, same thing with Meta. I was like, well, AI, Meta has all the distribution. Google's got all the distribution. They've got three, four billion users. Well, it'll be a flick of a switch for them to roll out their AI. But I was wrong. That's not what happened. ChatGPT turns out, uh, you know, you guys are at nine hundred million weekly active users now, and um, that growth has been incredible. Clearly, distribution was not enough, right?

Nick Turley 7:30 Mm.

Apoorv Agrawal 7:30 So the same question for distribution, what are the levers for us that have gotten us to the scale? Is it model quality? Is it, you know, product quality? Is it, you know, features? Is it the experience or product improvements like memory and personalization or search?

Nick Turley 7:44 Mm-hmm. Mm-hmm.

Apoorv Agrawal 7:45 Like, same question. Like, what would you say drove historical growth and, and, and success?

Nick Turley 7:50 We've got about ten percent of the world coming to us now.

Apoorv Agrawal 7:53 Wow.

Nick Turley 7:53 Ninety percent left to go, right? There is so much more opportunity to, to, to, to reach, reach more people and introduce them to the way that AI can, can, can benefit, benefit them, right? But when I, when I look backwards, and I only say that because, like, the next billion users might be very different in terms of, like, how you-

Apoorv Agrawal 8:11 Sure

Nick Turley 8:11 ... um, engage and reach and-

Apoorv Agrawal 8:13 Mm-hmm

Nick Turley 8:13 ... provide value. But when I look backward, it's been roughly a sort of one-third, one-third, one-third between sort of classic friction removal type of work. Like, one of the biggest, um, um, moments when you look at pure impact, um, was, you know, removing the authentication wall.

Apoorv Agrawal 8:32 Mm.

Nick Turley 8:32 And Sam will say, "I told you so," because I think that was his feedback from, like, day one.

Apoorv Agrawal 8:36 Mm.

Nick Turley 8:36 Um, it was like, you can't-- you shouldn't have to log into ChatGPT. But it's, like-

Apoorv Agrawal 8:40 Yeah

Nick Turley 8:40 ... stuff like that, that you do for any product, and it, it, it does matter.

Apoorv Agrawal 8:43 Yeah.

Nick Turley 8:43 Some, some things never change, right?

Apoorv Agrawal 8:45 Mm.

Nick Turley 8:46 Um, and but you know, and then, you know, another third or so is sort of, you know, are what I would sort of, sort of core product investments, and they're really typically things that we've done together between research and product.

Apoorv Agrawal 8:57 Oh.

Nick Turley 8:58 So search and personalization are really good examples of that, where we came together and we figured out not just UI/UX evolution, but also how to post-train, um, these changes into the model, and it was really the moments when we came together. Um, um, another recent example is, like, we have these writing blocks that render when you, you know, ask about queries, um, where you're, um, you know, trying to write with the model.

Apoorv Agrawal 9:19 Mm.

Nick Turley 9:19 And, like, putting really good craft into those experiences really matters and, um, um, you know, our users love it. And then-

Apoorv Agrawal 9:27 Mm

Nick Turley 9:27 ... another third of, of the growth has been just model improvements.

Apoorv Agrawal 9:31 Mm.

Nick Turley 9:31 Like step-- both step changes-

Apoorv Agrawal 9:32 Right

Nick Turley 9:32 ... like going from, um, GPT 3.5 back then to GPT-4, then going from GPT-4 behind a paywall to 4o-

Apoorv Agrawal 9:39 Yeah

Nick Turley 9:39 ... suddenly available to everyone, right?

Apoorv Agrawal 9:41 Yeah.

Nick Turley 9:42 But a lot of it is also the iteration that isn't splashy, that doesn't warrant like, you know, a named release. You know, we've, um-- I'm, I'm really excited about the updates we just made, um, um, with v-- 5.3, 5.4, et cetera.

Apoorv Agrawal 9:53 Mm-hmm. Mm-hmm.

Nick Turley 9:53 Because, um, that is when we, like, take a lot of user feedback and we, you know, methodically address it, and obviously that shows up, um, in our retention as well. So sort of one-third, one-third, one-third between classic, um, friction removal, um, and access-

Apoorv Agrawal 10:08 Mm

Nick Turley 10:08 ... um, core product investments, and then pure model improvements.

Apoorv Agrawal 10:12 And so the question that I've really been waiting to ask you is how

Apoorv Agrawal 10:15 do we get the next billion? [laughs]

Nick Turley 10:17 [laughs]

Apoorv Agrawal 10:17 And, and, you know, talk about that a little bit. There's a lot of-- it, it seems like, at least from the outside fog of war, if I was a consumer today in the market to pick my super assistant, um, you would have a couple of great options.

Nick Turley 10:30 Mm-hmm.

Apoorv Agrawal 10:31 You know, uh, Claude out there, they're, they're having some great traction the last couple of weeks. Um-

Nick Turley 10:37 Mm-hmm

Apoorv Agrawal 10:37 ... Gemini, mega distribution, Uber distribution, um, and us, right?

Nick Turley 10:42 Mm-hmm. Mm-hmm.

Apoorv Agrawal 10:43 Probably the, probably the, probably the leading product today, at least in user numbers.

Nick Turley 10:46 Mm-hmm.

Apoorv Agrawal 10:47 The next billion users-

Nick Turley 10:47 Mm-hmm

Apoorv Agrawal 10:48 ... where are they, where are they gonna come from?

Nick Turley 10:50 First of all, just to contextualize that goal, you know, we care about, um, two things at the end of the day. Obviously, reaching more people is really important. It's the direct manifestation of our mission, um, to the world, where, you know, the more people we can introduce to the benefits of AI-

Apoorv Agrawal 11:04 Mm

Nick Turley 11:04 ... uh, the better that is. Um, but we're also really excited to go deeper. Um, and that means taking the same billion users that find value in ChatGPT today, um, and actually providing more meaningful value in the world, like actually helping them achieve their goals, not just answering questions, right?

Apoorv Agrawal 11:21 Mm-hmm.

Nick Turley 11:21 Um, so I'll talk about how we get to more scale, but I think it's important to remember that, you know, the way this technology is evolving is, you know, the-- we're gonna go beyond pure chatbots, um, pretty fast.

Apoorv Agrawal 11:32 Ooh. Exciting.

Nick Turley 11:32 Um, I think on, on scale, you know, it's shocked me how many people have found value in ChatGPT as it works today because I don't think delegation is a natural skill for most.

Apoorv Agrawal 11:44 Mm.

Nick Turley 11:44 And ChatGPT is a pretty... You know, it, it, it's a, it's a power tool, right?

Apoorv Agrawal 11:50 Mm-hmm.

Nick Turley 11:50 You come to it, it doesn't tell you what it's for.

Apoorv Agrawal 11:52 Yeah.

Nick Turley 11:53 You kind of have to discover it on your own-

Apoorv Agrawal 11:54 Yeah

Nick Turley 11:54 ... and you have to use it, and then you'll learn about this prompt that was really cool, and then maybe you-

Apoorv Agrawal 11:58 Yeah

Nick Turley 11:58 ... you're on Twitter and you learn about another one, or you're on Instagram and you learn another one. But the product, it, it's, it's like a raw appliance, and I think for-

Apoorv Agrawal 12:05 Mm

Nick Turley 12:05 ... to-- you know, on-- one thing we really need to nail as we, you know, reach the next set of users is a product that has a bit more of an affordance.

Apoorv Agrawal 12:12 Mm.

Nick Turley 12:12 Um, because I think for most people, they're very, very busy.

Apoorv Agrawal 12:14 Mm.

Nick Turley 12:15 Um, and they-- everyone, I think, in the world has intelligence-constrained problems, like problems that-

Apoorv Agrawal 12:20 Yeah

Nick Turley 12:20 ... more intelligence could help with.

Apoorv Agrawal 12:21 Yeah.

Nick Turley 12:21 But you need to frame that to people.

Apoorv Agrawal 12:23 Yeah.

Nick Turley 12:23 Uh, and I still feel like we're a little bit Too much like a computer terminal-

Apoorv Agrawal 12:27 Mm

Nick Turley 12:27 ... and it needs to feel more like software, like a new-

Apoorv Agrawal 12:29 Right

Nick Turley 12:30 ... or, um, you know, an operating system of software.

Apoorv Agrawal 12:32 Right.

Nick Turley 12:33 Um, so that's one thing. Um, another thing, um, that gets at the same constraint is, is beginning to be proactive.

Apoorv Agrawal 12:40 Mm.

Nick Turley 12:41 Um, in a world where a lot of folks are, you know, too busy to delegate their problems to AI or don't w- quite know where to start, I think being able to help you proactively is really, really important as well.

Apoorv Agrawal 12:52 Mm.

Nick Turley 12:53 Um, but you know, it, it... I think all of these are, are product evolutions that we could make on, on top of the current tech. And the thing that gets me particularly excited is productizing our next-generation tech-

Apoorv Agrawal 13:04 Mm-hmm

Nick Turley 13:04 ... our reasoning models. Because the truth is, when you look at reasoning in ChatGPT today, it's relevant to a very small group of people. It's relevant for the people who are trying to get the most out of ChatGPT.

Apoorv Agrawal 13:15 Mm.

Nick Turley 13:15 But I fundamentally believe that reasoning, it's transformative, and if you can figure out how to productize reasoning in a way-

Apoorv Agrawal 13:22 Mm

Nick Turley 13:22 ... that works on people's behalf without them even knowing-

Apoorv Agrawal 13:25 Mm

Nick Turley 13:25 ... um, and that looks very much like, you know, the model doing long-horizon tasks on your behalf.

Apoorv Agrawal 13:30 Yeah.

Nick Turley 13:31 It doesn't mean you encounter the concept. It just means-

Apoorv Agrawal 13:33 Yeah

Nick Turley 13:33 ... it's benefiting you, right?

Apoorv Agrawal 13:34 Yeah.

Nick Turley 13:34 Um, so there's so much work to do, and, and-

Apoorv Agrawal 13:37 Yeah

Nick Turley 13:37 ... you know, the, the, the product certainly has to evolve to, to be relevant for, for, for this kind of scale.

Apoorv Agrawal 13:43 Yeah. One of the, one of the things that I've been hoping for a while, and, you know, Brad made a bet two years ago, um, when can ChatGPT help me take actions?

Nick Turley 13:55 Mm-hmm.

Apoorv Agrawal 13:55 When can ChatGPT help me be more proactive? And, um, I think his bet expired end of last year, so he's, he's, uh, we're very curious. W- when is that coming out?

Nick Turley 14:04 [laughs]

Apoorv Agrawal 14:04 I'll frame that for you because, you know, with, with, with, with search engines and Google 20-- you know, two decades ago-

Nick Turley 14:09 Yeah

Apoorv Agrawal 14:09 ... you would've gotten the 10 blue links.

Nick Turley 14:10 Yeah, yeah.

Apoorv Agrawal 14:10 You would've spent an hour getting the answer.

Nick Turley 14:13 Mm-hmm.

Apoorv Agrawal 14:13 You can now get the answer instantly-

Nick Turley 14:14 Mm-hmm, mm-hmm

Apoorv Agrawal 14:14 ... with ChatGPT.

Nick Turley 14:15 Mm-hmm.

Apoorv Agrawal 14:16 Uh, and it feels like the next step is, is actions.

Nick Turley 14:19 100%.

Apoorv Agrawal 14:20 And it feels like the next step is, you know, with, you know, Pulse is a, is a great proactive product.

Nick Turley 14:24 Mm-hmm, mm-hmm.

Apoorv Agrawal 14:25 Um, that, that, that feels... You know, it's-- I, I have a Pulse that runs weekly.

Nick Turley 14:28 Mm-hmm.

Apoorv Agrawal 14:29 But what I would really like is like, "Hey, Nick spoke about something," and just fi- just find me. Make sure I-

Nick Turley 14:33 Mm

Apoorv Agrawal 14:33 ... know that Nick spoke about this.

Nick Turley 14:35 Mm-hmm, mm-hmm.

Apoorv Agrawal 14:35 Or, or like, "Hey, this XYZ thing happened that I cared about a lot."

Nick Turley 14:38 Mm-hmm.

Apoorv Agrawal 14:39 Um, when, when is, when is one of the... that gonna get proactive? Um, what is the modality gonna look like?

Nick Turley 14:44 Yeah, yeah, yeah. So there's, there's two concepts, I think. Um, there, there's ChatGPT doing stuff rather than just answering.

Apoorv Agrawal 14:53 Mm-hmm.

Nick Turley 14:53 And then there's ChatGPT being proactive. And I think when you put them together-

Apoorv Agrawal 14:57 Mm

Nick Turley 14:57 ... you start feeling like-- it feels like a super assistant, um-

Apoorv Agrawal 14:59 Mm

Nick Turley 14:59 ... because I think these things compound.

Apoorv Agrawal 15:01 Mm.

Nick Turley 15:01 Um, on the action-taking piece, you know, strictly speaking, ChatGPT can do stuff today. Um, the action space is just very limited, right? It can search the web, which means it can use, you know, a search tool or a browser in the same way that a human would.

Apoorv Agrawal 15:15 Mm.

Nick Turley 15:15 Um, it can make images. It can do all the, all, all these things, right? But it, it doesn't have-- it clearly doesn't have the same action space that a human with a computer would have, and that is-

Apoorv Agrawal 15:22 Yeah

Nick Turley 15:22 ... what we aim to build. Um, and if you-- Timing is everything on these bets-

Apoorv Agrawal 15:26 Mm

Nick Turley 15:26 ... right? And I don't pretend to be great at timing either. Uh, you know, when I look at past attempts that we've made, like the ChatGPT agent, for example-

Apoorv Agrawal 15:32 Mm-hmm

Nick Turley 15:33 ... which kind of has capabilities like this, um, it was just slightly too early. The models weren't quite good enough to hit real escape velocity, and the problem is if you don't have escape velocity is that users don't learn to trust it. They don't even try.

Apoorv Agrawal 15:47 Mm.

Nick Turley 15:47 So when you look at a lot of things people were doing in the original version of ChatGPT agent, it was the things that happened to work.

Apoorv Agrawal 15:53 Mm.

Nick Turley 15:53 Like migrating your, uh, uh, a file server, uh, into the cloud or something like that. Useful stuff, but very niche.

Apoorv Agrawal 16:00 Yeah.

Nick Turley 16:01 Um, and, um, as this stuff gets better, um, we just have to get it to a point where people try to use it for real meaningful problems in their life because then we can start hill climbing, and this has been the magic of ChatGPT, where-

Apoorv Agrawal 16:15 Mm

Nick Turley 16:15 ... ChatGPT upon launch was good enough to get real attempts at use cases-

Apoorv Agrawal 16:19 Mm

Nick Turley 16:19 ... even if they didn't initially work. Like, ChatGPT was a pretty bad writer originally. It was a bad software engineer, but people tried and got enough value out of it that we could take those use cases and make them great.

Apoorv Agrawal 16:28 Mm.

Nick Turley 16:28 And I do think we're about to get to that point with general-purpose agents, where it works well enough that you get at least partial credit.

Apoorv Agrawal 16:36 Mm.

Nick Turley 16:36 And because you're getting partial credit, you get really good tasks back, and then the magic begins because, um, you know, once you have-

Apoorv Agrawal 16:43 Mm

Nick Turley 16:43 ... a, a set of use cases that you can, um, climb the hill on, we can make them awesome. So on tasks, I think we're close, but, uh, uh, I think even people inside of OpenAI would have had a hard time predicting exactly when this gets good. Um-

Apoorv Agrawal 16:56 Yeah

Nick Turley 16:56 ... we've been excited about it for a while.

Apoorv Agrawal 16:58 Yeah.

Nick Turley 16:58 On, um, proactivity, um, Pulse was a really great first step because-

Apoorv Agrawal 17:06 Yeah

Nick Turley 17:06 ... what we wanted to build was a form factor where you're not prompting the model, like the model's prompting you.

Apoorv Agrawal 17:11 Mm-hmm.

Nick Turley 17:12 Um, for the reasons that I described earlier, which is, you know, it's so hard for people to delegate and to figure out what their problems are. What if the AI understood your, your goals and the things you're interested in-

Apoorv Agrawal 17:21 Mm

Nick Turley 17:21 ... and just could start being proactive on your behalf?

Apoorv Agrawal 17:24 Mm-hmm.

Nick Turley 17:25 Um, Pulse is limited in the, the value it can provide for you because it's not connected to your life-

Apoorv Agrawal 17:30 Mm

Nick Turley 17:30 ... and it can't take action. So it's producing information for you-

Apoorv Agrawal 17:34 Mm

Nick Turley 17:34 ... and people love that. You know, I love that. I've got mine running too.

Apoorv Agrawal 17:37 Mm.

Nick Turley 17:38 But I think the magic begins when you have actions and proactivity-

Apoorv Agrawal 17:41 Mm

Nick Turley 17:41 ... because then it can begin speculatively actually, you know-

Apoorv Agrawal 17:44 Mm

Nick Turley 17:44 ... um, detecting, "Hey, you just landed where you were supposed to go."

Apoorv Agrawal 17:47 Mm.

Nick Turley 17:48 Um, you know, "I'm, I'm, I'm gonna call a cab for you." Or-

Apoorv Agrawal 17:51 Mm

Nick Turley 17:51 ... um, you know, if you're at work, it's like, "Hey, I proactively ran this analysis because I saw your metrics dropped."

Apoorv Agrawal 17:57 Mm.

Nick Turley 17:58 Um, so I think these things really compound, and we need to b- nail multiple of the building blocks to really achieve the transformation-

Apoorv Agrawal 18:04 Yeah

Nick Turley 18:05 ... uh, and the form factor that we hope for.

Apoorv Agrawal 18:07 A- as you were answering those questions, I now have, uh, 15 more questions for you.

Nick Turley 18:10 [laughs]

Apoorv Agrawal 18:11 So, so, so-

Nick Turley 18:11 I hope you have 15 more minutes

Apoorv Agrawal 18:13 ... I hope you have 15 more minutes. But, um, okay,

Apoorv Agrawal 18:15 one by one, we'll start with what you said on actions and tasks.

Nick Turley 18:18 Mm-hmm, mm-hmm.

Apoorv Agrawal 18:19 Got it on timing.

Nick Turley 18:19 Mm-hmm.

Apoorv Agrawal 18:20 Tough to say. But is there a shape or ordinality of tasks and, and, or, or agents that, that you think, "Hey, this is the kind of thing that's likely to come first," whenever it does?

Nick Turley 18:29 I mean, the thing that's already come first is the, uh, domain-specific agents, right? Uh, if you look at what's happening in, in code-

Apoorv Agrawal 18:35 Hmm

Nick Turley 18:35 ... we're, we're, we're fully there.

Apoorv Agrawal 18:37 Hmm.

Nick Turley 18:38 You know, it's, it's mind-bending, but we've got so many engineers, um, who, who don't open their IDE, like, ever. And-

Apoorv Agrawal 18:45 Right

Nick Turley 18:45 ... for me, as someone who, you know, used to code and then unfortunately got very, very busy, it's brought me back in the game.

Apoorv Agrawal 18:50 Hmm.

Nick Turley 18:51 So, um, Codex and, you know, products like it is clearly a product that has escape velocity, where people are absolutely using it-

Apoorv Agrawal 19:00 Yeah

Nick Turley 19:00 ... for all kinds of agentic work, and if you just take what people are doing-

Apoorv Agrawal 19:03 Hmm

Nick Turley 19:03 ... and make it work even better-

Apoorv Agrawal 19:05 Hmm

Nick Turley 19:05 ... you kind of get all the way there.

Apoorv Agrawal 19:07 Hmm.

Nick Turley 19:08 Um, you know, I won't be surprised if you see this happen for other forms of sort of quantitative knowledge work-

Apoorv Agrawal 19:15 Hmm

Nick Turley 19:15 ... just because it hap- happens to have the properties that code has. W- it's, it's testable.

Apoorv Agrawal 19:19 Hmm.

Nick Turley 19:19 You know if it worked or not.

Apoorv Agrawal 19:21 Hmm.

Nick Turley 19:21 Um, it's, you know, very RL friendly.

Apoorv Agrawal 19:23 Hmm.

Nick Turley 19:23 Um-

Apoorv Agrawal 19:24 Yeah

Nick Turley 19:24 ... but, um, uh, the, the s- the domain-specific ones already work. I think the thing that everyone's working for is, you know, general purpose agents that just-

Apoorv Agrawal 19:33 Yeah

Nick Turley 19:33 ... kind of work for anything.

Apoorv Agrawal 19:35 Yeah.

Nick Turley 19:35 And I, you know, that's what I think y- you need to, to win a consumer because it's very hard to train people into, like-

Apoorv Agrawal 19:41 Yeah

Nick Turley 19:41 ... okay. Um, it can work. Like, Deep Research was a consumer product, and it's really was our first agentic-

Apoorv Agrawal 19:46 Yeah

Nick Turley 19:46 ... uh, thing out there. Uh, but I think what consumers want is I can just ask it anything, and it'll do what-

Apoorv Agrawal 19:53 Yeah

Nick Turley 19:53 ... um, what needs to be done-

Apoorv Agrawal 19:55 Yeah

Nick Turley 19:55 ... uh, without any sort of retraining. And, um-

Apoorv Agrawal 19:58 Yeah

Nick Turley 19:58 ... we'll get there, just a matter of time.

Apoorv Agrawal 20:00 At least a psychological goal is, uh, flight bookings-

Nick Turley 20:04 Totally

Apoorv Agrawal 20:04 ... hotel bookings, restaurant bookings, shopping.

Nick Turley 20:07 All this stuff. Um, there are so many consumer problems, and those are just the type of things that you would kick off, right?

Apoorv Agrawal 20:12 Yeah.

Nick Turley 20:13 The minute you have productivity, there's, there's things you don't even think of as agentic tasks.

Apoorv Agrawal 20:18 Hmm.

Nick Turley 20:18 Um, like you're trying to get in shape.

Apoorv Agrawal 20:20 Hmm.

Nick Turley 20:20 You don't think of that as a task you would delegate-

Apoorv Agrawal 20:22 Yeah

Nick Turley 20:22 ... um, unless you have a trainer, in which case you do. But most people don't.

Apoorv Agrawal 20:25 Yeah.

Nick Turley 20:26 Right? But if the AI knew that, it could totally start working in the background for you over-

Apoorv Agrawal 20:30 Yeah

Nick Turley 20:30 ... very long periods of time and, and getting you, you know, um, "Here's your fitness plan. Okay, I actually signed you up for this thing."

Apoorv Agrawal 20:36 Yeah.

Nick Turley 20:36 Uh, you could imagine it being quite helpful if it's aligned with your, with your long-term interests.

Apoorv Agrawal 20:39 You're gonna give Ozempic a run for the money.

Nick Turley 20:41 [laughs]

Apoorv Agrawal 20:43 [laughs]

Nick Turley 20:43 We gotta be careful what businesses we get into, but, uh, uh, hopefully we can help.

Apoorv Agrawal 20:47 That'll be great. Cannot wait. Cannot wait. Uh, the second thing you said was, um, you know, pr- proactive users, and that might require us to go beyond chatbots.

Nick Turley 20:56 Mm-hmm.

Apoorv Agrawal 20:57 What's an example of a modality that might take ChatGPT

Apoorv Agrawal 21:00 beyond a chatbot?

Nick Turley 21:03 So chat will always be close to my heart.

Apoorv Agrawal 21:05 Yeah.

Nick Turley 21:05 It's the way we grew up.

Apoorv Agrawal 21:06 Yeah.

Nick Turley 21:07 Um, and it's an important modality to stay.

Apoorv Agrawal 21:08 Yeah.

Nick Turley 21:08 Like, I, I, I think it's less about chat and more about natural language to me-

Apoorv Agrawal 21:13 Hmm

Nick Turley 21:13 ... where, you know, it's the fact that you can express yourself to the machine in ways that are very natural to you.

Apoorv Agrawal 21:18 Hmm.

Nick Turley 21:18 Whether or not that's text, whether or not that's voice-

Apoorv Agrawal 21:20 Hmm

Nick Turley 21:20 ... whether or not that is, you know, um, structured UI that is rendered by the model.

Apoorv Agrawal 21:24 Hmm.

Nick Turley 21:24 That is just very, very powerful, and that's here to stay.

Apoorv Agrawal 21:26 Yeah.

Nick Turley 21:26 But, um, I think the thing-

Apoorv Agrawal 21:29 SA Server.

Nick Turley 21:29 [laughs] That's right. Uh, uh, for, for those that don't know, that's the name of our code base, um-

Apoorv Agrawal 21:35 Yeah

Nick Turley 21:35 ... um, short for Super Assistant Server, um, because, you know, it's proof that this was always the vision and is always the vision.

Apoorv Agrawal 21:40 Mm-hmm.

Nick Turley 21:40 Um, but, uh, you know, the, the, the thing that, that'll change, I think, is that chat is a great way of expressing your intent. It's a good way of communicating with the machine, but it's not a great output, um, where in many cases what you want back is an artifact. Like, here's your, your, your, you know, plan for your trip.

Apoorv Agrawal 21:58 Hmm.

Nick Turley 21:58 Here is the analysis.

Apoorv Agrawal 22:00 Hmm.

Nick Turley 22:00 Here is, um, you know, an outcome that I delivered for you. You know, I just made you five bucks.

Apoorv Agrawal 22:06 Hmm. [laughs]

Nick Turley 22:06 Like, this is what I want my AI doing for me, right?

Apoorv Agrawal 22:08 Making some money.

Nick Turley 22:09 Yeah, totally. I mean, this is what people care about, right? And, and-

Apoorv Agrawal 22:11 Yeah

Nick Turley 22:11 ... and, and I think chat will always be there as the way that you sort of disambiguate your intent and you kick off the task.

Apoorv Agrawal 22:17 Yeah.

Nick Turley 22:17 But I don't think it's necessarily the, the, the final de- deliverable, and I think that's, that's, that's the way in which we can evolve. So hopefully that's a very graceful transition because w- I'm very lucky, and it's hard-earned-

Apoorv Agrawal 22:28 Yeah

Nick Turley 22:28 ... you know, to, to have a billion people coming to you weekly for a thing that they love.

Apoorv Agrawal 22:32 Yeah.

Nick Turley 22:32 But I think it's a great jumping-off point because we have so much unsatisfied intent, uh, from people where they're trying to-

Apoorv Agrawal 22:38 [laughs]

Nick Turley 22:38 ... clearly trying to do something, and ChatGPT is helpful enough, but it could be so much more helpful. Um, and, and I think that's where we evolve.

Apoorv Agrawal 22:45 Yeah. And you must be sitting on so much of this data where people are showing up to ChatGPT and attempting. As you said, uh, three years ago, they were at least making the attempt.

Nick Turley 22:54 Yeah.

Apoorv Agrawal 22:55 So, so you might have at least a frequency histogram of like, "Here are, here are all the things that people wanna achieve with us."

Nick Turley 23:00 We do. Uh, we have, like, really awesome, you know, um, classifiers that run automatically.

Apoorv Agrawal 23:06 Hmm.

Nick Turley 23:06 It's fully privacy-preserving but gives us a sense of, you know, what use cases people have.

Apoorv Agrawal 23:10 Yeah.

Nick Turley 23:10 And it's important, right? Because when you make a new model, when you make a model update, you wanna know what use cases just got better and what use cases got worse.

Apoorv Agrawal 23:17 Yeah.

Nick Turley 23:18 Um, and that's not always tr- always trivial to figure out unless you have really good analytics on the system. But so much of my learning is actually qualitative, where I will just, you know, uh, I have a habit of reaching out to a fairly random set of users to just figure out what they're doing.

Apoorv Agrawal 23:32 Hmm.

Nick Turley 23:32 And I've never worked on a product where three and a half years later you're still learning every time.

Apoorv Agrawal 23:36 Hmm.

Nick Turley 23:36 Because usually by that time you know what the use cases are that your product can, you know, deliver on. But our tech is so unusual in the fact that I keep-

Apoorv Agrawal 23:44 Yeah

Nick Turley 23:44 ... learning about something crazy I, I didn't know was possible.

Apoorv Agrawal 23:47 Wow. That's awesome.

Nick Turley 23:49 Yeah.

Apoorv Agrawal 23:49 Well, basically a billion users, I suspect a small fraction of them are power users-

Nick Turley 23:54 Mm-hmm

Apoorv Agrawal 23:54 ... who are, uh, getting maybe thousands, maybe, maybe, maybe tens of thousands of, of, of value-

Nick Turley 23:59 Yeah

Apoorv Agrawal 23:59 ... on,

Apoorv Agrawal 24:00 on their two hundred dollar subscription.

Nick Turley 24:01 Yeah.

Apoorv Agrawal 24:02 Um, the vast majority is, is, you know, middle of the pack, and then, and, and a few, call it casual users who are, you know, start using ChatGPT as search maybe.

Nick Turley 24:12 Yeah.

Apoorv Agrawal 24:12 Or, or, or-

Nick Turley 24:13 Yeah

Apoorv Agrawal 24:13 ... or, or teach me about AI or, or help me with my homework. Um, what is your focus like maybe in those constituents, power users, casual users, and, and early users, or however you frame it, what is our focus on for each of those three factions?

Nick Turley 24:26 Yeah, yeah. Well, first of all, I feel accountable to our entire user base.

Apoorv Agrawal 24:31 Yeah.

Nick Turley 24:31 And in fact, our non-users too, because, you know, products like, like ChatGPT can have real externalities on, on-

Apoorv Agrawal 24:37 Yeah

Nick Turley 24:37 ... on all humans.

Apoorv Agrawal 24:38 Yeah.

Nick Turley 24:38 But when I think about sort of the way we build, it's really useful to imagine the extremes. Um, one extreme being a user who doesn't care about AI at all-

Apoorv Agrawal 24:49 Mm-hmm, mm

Nick Turley 24:50 ... um, who has a b- busy life, um, and, um, uh, needs to be convinced of, you know, the value that we can provide.

Apoorv Agrawal 24:57 Hmm.

Nick Turley 24:57 'Cause that forces you to really nail the interface and to-

Apoorv Agrawal 25:00 Mm

Nick Turley 25:00 ... expose the capabilities, uh, that are hidden in the model in a way that people can actually grok. And then the other useful extreme i- is, you know, um, our, our power user base.

Apoorv Agrawal 25:09 Mm.

Nick Turley 25:09 Because power users are the users who teach us what's possible. Um, um, it's actually impossible for us to do all the product discovery, um, on our own simply by, because of how empirical this technology is-

Apoorv Agrawal 25:24 Mm

Nick Turley 25:24 ... um, and how much you actually learn post-launch.

Apoorv Agrawal 25:27 Mm.

Nick Turley 25:28 So building, you know, um, for each of those extremes can be valuable.

Apoorv Agrawal 25:33 Mm.

Nick Turley 25:34 Uh, but our user base is incredibly diverse, and, um, people have so many different use cases. And th- this is why, you know, I like to look, look at all kinds of different segmentations, not just frequency, but also, you know, what use cases are you coming to us for? Um, but, um, definitely huge variety in the ChatGPT user base.

Apoorv Agrawal 25:52 Yeah.

Nick Turley 25:53 I look up to macOS, for example, as an example-

Apoorv Agrawal 25:54 Mm

Nick Turley 25:54 ... where it really w- works for people who don't understand technology at all. It's entirely magical.

Apoorv Agrawal 25:59 Mm.

Nick Turley 25:59 But if you are a power user, you've got Terminal, you've got settings, you can configure almost anything in macOS.

Apoorv Agrawal 26:03 Mm.

Nick Turley 26:04 And it's really beautifully done where-

Apoorv Agrawal 26:06 Mm

Nick Turley 26:06 ... the complexity is progressively disclosed.

Apoorv Agrawal 26:08 Mm.

Nick Turley 26:08 So, um, you can interact with it and love, love the simplicity of it all, but you can also... You got all the knobs, and, and developers love it, right?

Apoorv Agrawal 26:15 Love it.

Nick Turley 26:15 And so I think this is kind of the inspiration for how we want to be in ChatGPT. That doesn't mean we always live up to it, but-

Apoorv Agrawal 26:20 Mm

Nick Turley 26:20 ... um, it means that building for power users is extremely important.

Apoorv Agrawal 26:23 Mm.

Nick Turley 26:24 And, you know, that's not just a property that I'm think is sort of aesthetically exciting.

Apoorv Agrawal 26:29 Mm.

Nick Turley 26:29 It's also really important in AI because it's the power users who show you what's possible.

Apoorv Agrawal 26:33 Mm.

Nick Turley 26:35 They are actually doing the product discovery-

Apoorv Agrawal 26:37 Mm

Nick Turley 26:37 ... 'cause it would be impossible for us with such an empirical tech to do all the pr- product discovery on our own.

Apoorv Agrawal 26:42 Mm.

Nick Turley 26:42 So the type of user who subscribes to ChatGPT Pro, who used Codex before it quite worked-

Apoorv Agrawal 26:49 Mm

Nick Turley 26:50 ... who is now the strongest advocate of, of, of tools like, tools like tokens and teaching us what's, what's possible-

Apoorv Agrawal 26:55 Mm

Nick Turley 26:56 ... um, that is an incredible valuable-

Apoorv Agrawal 26:58 Yeah

Nick Turley 26:58 ... incredibly valuable member of the community.

Apoorv Agrawal 27:00 Mm.

Nick Turley 27:00 And it might not show up in your, you know, um, weekly active users. It's just one number, right?

Apoorv Agrawal 27:06 Yeah.

Nick Turley 27:06 Um, but this is exactly why there isn't, like, a single North Star.

Apoorv Agrawal 27:10 Mm.

Nick Turley 27:10 And you really need to, n- need, need to take these different segments very seriously. So-

Apoorv Agrawal 27:14 Yeah

Nick Turley 27:14 ... um, I love building for power users.

Apoorv Agrawal 27:16 Yeah.

Nick Turley 27:17 Um, and, uh, you, you, you asked on, you know, token consumption, et cetera.

Apoorv Agrawal 27:22 Oh, yeah.

Nick Turley 27:23 It's so fascinating to see. There's people who get incredible value, um, out of these products, and, uh, watching what they do is very informative.

Apoorv Agrawal 27:32 Okay. So, so you're very focused on the entire user base.

Nick Turley 27:35 Mm-hmm.

Apoorv Agrawal 27:36 Um, learn a lot from the power users.

Nick Turley 27:38 Mm-hmm.

Apoorv Agrawal 27:38 Um, you know, the other thing I might say is the power users right now are getting, um, a lot of value-

Nick Turley 27:45 Mm-hmm

Apoorv Agrawal 27:45 ... almost too much value, and a lot of-

Nick Turley 27:47 No such thing.

Apoorv Agrawal 27:48 No such thing. [laughs] The, the, the analog that is most common is-

Nick Turley 27:52 Mm

Apoorv Agrawal 27:52 ... the Uber and Lyft of the 2015 era.

Nick Turley 27:55 Mm-hmm.

Apoorv Agrawal 27:55 Right? And, you know, it took a, it took a while, but I know you were thinking about it a lot. I know you guys are

Apoorv Agrawal 28:00 thinking about pricing quite a bit.

Nick Turley 28:01 Yeah.

Apoorv Agrawal 28:02 Um, maybe tell us a little bit about pricing. Um, you know, right now pricing's pretty simple. Uh-

Nick Turley 28:07 Mm-hmm

Apoorv Agrawal 28:07 ... is there a, is there a path for, for, for folks who are getting a lot of great value to price that product differently and, and, and, you know, meet them where they are and, and, and, and, and the other way on the other side?

Nick Turley 28:17 And pricing's... There's no world in which pricing doesn't s- significantly evolve when-

Apoorv Agrawal 28:22 [laughs]

Nick Turley 28:22 ... the technology is changing this quickly.

Apoorv Agrawal 28:25 Yeah.

Nick Turley 28:25 Right? ChatGPT originally was entirely free, and the reason for that was that it was intended to be a demo.

Apoorv Agrawal 28:32 Yeah.

Nick Turley 28:32 And we were gonna wind it down after a month.

Apoorv Agrawal 28:34 Mm-hmm.

Nick Turley 28:34 We then realized that the demo went viral, and people loved the demo, and it was actually a product, and... But we realized it'd be a product, you can't take the product down every time you're at capacity. So we, you know, shipped subscriptions simply 'cause it could shape the demand.

Apoorv Agrawal 28:47 Mm.

Nick Turley 28:48 It was a way of gracefully turning users away-

Apoorv Agrawal 28:50 Mm

Nick Turley 28:50 ... when we had to turn away someone, and it felt like the fairest and most equitable way of doing so-

Apoorv Agrawal 28:54 Mm

Nick Turley 28:55 ... is saying, "Hey, you know, if you really need this product, pay a subscription fee, and you got it."

Apoorv Agrawal 28:59 Mm.

Nick Turley 28:59 Then we figured out how to make the product stable, and we had the choice of do we keep the subscription thing, or do we go back to free? And we realized we had consistently more tech that we couldn't scale, GPT-4 being the first example-

Apoorv Agrawal 29:09 Mm

Nick Turley 29:09 ... because we had way too many free users to serve GPT-4, and we put it behind the Plus plan.

Apoorv Agrawal 29:14 Wow.

Nick Turley 29:14 And so, you know, the way we stumbled into subscriptions was, was sort of accidental by trying to just solve for the user. Um, and it felt like the right way at the time, um, to, to provide maximal access to our, to, to, to our tech. Um, since then, we've had so many other breakthroughs, including test-time compute, where you can scale up, um, um, intelligence, um, um, kind of, um, as much as you want-

Apoorv Agrawal 29:43 Yeah

Nick Turley 29:43 ... more or less. And, um, you know, it took us, you know, and the entire industry a little bit of time to turn that into product value, but we're here now. We're, you know... Our, our, our, our, our, our power users wanna use more and more and more intelligence. And, you know, it, it's possible that, you know, in, in the current era, having unlimited plan is like having unlimited electricity plan. You know, it just doesn't make sense because, like, you know, people may need a lot, a lot of electricity, and they're getting a lot of value out of that. Uh, there's a reason you can't buy that, right?

Apoorv Agrawal 30:11 Mm.

Nick Turley 30:11 So, um, obviously I wanna be really thoughtful about the way that we evolve our, our plans and SKUs and subscriptions, but yeah, it'd be incredibly subs- you know, surprised if it didn't change given the magnitude and profoundness of, of the technical breakthroughs that we've had and the product breakthroughs that follow.

Apoorv Agrawal 30:28 Yeah. Um, and, you know, relatedly, so, so I, I imagine you're gonna have something for the power users.

Nick Turley 30:36 Mm-hmm.

Apoorv Agrawal 30:37 Um, what about the other side? How do we get, um, the casual users in through the-

Nick Turley 30:41 Yeah

Apoorv Agrawal 30:42 ... in through the wheel and, and still, uh, monetize them?

Nick Turley 30:45 As mentioned, you know, our, our business model is, will evolve and the North Star is access, right?

Apoorv Agrawal 30:50 Yeah.

Nick Turley 30:51 We, we, we would like to pick a, um, way of, of, of, of, of pr- you know, providing an off- we wanna pro- provide an offering that maximizes, um, the number of people can, who can access our most powerful tools. And I think for the longest time that has been subscriptions.

Apoorv Agrawal 31:07 Mm.

Nick Turley 31:08 Um, subscriptions have the downside of the fact that, you know, in many markets people don't-

Apoorv Agrawal 31:12 Mm

Nick Turley 31:12 ... have credit cards or they don't use credit cards to subscribe to software.

Apoorv Agrawal 31:15 Mm.

Nick Turley 31:15 Um, and we're interested, uh, in other ways that can maximize access of, of, of the tech. Um, our ads pilots are in that spirit. You know, we, we really view it of a, as a tool of bringing ChatGPT and our intelligence-

Apoorv Agrawal 31:27 Mm

Nick Turley 31:27 ... most broadly-

Apoorv Agrawal 31:28 Mm

Nick Turley 31:28 ... to anyone around the world.

Apoorv Agrawal 31:29 Mm.

Nick Turley 31:30 Um, and it is an example of how we constantly need to evolve and figure out the best way to, to, you know, bring the demand in line with what we are able to offer.

Apoorv Agrawal 31:40 Makes sense. Makes sense. You know, the ads, the ads piece is, um, is, is, has, has been a tricky one 'cause, you know, Sam has historically expressed reluctance about ads and, um, you know, you've gotta maintain a lot of trust-

Nick Turley 31:54 Mm-hmm

Apoorv Agrawal 31:54 ... um, with, wh- while delivering that. So, um, w- I guess what changed?

Nick Turley 32:00 I think i- i- we've talked about this several times in my, my history at OpenAI, and every time it, we, it came up we said if, if we were to do ads we'd have to be really thoughtful about-

Apoorv Agrawal 32:11 Yeah

Nick Turley 32:11 ... the way we do it. Um, so the first thing we did, you know, starting, you know, end of last year was to really engage the company on if we put ads in ChatGPT, how should we approach it? What should the principles weigh? What be-- how do you preserve the things that are magical about ChatGPT while getting the benefits of ads, which, you know, is, is, is our ability to bring-

Apoorv Agrawal 32:33 Mm

Nick Turley 32:33 ... um, our most advanced tech to, to, to anyone, um, regardless of their ability to pay.

Apoorv Agrawal 32:38 Mm.

Nick Turley 32:38 And, um, I really love where we ended up on the principles side.

Apoorv Agrawal 32:42 Mm-hmm.

Nick Turley 32:42 On the experience side-

Apoorv Agrawal 32:44 Mm

Nick Turley 32:44 ... um, we're very, very early. But on the principles side I feel really proud because, you know, it's, it's very important that the answer of ChatGPT be independent-

Apoorv Agrawal 32:51 Mm

Nick Turley 32:51 ... as an example. Respecting user privacy is very important, and there's a lot to learn, um, from, you know, the way that tech has evolved over the last few years.

Apoorv Agrawal 32:58 Mm.

Nick Turley 32:59 Um, or really last decade. Um-

Apoorv Agrawal 33:01 Mm.

Nick Turley 33:02 Um, and, uh, I, I like that the principles are out there before we've even really gotten started. Um, like, we're very early with our pilots.

Apoorv Agrawal 33:09 Mm.

Nick Turley 33:10 Um, you know, it, it's kind of interesting. I was obviously very anxiously and eagerly looking at, um, our support, um, inbounds and data, and, um, the most common inquiry about ads is not, you know, how do I disable ads or turn off ads? But it's like how do I run an ad?

Apoorv Agrawal 33:27 Yeah. [laughs]

Nick Turley 33:27 Um, 'cause the entire ecosystem is really excited to be part-

Apoorv Agrawal 33:31 Yeah

Nick Turley 33:31 ... of the story and to figure out a way to talk to ChatGPT users. So, um, there's a lot more to come, um, but I, I, I'm, I'm very eager to get this right.

Apoorv Agrawal 33:41 Yeah. Yeah, I'm sure you guys will. Um,

Apoorv Agrawal 33:46 switching gears, Nick, um, something you and I have spoken about a little bit is, you know, distribution and partnerships. Uh-

Nick Turley 33:52 Mm-hmm

Apoorv Agrawal 33:53 ... there was a couple of big partnerships last year. Um, Apple reliance with, with Gemini.

Nick Turley 33:58 Mm.

Apoorv Agrawal 33:58 Those are two big user bases, right? A lot of India, a lot of the iOS users. Um, ta- talk, tell us a little bit how you think about partnerships for ChatGPT to, to, to meet the, meet the user base, um, and, and maybe specifically on those two as well.

Nick Turley 34:14 Look, um, I think partnerships are a great way to, to bring-

Apoorv Agrawal 34:18 Mm

Nick Turley 34:18 ... two, two, two, two, two products together and to-

Apoorv Agrawal 34:21 Yeah

Nick Turley 34:21 ... you know, um, expose, um, something like ChatGPT to, um, people who might not otherwise, otherwise have encountered it. Um, the thing that I care about most when considering something, um, like a partnership is what is the, what is the user experience-

Apoorv Agrawal 34:39 Mm

Nick Turley 34:39 ... and can we make it amazing? Um-

Apoorv Agrawal 34:41 Yeah

Nick Turley 34:41 ... because at the end of the day, um, when you look at, um, what's going on in the market, um, you, you can get users to click on things. You can get them to tap any sort of product, um, especially if it looks like a product they recognize, et cetera.

Apoorv Agrawal 34:56 Mm.

Nick Turley 34:56 Um, but if the experience isn't truly awesome, um, you know, people will churn or they will, you know, at least not retain in the way that, you know, um, we've been lucky to retain them on ChatGPT.

Apoorv Agrawal 35:06 Mm.

Nick Turley 35:06 So for that reason, you know, I'm super interested in, in paths like that, um, but it needs to be great.

Apoorv Agrawal 35:11 Yeah.

Nick Turley 35:12 Uh, it needs to be creative to the user. Um, we are very lucky to have, um, a great brand and a recognizable product-

Apoorv Agrawal 35:18 Yeah

Nick Turley 35:18 ... um, for many, many folks, and, uh, I wanna make sure that anything we do is, is accretive to, um, um, to all that.

Apoorv Agrawal 35:25 Nick, you are a, you're a master of trade-offs. You must be making a lot of trade-offs right now. Uh-

Nick Turley 35:30 Yeah

Apoorv Agrawal 35:31 ... um, tell us about, tell us about some of the trade-offs you're making. Tell us about, you know, an, uh, a trade-off that you might be making that people don't appreciate, uh, from the outside.

Nick Turley 35:41 There are a lot of trade-offs, um, in- indeed, and, um, for different reasons, right? Um, um, the-- one I encounter a lot is trading off delivering on the people, uh, on, on the use cases that exist in the product today and making them better versus productizing, you know, step change technology that's gonna generate a whole 'nother set of use cases.

Apoorv Agrawal 36:06 Mm.

Nick Turley 36:07 'Cause when you think about how ChatGPT came to be, it was a totally open-ended product.

Apoorv Agrawal 36:11 Yeah.

Nick Turley 36:11 It was basically a user experience around a technical breakthrough.

Apoorv Agrawal 36:15 Mm.

Nick Turley 36:16 And we couldn't have told you all the ways that people find it valuable-

Apoorv Agrawal 36:20 Yeah

Nick Turley 36:20 ... but putting it out there was really important because it allowed us, us to discover and the world to discover what you, you can do.

Apoorv Agrawal 36:26 Mm-hmm.

Nick Turley 36:26 And then post-ChatGPT, um, we can obviously very systematically go and improve on the things that people actually wanna use it for. And when you're at a company in this moment where you both have such amazing traction with what exists today and the most mind-bending breakthroughs on the research side- The balance you have to strike is making the core product you have better today with all the things that matter, latency, um, reliability-

Apoorv Agrawal 36:55 Mm.

Nick Turley 36:55 ... um, making the use cases really great that people come to-

Apoorv Agrawal 36:58 Mm

Nick Turley 36:58 ... with, um, you know, providing access to, to the step change. Um-

Apoorv Agrawal 37:03 Mm.

Nick Turley 37:04 And, um, uh, you know, we try to get the balance right, but we're a small team, and we don't always get it right. And for that reason, that's one of the most difficult, um, trade-offs that I have to deal with.

Apoorv Agrawal 37:14 Nick, I imagine one

Apoorv Agrawal 37:15 of the hardest trade-offs you guys make here is those, uh, GPUs that are melting between ChatGPT, between Codex, Research. Uh, how do you guys, uh, allocate the GPUs? [chuckles]

Nick Turley 37:27 That is a very good question. Um, and I'll let you know when I figure it out.

Apoorv Agrawal 37:31 [chuckles]

Nick Turley 37:31 Um, just kidding. You know, we, we've gotten, we've gotten a lot better at this. Um, I really hope, by the way, to be at a point one day, and I've yet to reach that point, where-

Apoorv Agrawal 37:39 Mm-hmm

Nick Turley 37:39 ... we don't have to face this trade-off-

Apoorv Agrawal 37:41 Yeah

Nick Turley 37:41 ... because it's really painful to have real user demand-

Apoorv Agrawal 37:45 Mm

Nick Turley 37:45 ... uh, for products that you can't serve.

Apoorv Agrawal 37:47 Mm.

Nick Turley 37:47 Um, again, if, if you only ever worked in software, that's an entirely unusual dynamic, right, where you just-

Apoorv Agrawal 37:52 Yeah

Nick Turley 37:52 ... you know, where you are limited by this zero-sum resource-

Apoorv Agrawal 37:55 Yeah

Nick Turley 37:55 ... out there. Um, the marketplaces have it. Um-

Apoorv Agrawal 37:59 Yeah

Nick Turley 37:59 ... um, but you know, I think pure software doesn't, doesn't really have the dynamic, right?

Apoorv Agrawal 38:03 Yeah.

Nick Turley 38:03 So, um, yeah, one thing we try to do, obviously, we prioritize our existing users, um, first.

Apoorv Agrawal 38:11 Right.

Nick Turley 38:11 We wanna provide a fast, reliable product.

Apoorv Agrawal 38:13 Mm-hmm.

Nick Turley 38:14 Um, and that is critical and table stakes.

Apoorv Agrawal 38:17 Mm.

Nick Turley 38:18 Then when you look at new capabilities, the sort of na- naive, um, business school thing to do would be to probably, you know, look at revenue, incremental revenue per GPU-

Apoorv Agrawal 38:27 Yeah

Nick Turley 38:27 ... or something like that.

Apoorv Agrawal 38:28 Yeah.

Nick Turley 38:28 But this is where it's more an art than a science because we often have new breakthrough capabilities that are entirely zero to one. Deep Research was one of those. You know, we, we-

Apoorv Agrawal 38:35 Mm

Nick Turley 38:35 ... couldn't have told you is there gonna be demand for a-

Apoorv Agrawal 38:37 Mm

Nick Turley 38:38 ... you know, consumer demand for a research product. But if you don't productize it, uh, um, to find out-

Apoorv Agrawal 38:43 Mm-hmm. Mm-hmm

Nick Turley 38:44 ... um, you, you will never know. So, you know, this is where we have to be a little bit thoughtful on how we balance, you know, um, things that are no-brainers that people are really gonna love with things-

Apoorv Agrawal 38:52 Mm

Nick Turley 38:52 ... that are, um, brand new ideas. [swallows] Then obviously on the research side, there's a reason that Mark has the job he has because-

Apoorv Agrawal 38:58 Mm-hmm

Nick Turley 38:58 ... a big part of, um, his, his job is, is figuring out what research to fund and, um, um, you know, uh, obviously GPU is a big part of that. So, um, very nuanced topic-

Apoorv Agrawal 39:09 Yeah

Nick Turley 39:09 ... and that we're continuously getting better at. But, um, for me, the priority is always on our users.

Apoorv Agrawal 39:13 Yeah. The other takeaway that I had is you can... You don't have line of sight to a time when you won't have that problem.

Nick Turley 39:22 It's, it's been so fascinating, um, because, you know, uh, we obviously have been incredibly lucky to, uh, encounter more and more users who wanna use our technology, but then the value that we're able to provide for each user is going up-

Apoorv Agrawal 39:37 Yeah

Nick Turley 39:37 ... as well. And, um, you know, GP- or GPU consumption, you know, correlates pretty well with that value.

Apoorv Agrawal 39:43 [chuckles]

Nick Turley 39:44 Um, and, uh, when you just look at token consumption per user, um, especially in the enterprise too, which is-

Apoorv Agrawal 39:49 Mm

Nick Turley 39:50 ... you know, um, a, a massive opportunity-

Apoorv Agrawal 39:52 Yeah

Nick Turley 39:52 ... um, you see, um, a lot of very GP- GPU hungry, hungry workflows.

Apoorv Agrawal 39:57 Mm.

Nick Turley 39:57 And, um, um, yes, demand keeps going up even as prices go down.

Apoorv Agrawal 40:01 This is a fascinating insight. People used to think that humans were, you know, you can't kinda make more humans. Well, on a, takes nine months and then-

Nick Turley 40:09 [laughs]

Apoorv Agrawal 40:09 ... 19 years. But, uh, but you're saying, well, that's actually a more, more in- more, less finite resource than GPUs.

Nick Turley 40:17 Yeah, I mean, on, on the human side, um, you can hire, uh, more humans, and obviously we've been doing, busy doing that and bringing-

Apoorv Agrawal 40:24 Yeah

Nick Turley 40:24 ... the best talent, um, across functions to OpenAI. Um, in the world with agents, you can also get more leverage per human.

Apoorv Agrawal 40:31 Mm.

Nick Turley 40:31 Um, um, you can make your humans very effective, um, at their job to do more. But GPUs are zero-sum. Um, and if you don't have more GPUs, you really have to figure out how do you make very, very hard trades, and I hate making hard trades for our users.

Apoorv Agrawal 40:45 Yeah.

Nick Turley 40:45 Um, hence the desire to, um, um, have more GPUs. But, uh, it's, it's useful to start with the most zero-sum trade-off when you do your planning. So I think starting-

Apoorv Agrawal 40:53 Fascinating

Nick Turley 40:53 ... working backwards from GPUs, um, is usually pretty, pretty good idea.

Apoorv Agrawal 40:57 Yeah. You know, one of the-- We, we have all these, uh, external data sources for charts of users and usage and-

Nick Turley 41:03 Mm-hmm

Apoorv Agrawal 41:03 ... act- activity and retention and all those things. What we don't have is tokens per user over time.

Nick Turley 41:09 Mm-hmm.

Apoorv Agrawal 41:09 And I bet that chart is like a s- sweet line going this way.

Nick Turley 41:13 I think internal is pretty good, or internal, uh, employees-

Apoorv Agrawal 41:16 [chuckles]

Nick Turley 41:16 ... is a pretty good indicator for what's about to happen.

Apoorv Agrawal 41:18 Yeah.

Nick Turley 41:18 And, um, yes, um, the charts are, are, are, are mind-boggling.

Apoorv Agrawal 41:22 Yeah. Yeah, yeah. Fascinating. Uh, uh, okay, a couple of quick ones on the present before we go into the, the landscape-

Nick Turley 41:28 Yeah

Apoorv Agrawal 41:28 ... which is, you know,

Apoorv Agrawal 41:30 shopping.

Nick Turley 41:31 Mm.

Apoorv Agrawal 41:31 Uh, you know, we, we just moved into a new house. We took some photos, and we were hoping that all our furniture would magically appear, that ChatGPT helped us paint.

Nick Turley 41:40 Mm-hmm. Mm-hmm.

Apoorv Agrawal 41:41 But, you know, a lot of, a, a lot, a lot of, uh, r- recent updates on ChatGPT shopping. Tell us, tell us about it. What are you thinking-

Nick Turley 41:47 Yeah

Apoorv Agrawal 41:48 ... on, on shopping as-- ChatGPT as a shopping assistant?

Nick Turley 41:51 Shopping is one of those use cases that exists organically in ChatGPT today, and they work. Um, you can ask ChatGPT about any purchase you might be planning and get pretty excellent advice.

Apoorv Agrawal 42:02 Mm.

Nick Turley 42:02 Um, but it's also one of those cases where the experience that exists in chat, um, today, it, it's not, it's not the perfect experience that you would want, um, because shopping is very visual, for example. So you're gonna wanna actually see products and images and-

Apoorv Agrawal 42:18 Mm

Nick Turley 42:18 ... be able to compare and contrast, not just read, you know, walls of text.

Apoorv Agrawal 42:21 Mm.

Nick Turley 42:21 Um, people care about the sources of, you know, um, um, um, you know, where can I learn more-

Apoorv Agrawal 42:28 Mm

Nick Turley 42:28 ... about, you know-

Apoorv Agrawal 42:29 Mm

Nick Turley 42:29 ... a given product, et cetera.

Apoorv Agrawal 42:30 Mm-hmm.

Nick Turley 42:31 Um, and, um, so there's a lot of work to do to make the discovery really, really good and allowing people to use ChatGPT as a, as a, um, as an assistant to find the right product to buy.

Apoorv Agrawal 42:41 Mm.

Nick Turley 42:42 Um, and that's where our focus lies-

Apoorv Agrawal 42:44 Mm

Nick Turley 42:44 ... is making that really great and making that really great in a way that works for our retail partners as well. Um-

Apoorv Agrawal 42:49 Mm

Nick Turley 42:49 ... because as I mentioned earlier, there's huge appetite from the ecosystem to be-

Apoorv Agrawal 42:53 Mm

Nick Turley 42:53 ... uh, part of the, the ChatGPT journey.

Apoorv Agrawal 42:55 Mm.

Nick Turley 42:57 And, um, nailing the discovery piece has, has, has, has, has been, um, the most promising, um, focus area to date.

Apoorv Agrawal 43:04 Nick, on Chat-ChatGPT, you must see a breadth of information. You must see a breadth of, uh, use cases that people are doing-

Nick Turley 43:11 Mm-hmm

Apoorv Agrawal 43:11 ... with ChatGPT and, and tell us something about, you know, what, what does the world underestimate about ChatGPT that you have maybe been surprised by or, or, or, or a listener might be su-su-surprised by?

Nick Turley 43:22 There's been a real change in the way that people think of ChatGPT-

Apoorv Agrawal 43:26 Mm-hmm

Nick Turley 43:27 ... um, over the last year or so, where, um, it's increasingly like, you know, a true thought partner to people.

Apoorv Agrawal 43:34 Mm-hmm. Mm-hmm.

Nick Turley 43:34 Um, it's not just a thing that, you know, answers your question, um, but it's a thing that you can... It's a sparring partner that you can actually-

Apoorv Agrawal 43:42 Yeah

Nick Turley 43:42 ... think things through with. And, um, that shows up in all kinds of domains, ranging from life advice where, you know, if you've got a-

Apoorv Agrawal 43:49 Mm-hmm

Nick Turley 43:49 ... relationship problem, you can actually get a lot of value out of ChatGPT just helping you think through how to handle it and how to talk to-

Apoorv Agrawal 43:55 Yeah

Nick Turley 43:55 ... you know, your partner about it, um, all the way to a work setting where, you know, you're, you're working on, um, um, an analysis or you're, like, trying to figure out how to frame-

Apoorv Agrawal 44:04 Yeah

Nick Turley 44:04 ... something or you're, you know, trying to build something-

Apoorv Agrawal 44:07 Mm-hmm

Nick Turley 44:07 ... and ChatGPT really shows up as, um, as a, um, second brain-

Apoorv Agrawal 44:12 Yeah

Nick Turley 44:13 ... um, of sorts, and, uh, I think that's qualitatively different in terms of how the mental po- model it occupies with people.

Apoorv Agrawal 44:20 Yeah.

Nick Turley 44:20 And you see it in the usage patterns and the use cases that exist, and I think the more we nail things like, you know, proactivity, which we talked about earlier, in tasks, et cetera-

Apoorv Agrawal 44:27 Yeah

Nick Turley 44:27 ... I think the more it's gonna feel like a teammate in the workplace-

Apoorv Agrawal 44:30 Yeah

Nick Turley 44:30 ... and like a super assistant at home.

Apoorv Agrawal 44:32 Yeah.

Nick Turley 44:32 And, uh, I think that's gonna, um, um, meaningfully change the, the, the use cases that people come for.

Apoorv Agrawal 44:38 Yeah. You know, I've been... The, the most high-stakes thing I do with ChatGPT is, uh-

Nick Turley 44:42 [laughs]

Apoorv Agrawal 44:43 We have a, we have a new baby.

Nick Turley 44:45 Mm.

Apoorv Agrawal 44:45 And when the baby's, uh, crying at 3:00 in the morning, ChatGPT-

Nick Turley 44:48 Yeah

Apoorv Agrawal 44:49 ... what, you know, what's, what's going on?

Nick Turley 44:51 First of all, congrats. Second of all-

Apoorv Agrawal 44:53 [laughs]

Nick Turley 44:53 ... I've heard this from all parents in my life-

Apoorv Agrawal 44:55 Yes

Nick Turley 44:55 ... that ChatGPT has become indispensable-

Apoorv Agrawal 44:57 Indispensable

Nick Turley 44:57 ... as a thought partner, and it makes sense, right? If you have a really specific scenario or you-

Apoorv Agrawal 45:02 Yeah

Nick Turley 45:02 ... you think it's a specific scenario to you-

Apoorv Agrawal 45:04 Yeah

Nick Turley 45:04 ... ChatGPT really comes through-

Apoorv Agrawal 45:06 [laughs]

Nick Turley 45:06 ... and can, um, you know, um, um, he-help you, you know, um, um, build confidence, and I think that's such an empowering thing.

Apoorv Agrawal 45:13 Mm-hmm.

Nick Turley 45:14 Right? Like, and I imagine new parents aren't always the most confident about what is the right thing to do.

Apoorv Agrawal 45:19 Yeah.

Nick Turley 45:19 And if ChatGPT can make you, um, you know, feel like you, you, you, you are, you have agency and control and, you know, you know, uh, I think, I think it's really valuable.

Apoorv Agrawal 45:28 Yeah. It's huge. Well, thank you for making ChatGPT. It's literally-

Nick Turley 45:32 [laughs]

Apoorv Agrawal 45:32 ... getting me an extra hour of sleep every day.

Nick Turley 45:34 It took a village, but, uh, that is a great metric.

Apoorv Agrawal 45:36 [laughs]

Nick Turley 45:36 You know, that should be the North Star metric-

Apoorv Agrawal 45:38 That's right

Nick Turley 45:38 ... is, like, incremental hours of sleep.

Apoorv Agrawal 45:39 That's great.

Nick Turley 45:40 That's a great one.

Apoorv Agrawal 45:40 Incremental hours of sleep.

Nick Turley 45:41 Yeah.

Apoorv Agrawal 45:41 Incremental hours of joy in life.

Nick Turley 45:43 There you go. I mean, you joke, but, like, we talk about this a lot and, um, because spiritually, that, that, that is pretty close to what we-

Apoorv Agrawal 45:50 Yeah

Nick Turley 45:50 ... hope we can do, right? Is, is, is help you re-reach whatever you consider self-actualization.

Apoorv Agrawal 45:55 Yeah.

Nick Turley 45:55 Uh, whether or not that's sleep or joy-

Apoorv Agrawal 45:57 Yeah

Nick Turley 45:57 ... uh, or any other goal you might have.

Apoorv Agrawal 45:59 Yeah. Yeah, yeah. Well, thank you for... Thank you to the village. We're gonna switch gears-

Nick Turley 46:03 Okay

Apoorv Agrawal 46:03 ... and talk about the landscape.

Nick Turley 46:05 Sure.

Apoorv Agrawal 46:05 There's a lot going on on the field. Um, you know, how would you frame ChatGPT's differentiation, uh, eh, to, to, to people out there? Um, there's a lot of different products out there.

Nick Turley 46:20 Look, it's the best time in history to be a consumer of technology.

Apoorv Agrawal 46:26 It is indeed.

Nick Turley 46:26 Um, because, uh, you got options and-

Apoorv Agrawal 46:29 [laughs]

Nick Turley 46:30 ... and the competition is intense.

Apoorv Agrawal 46:32 Yeah.

Nick Turley 46:32 And I think that's beautiful, and it's actually good for us too because if you were to pre-mortem why a company like OpenAI does not achieve its mission, it's probably focus because of the sheer number of opportunities that become possible when-

Apoorv Agrawal 46:44 Right

Nick Turley 46:44 ... you approach AGI, right? And having competition and, and options out there, I think, um, it, it forces us to focus on our customers too-

Apoorv Agrawal 46:52 Yeah

Nick Turley 46:52 ... um, and on things that really matter-

Apoorv Agrawal 46:54 Yeah

Nick Turley 46:54 ... which aren't always the most flashy things, right? Sometimes-

Apoorv Agrawal 46:56 Yeah

Nick Turley 46:56 ... it's latency, reliability-

Apoorv Agrawal 46:57 [laughs]

Nick Turley 46:58 ... um, the quality of the user experience.

Apoorv Agrawal 47:00 Yeah.

Nick Turley 47:00 So, uh, I think it's a really good thing.

Apoorv Agrawal 47:03 Yeah.

Nick Turley 47:03 I think the biggest differentiation of ChatGPT is the team behind it because-

Apoorv Agrawal 47:07 Mm-hmm

Nick Turley 47:07 ... you know, we're not static, right? Anything we build will get copied and, you know, sometimes in, in, in, in ways that are high craft, sometimes in ways that, um, you know, are sort of, um-

Apoorv Agrawal 47:16 For sure

Nick Turley 47:16 ... you know, um, checkboxes. And, um, it's really important to us that we evolve the category, um-

Apoorv Agrawal 47:22 Yeah

Nick Turley 47:22 ... and, um, build the super assistant that we've always imagined. And I think the way, the reason that I have confidence that that's possible at a speed that outpaces, you know, um, um, the, you know, the, the dynamic of being copied is that we have an amazing team-

Apoorv Agrawal 47:40 Yeah

Nick Turley 47:41 ... um, and that we have an amazing team across research and engineering and design and all the different functions that it takes to make something amazing, and I think our unique ability has been to bring those functions together-

Apoorv Agrawal 47:50 Mm

Nick Turley 47:50 ... to build something that is sort of at the intersection of useful and possible-

Apoorv Agrawal 47:53 Yeah

Nick Turley 47:54 ... right in that moment. Um, so, you know, my best answer for you is we keep pushing forward, and we hope to be, um, um, expanding what people think of this product as.

Apoorv Agrawal 48:03 You know, last, uh, winter, we had obviously, uh, you know, what was called Code Red. Google had a, a great model. Uh, there was a lot of, you know, talk about it, Marc Benioff switching-

Nick Turley 48:15 Mm

Apoorv Agrawal 48:15 ... very vocally to, to, to, to Gemini and us delaying ads and health agents and shopping, basically hit pause on everything, making ChatGPT better. Talk, talk to us about that moment, um, bo-both about what led to that, um, and what was happening in that moment.

Nick Turley 48:31 Yeah. Um, so first off, Code Reds are a tool we use, um, to create focus.

Apoorv Agrawal 48:37 Mm.

Nick Turley 48:37 And as you can imagine, when you're in a place like OpenAI, and this is what makes us special too, um, special to work here is, is, is there are so many different things-

Apoorv Agrawal 48:46 Mm

Nick Turley 48:46 ... going on. It's a research lab. We are pursuing many different ideas, right?

Apoorv Agrawal 48:49 Mm.

Nick Turley 48:50 And there's been these moments where, um, we've wanted the company to come together, um, um, to solve a problem- ... across boundaries, you know, no matter what your project might have been.

Apoorv Agrawal 49:01 Mm.

Nick Turley 49:01 At end of last year, we had one of those moments where we felt like we, we need to show up for our users. We need to focus the things, focus on the basics, um, like reliability, performance, um-

Apoorv Agrawal 49:11 Mm

Nick Turley 49:11 ... the way that talking to the model feels-

Apoorv Agrawal 49:13 Mm

Nick Turley 49:13 ... um, making personalization really great.

Apoorv Agrawal 49:15 Mm.

Nick Turley 49:15 Um, all these elements that, you know, our, our, our users care about. And, um, I loved it, um, because it, it, it was really an opportunity to work with a bunch of folks who I don't normally get to work with on making-

Apoorv Agrawal 49:26 Mm

Nick Turley 49:26 ... the product great.

Apoorv Agrawal 49:27 Mm.

Nick Turley 49:27 And we just exited the code red, um-

Apoorv Agrawal 49:30 Mm

Nick Turley 49:30 ... which we knew we would, um, with the launch of 5.3, which, you know, is a, is a great model for the everyday user. Um, it's great to talk to.

Apoorv Agrawal 49:37 Mm.

Nick Turley 49:37 And, um, 5.4, which is, um-

Apoorv Agrawal 49:39 Yes

Nick Turley 49:39 ... workhorse if you're trying to do real knowledge work.

Apoorv Agrawal 49:41 Mm.

Nick Turley 49:42 And, um, you know, undoubtedly we're gonna continue to use the tool, um, um, of a code red whenever we wanna create focus. Um, but um, I'm excited because I think, um, ChatGPT, um, is, is, is in a great spot.

Apoorv Agrawal 49:55 Yeah. So code red is over now.

Nick Turley 49:57 That's correct.

Apoorv Agrawal 49:57 It's not the new normal.

Nick Turley 49:59 Um, it's not the new normal. Uh, we want it to be a special thing, but it is a tool I, I suspect we will continue to use.

Apoorv Agrawal 50:05 That's great.

Nick Turley 50:05 Yeah.

Apoorv Agrawal 50:06 That's great. And, um, maybe tangibly, if you were to point at how did code red change ChatGPT, uh, or maybe the ops or, or how-

Nick Turley 50:14 Mm

Apoorv Agrawal 50:14 ... the team operates?

Nick Turley 50:15 The thing I try to get, like, you know, foster with the team is focus. Um-

Apoorv Agrawal 50:19 Yeah

Nick Turley 50:19 ... so we are certainly more focused than we were six months ago-

Apoorv Agrawal 50:22 Yeah

Nick Turley 50:22 ... on, um, you know, the, uh, the things we really wanna nail, and some of those things are very behind the scenes, like-

Apoorv Agrawal 50:29 Yeah

Nick Turley 50:29 ... latency, reliability-

Apoorv Agrawal 50:31 Yeah

Nick Turley 50:31 ... those kind of things.

Apoorv Agrawal 50:32 Okay.

Nick Turley 50:32 And some of those things are, like, very concerted efforts, like, you know, evolving-

Apoorv Agrawal 50:36 Yeah

Nick Turley 50:36 ... ChatGPT into the Super Assistant. And, um-

Apoorv Agrawal 50:38 Yeah

Nick Turley 50:38 ... so focus is the main lasting artifact.

Apoorv Agrawal 50:41 Mm.

Nick Turley 50:41 Um, and as you imagine, um, um, it's, it's hard to stay focused sometimes when there's so much going on in the space, but, um, that's the hard job. And you-

Apoorv Agrawal 50:50 Yeah

Nick Turley 50:50 ... had asked me about trade-offs earlier.

Apoorv Agrawal 50:51 Yeah.

Nick Turley 50:52 Um, um, gett- getting the team to focus on the things that really matter to users is certainly one of them, um-

Apoorv Agrawal 50:56 Yeah

Nick Turley 50:56 ... that's always worth it.

Apoorv Agrawal 50:58 Yeah. You know, in the back of my mind as I ask you that question-

Nick Turley 51:00 Yeah

Apoorv Agrawal 51:00 ... is all the other founders that are, that are in, in the arena-

Nick Turley 51:04 Mm-hmm

Apoorv Agrawal 51:04 ... right now. Uh, and, and just a reminder that, hey, code red is a tool for, for, for you. Wartime and founder, as you used to call it, is, is, is a tool. Um.

Nick Turley 51:12 Yeah. I, I think, you know, every company does it differently, um-

Apoorv Agrawal 51:15 Yeah

Nick Turley 51:15 ... in terms of how you, how you get stuff done. Um, but I think it, it's really valuable to have terminology that, um, you know, means something-

Apoorv Agrawal 51:22 Yeah

Nick Turley 51:22 ... um, that, you know, signals to people it's okay to drop your-

Apoorv Agrawal 51:26 Yeah

Nick Turley 51:26 ... other stuff, and it's okay to, you know, focus on this thing together, even if that wasn't your original job.

Apoorv Agrawal 51:30 Yeah.

Nick Turley 51:30 Um, so I think it works really well, um, at a place like OpenAI. Um, but-

Apoorv Agrawal 51:35 Yeah

Nick Turley 51:35 ... I imagine startups would have an equivalent.

Apoorv Agrawal 51:37 Yeah. You know, one of the things that caught everybody's imagination, uh, on our team was, uh, what Peter was doing at OpenClaw.

Nick Turley 51:43 Mm-hmm.

Apoorv Agrawal 51:44 Incredibly

Apoorv Agrawal 51:45 potent to put all the tools together. Obviously, Peter's a great builder. Congrats on, on, on bringing on Peter to the team. Um, tell us a little about what, what, what Peter's working on and when, when might the, uh, uh, the billions on, on ChatGPT have something to, to, to, to see there?

Nick Turley 52:00 Well, first of all, I'm very excited for, for Peter to be here. Um, I, uh, um, um, uh, was excited to have another German speaker in the house. Um-

Apoorv Agrawal 52:09 Oh, yeah. Peter's-

Nick Turley 52:09 Yeah. He is. He's Austrian, not German. So we're, we're, uh, um, exchanging Guten Morgens. Um, but the, uh, um... Yeah. The, the OpenClaw is so inspiring because, um, it brought to life in, in many ways, um, a vision that, that, that, you know, we'd had, um, um, in, in different forms-

Apoorv Agrawal 52:30 Mm-hmm

Nick Turley 52:30 ... you know, admittedly-

Apoorv Agrawal 52:32 Mm

Nick Turley 52:32 ... um, around this kind of AI that is fully embodied, that, you know-

Apoorv Agrawal 52:35 Mm

Nick Turley 52:35 ... exists across different UIs, that can do stuff for you, that has state, that has an interaction pattern that feels a little bit more like talking to a human, you know, because, you know, um, you know, OpenClaw allows you to, you know, interact in a very, very natural way-

Apoorv Agrawal 52:49 Mm-hmm

Nick Turley 52:49 ... where you can, you know, um, send many texts back and forth, and it-

Apoorv Agrawal 52:52 Mm

Nick Turley 52:52 ... is very curt. And so there's a lot of elements of, of, of, of OpenClaw that I think were very clarifying to-

Apoorv Agrawal 52:59 Okay

Nick Turley 52:59 ... to folks across the industry. And, um-

Apoorv Agrawal 53:01 Mm-hmm

Nick Turley 53:01 ... but the best, you know... I, I'm, I'm super excited to just, like, learn, learn from Peter and bring in, uh, into the company and figure out what we can do together. So there, there's, there's a lot more to come.

Apoorv Agrawal 53:11 All right. So now onto the most fun section, rapid fire.

Nick Turley 53:14 All right.

Apoorv Agrawal 53:15 You ready?

Nick Turley 53:16 Sure.

Apoorv Agrawal 53:16 We'll, we'll start with my most... my, my, my favorite game, which is long-short. Uh, pick an idea, a startup, a, a, a business, a product that you love, you think you're, you're very bullish on.

Nick Turley 53:27 Yeah. I'm, I'm... If I were starting a company today, I, I, I'm really excited about these companies that are going into companies and getting extremely hands-on-

Apoorv Agrawal 53:35 Mm

Nick Turley 53:35 ... and doing effect- effectively professional services-

Apoorv Agrawal 53:38 Mm

Nick Turley 53:38 ... um, with AI because we've saturated all the evals, um, and you need to get proximate to the problems.

Apoorv Agrawal 53:45 Mm.

Nick Turley 53:45 So, um, that's... It's, it's those companies that, that, um, I'm, I'm paying attention to.

Apoorv Agrawal 53:51 Fascinating. So this is, you know, this is an exam- This would be like, hey, you're going and either acquiring or, or, or going inside an operating forum that has scale and, and, and, and, and a humming engine-

Nick Turley 54:00 Exactly

Apoorv Agrawal 54:00 ... and making that a more efficient engine.

Nick Turley 54:02 Yeah. Or, or, or just like, you know, you're, you're doing contracts for, for, for, for customers that have really hard problems-

Apoorv Agrawal 54:07 Mm

Nick Turley 54:07 ... and you're actually going in and committing to solving the problem.

Apoorv Agrawal 54:10 Mm.

Nick Turley 54:10 Um, because-

Apoorv Agrawal 54:12 And getting outcomes.

Nick Turley 54:12 Yeah, because, like, you know, there's, there's a reason, I think, that we've made so much progress on math and coding, um, but m- not on many other domains because those are domains we are proximate to-

Apoorv Agrawal 54:21 Mm

Nick Turley 54:21 ... we as people who work in labs. And there's all kinds of other domains that we are not as proximate to.

Apoorv Agrawal 54:27 Mm.

Nick Turley 54:27 And if you get proximate, I think you can, you know, build something transformative, and I think this is more important now-

Apoorv Agrawal 54:32 Mm

Nick Turley 54:32 ... um, bec- precisely because, um, you know, the, the, the easy problems have been solved.

Apoorv Agrawal 54:38 Yeah.

Nick Turley 54:38 Um, um, the obvious problems have been solved by the models. Credit where credit is due, I think Not- NotebookLM is awesome and-

Apoorv Agrawal 54:44 Mm

Nick Turley 54:44 ... differentiated and helps me learn new stuff. I think it's great.

Apoorv Agrawal 54:47 It's so good.

Nick Turley 54:48 Yeah.

Apoorv Agrawal 54:48 It's so good.

Nick Turley 54:49 And I think this is the example of you can innovate, and you can build something, um, totally different. It's awesome.

Apoorv Agrawal 54:54 Yeah. Yeah, yeah. It's so good, particularly for, uh, I found it for some more technical, uh, uh, learning to be a very, uh, approachable way to, to-

Nick Turley 55:03 Totally

Apoorv Agrawal 55:03 ... to learn.

Nick Turley 55:04 And it's really cool. I feel like AI An underrated capability of AI is to just transform things into a different medium.

Apoorv Agrawal 55:11 Mm-hmm.

Nick Turley 55:12 And I think that's so important for learning. Like, we just la- um, um, launched these, like, dynamic, um, math blocks, which-

Apoorv Agrawal 55:19 Oh

Nick Turley 55:19 ... allow you to visually understand math inside ChatGPT. Um, learning is obviously-

Apoorv Agrawal 55:22 That's right

Nick Turley 55:23 ... a big use case for us too.

Apoorv Agrawal 55:24 Hmm.

Nick Turley 55:24 And I think just being able to transform things from text to visual, you know, soon from, you know, visual to video and, like, all these different media is amazing because people have such different ways of processing information. Um, and some people are, like, au- auditory learners, some people are visual, some people like reading. Uh, so I think that's really magical, um, and, um, a great, great angle to take.

Apoorv Agrawal 55:45 Yeah. Amazing. Amazing, amazing. Uh, you know, one of the things I think about a lot is education and, and, and education for, for, for kids now in school. Um, the world's changing so fast.

Nick Turley 55:57 Mm-hmm.

Apoorv Agrawal 55:57 I'm not sure our education system's changing that fast.

Nick Turley 56:00 Yeah.

Apoorv Agrawal 56:01 What advice would you have for, for students who are in school now, um, you know, who might have to adapt faster than the system around them might adapt?

Nick Turley 56:11 It's a really good question and something that I've thought a lot about, um, myself. And, you know, the... I think the most important perma skill in this era is, uh, curiosity-

Apoorv Agrawal 56:28 Yeah

Nick Turley 56:28 ... I think because if the machine can answer all your questions, you better have good questions.

Apoorv Agrawal 56:35 [laughs]

Nick Turley 56:36 Um, and the only way to have good questions, I think, is to pursue the things you were actually excited about-

Apoorv Agrawal 56:42 Yeah

Nick Turley 56:42 ... from an early age and throughout your entire life.

Apoorv Agrawal 56:44 Yeah.

Nick Turley 56:44 And, um, I reflect on this because the only reason I'm here and working on this stuff is because I thought it was neat, uh, when I got, you know-

Apoorv Agrawal 56:51 [laughs]

Nick Turley 56:51 ... nerd sniped in, um, um-

Apoorv Agrawal 56:53 That's right

Nick Turley 56:54 ... in the interview process, right? And it was like-

Apoorv Agrawal 56:55 That's right

Nick Turley 56:56 ... this is so cool. And, and so no matter what you're doing, I think that's an important skill is to be curious and learn to stay curious. And I think, I'm, I'm confident that, um, if you foster that skill, um, you will know how to adapt to, you know, an evolving-

Apoorv Agrawal 57:13 Yeah

Nick Turley 57:13 ... landscape of tools and AIs and, and-

Apoorv Agrawal 57:15 Yeah

Nick Turley 57:15 ... jobs. Um, so that would be my advice.

Apoorv Agrawal 57:17 Yeah. Curiosity is, has always been the perma skill. Our friend, uh, Bill Gurley wrote about it in, in his book, Running Down a Dream. Um, about curiosity.

Nick Turley 57:25 I'm gonna have to check that out.

Apoorv Agrawal 57:26 Yeah. Um, what, what is a job that, uh, gets more valuable, not less, as, uh, AI gets better, as, as AGI arrives?

Nick Turley 57:37 Well, I think maybe the easy answer is being an entrepreneur-

Apoorv Agrawal 57:39 [laughs]

Nick Turley 57:40 ... um, um, because it's, it's the best time to build ever-

Apoorv Agrawal 57:44 Yeah

Nick Turley 57:44 ... in terms of, like, being able to self-actualize your-

Apoorv Agrawal 57:48 Yeah

Nick Turley 57:48 ... your idea.

Apoorv Agrawal 57:49 Yeah.

Nick Turley 57:49 But maybe, maybe one that is maybe non-obvious is-

Apoorv Agrawal 57:52 Mm-hmm

Nick Turley 57:52 ... I, I think writing actually-

Apoorv Agrawal 57:53 Hmm

Nick Turley 57:54 ... um, is very important. And it's not-

Apoorv Agrawal 57:56 Yeah

Nick Turley 57:56 ... because the AI can't write. You know, AI will become amazing at writing just-

Apoorv Agrawal 58:01 Hmm

Nick Turley 58:02 ... like any other domains. But because-

Apoorv Agrawal 58:03 Yeah

Nick Turley 58:03 ... I think the skill of writing forces you to be very clear-

Apoorv Agrawal 58:07 Yeah

Nick Turley 58:07 ... on what you have to say. And even though prompt engineering is obviously gonna go away and has gone away to, to much extent-

Apoorv Agrawal 58:15 Hmm

Nick Turley 58:16 ... the idea of expressing what you want-

Apoorv Agrawal 58:19 Hmm

Nick Turley 58:19 ... to a machine-

Apoorv Agrawal 58:20 Hmm

Nick Turley 58:20 ... requires you to be a pretty good writer and a very precise, um, um, writer. So I, I would say that that is a, you know-- And any profession that involves very clear writing and therefore thinking, I think is, is, is, um, um, well set up.

Apoorv Agrawal 58:36 Yeah. 100%. Honestly, I mean, there-- this is the whole thing about Sloth, right? There's just so much AI Sloth.

Nick Turley 58:43 That's the other thing. I think there's gonna be a permanent need for high-quality, trusted, authoritative content. And tools like ChatGPT can help you discover that content.

Apoorv Agrawal 58:54 Yeah.

Nick Turley 58:54 But I think the need for amazing, um, um, content is, is, is also here to stay.

Apoorv Agrawal 58:59 Uh, and final question: What has been your AGI, feel the AGI moment? When did you feel it?

Nick Turley 59:04 I've had so many, honestly. Um, and it, it's, it's definitely not stopped. Um, a few weeks or so after I joined OpenAI, GPT-4 had finished training.

Apoorv Agrawal 59:19 Hmm.

Nick Turley 59:21 And I remember trying it out, and it actually-- it didn't impress me at all-

Apoorv Agrawal 59:25 [laughs]

Nick Turley 59:25 ... um, and- nor anyone else that week 'cause it kinda didn't work. And it's because we hadn't figured out how to post-train it. And, uh, I think seeing it go from kind of, "Wait, is this really a thing?" Or was GPT-3 kind of it, to, "Wow, actually, this is an entire step change," with what felt to me at the time, who didn't understand much about AI at all, as, like, just some tweaks or some-- a little bit of final stretch work-

Apoorv Agrawal 59:53 Hmm

Nick Turley 59:53 ... was profoundly humbling 'cause you can realize that you might-- it might not look like we are close-

Apoorv Agrawal 59:59 Yeah

Nick Turley 59:59 ... to really powerful, useful AI, but we probably are. Um-

Apoorv Agrawal 1:00:04 Yeah.

Nick Turley 1:00:04 And then the moment that, you know, really, you know-- There was two things that GPT-4 did that felt like AGI to me. Um-

Apoorv Agrawal 1:00:10 Yeah

Nick Turley 1:00:11 ... one is it could do poetry, and I didn't think it was possible for-

Apoorv Agrawal 1:00:13 Yeah

Nick Turley 1:00:13 ... an AI model to do poetry. It just kind of fundamental, philosophically, it just didn't feel, like, in scope. [laughs] And then the other one was it could produce code that actually worked and compiled.

Apoorv Agrawal 1:00:24 Yeah.

Nick Turley 1:00:24 And then at my next moment where I stared at the ceiling just in awe was when I realized GPT-4 could just simulate an entire computer terminal. Um, um, like a, you know, a full computer with commands, et cetera. And I'm like, "Wait, how would this be imbued in a, in a, in a language model?"

Apoorv Agrawal 1:00:40 Yeah.

Nick Turley 1:00:40 And there's been so many moments since then, honestly. Like-

Apoorv Agrawal 1:00:43 Yeah

Nick Turley 1:00:43 ... reasoning was a moment. O-one of the moments was when, I, I think Mark and I were, uh, uh, giving a demo of reasoning, uh, in front of the, um, whole company.

Apoorv Agrawal 1:00:53 Hmm.

Nick Turley 1:00:53 Um, and, um, this was a moment where we were still trying to kind of find use cases that were hard enough for the AI, for the reasoning to, you know, make a, make a difference. You know, we're way past that point, we know. But at the time, you know, I think we were having to do a puzzle in front of everyone. And, um, I think one of the moments that made me totally feel the AGI is like, um, we were in the middle of the demo, and everyone started laughing. And I was like, "Wait, what, what is funny?" And then I stared at the screen because we were showing this, this chain of thought as it was streaming out of the model. Um, and the model swore and said like, "Oh, damn it, may have to adjust," because it realized it had made a mistake in the puzzle. And, uh, the fact that it did that, but in particular the fact that it did that in a way that was entirely emergent from the, you know, RL process completely blew my mind, um, and, you know, made me, you know, um, uh, f- f- feel, feel quite humble about what else these models might be able to do. So, um, uh-

Apoorv Agrawal 1:01:46 Yeah

Nick Turley 1:01:46 ... that was one of those moments.

Apoorv Agrawal 1:01:47 Yeah.

Nick Turley 1:01:47 Uh, and then most recently watching people use Codex, like, like-

Apoorv Agrawal 1:01:50 So cool

Nick Turley 1:01:51 ... watching people have, walk around with their computer open, uh-

Apoorv Agrawal 1:01:54 [laughs]

Nick Turley 1:01:55 ... because they don't want the task to end. Um, yeah, uh, watching people who have never coded in their life make stuff and bring ideas, uh, to life-

Apoorv Agrawal 1:02:04 Yeah

Nick Turley 1:02:04 ... that feel, feels like an AGI. So, uh, honestly, it, it just, it's just accelerating for me, and it doesn't wear off at all. Um-

Apoorv Agrawal 1:02:12 Yeah

Nick Turley 1:02:12 ... and, uh, everyone has a different thing obviously, but those were some of mine.

Apoorv Agrawal 1:02:17 Yeah. You know, it's, uh, 10 years ago there was a product called Kite. I don't know if you remember. It was for software engineers. It was like an AI coding product.

Nick Turley 1:02:24 Mm-hmm.

Apoorv Agrawal 1:02:25 That's when I, that's when I felt the hunger for, for, for, for, for, for personal AI, and, you know, nothing happened for 10 years, and then everything happened-

Nick Turley 1:02:32 [laughs]

Apoorv Agrawal 1:02:32 ... in the last 10 months.

Nick Turley 1:02:34 The timing thing is really hard because it's, it's actually quite possible to predict where things will end up, I think-

Apoorv Agrawal 1:02:40 Yeah

Nick Turley 1:02:40 ... um, in terms of the kind of product and form factors you're gonna have. But to know when it happens, it's really hard for me to make statements on anything between-

Apoorv Agrawal 1:02:46 Yeah

Nick Turley 1:02:46 ... sort of eventually and in three months-

Apoorv Agrawal 1:02:49 Yeah

Nick Turley 1:02:49 ... because of all the ambiguity around, you know-

Apoorv Agrawal 1:02:52 Yeah

Nick Turley 1:02:52 ... um, um-

Apoorv Agrawal 1:02:53 Well, that's a tight enough window, you know?

Nick Turley 1:02:55 Yeah.

Apoorv Agrawal 1:02:55 Now in three months is a tight enough window.

Nick Turley 1:02:56 Three months is pretty, pretty okay.

Apoorv Agrawal 1:02:59 Yeah.

Nick Turley 1:02:59 Try to stick to the three-month plan more or less. Um-

Apoorv Agrawal 1:03:02 Yeah

Nick Turley 1:03:02 ... um, though, you know, my, my team would probably tell me we don't, but I, I try. But yeah, anything in between three months and eventually is, is, is difficult. Um-

Apoorv Agrawal 1:03:10 Yeah. Yeah, yeah. Well, thanks for doing it. You've got a lot going on. This was a total treat. We're so excited to see all the great products you release for us. Um, you know-

Nick Turley 1:03:20 Give my best

Apoorv Agrawal 1:03:20 ... if we ever, we can do anything to be of help, let us know.

Nick Turley 1:03:23 Awesome. Thanks very much. Thanks for having me.

Apoorv Agrawal 1:03:24 Of course, man. This was fun.

Nick Turley 1:03:25 Yeah. [upbeat music]

Apoorv Agrawal 1:03:36 As a reminder to everybody, just our opinions, not investment advice