OpenAIInside ChatGPT, AI assistants, and building at OpenAI — the OpenAI Podcast Ep. 2
EVERY SPOKEN WORD
75 min read · 15,473 words- 0:00 – 0:40
Intro: Meet Nick Turley and Mark Chen
- AMAndrew Mayne
Hello, I'm Andrew Mayne, and this is the OpenAI Podcast. My guests today are Mark Chen, who is the Chief Research Officer at OpenAI, and Nick Turley, who is the Head of ChatGPT. We're gonna be talking about the early viral days of ChatGPT. We're gonna talk about ImageGen, how OpenAI looks at code and tools like Codex, what kind of skills they think that we might need for the future, and we're gonna find out how ChatGPT got its totally normal name.
- MCMark Chen
Even half of research doesn't know what those three letters stand for.
- NTNick Turley
You know, you're gonna have an intelligence in your pocket, that it can be your tutor, it can be your advisor, it can be your software engineer.
- MCMark Chen
There was a real decision the night before. Do we actually launch this thing?
- 0:40 – 3:50
Origin of the name "ChatGPT"
- AMAndrew Mayne
First off, how did OpenAI decide on that awesome name?
- NTNick Turley
Uh, it was gonna be Chat with GPT-3.5, and we had a late-night decision to simplify [chuckles] -
- AMAndrew Mayne
Wait, wait, so say that again- say that name again.
- NTNick Turley
Uh, it was gonna be Chat with GPT-3.5-
- AMAndrew Mayne
Chat-
- NTNick Turley
... which rolls off the tongue even, even more nicely.
- AMAndrew Mayne
That's, uh- and, and you said that was a late-night decision, meaning, like, weeks before you finally decided what to call it, right?
- NTNick Turley
Right, right, right. No, weeks before, we hadn't started on the project yet, I think.
- AMAndrew Mayne
Oh, goodness.
- NTNick Turley
But, you know, I think we, we realized that that would be hard to pronounce and, um, came up with a great name instead.
- AMAndrew Mayne
So that was the night before? [chuckles]
- NTNick Turley
Roughly.
- AMAndrew Mayne
Yeah.
- NTNick Turley
Might have been the day before.
- AMAndrew Mayne
Yeah.
- NTNick Turley
It was all kind of a blur at that point.
- AMAndrew Mayne
I would imagine a lot of that was a blur, and I remember here, uh... I remember being in a meeting where we talked about the low-key research preview, which, like, really was, like, we really thought like, "Oh, this is..." 'Cause it's- it was the 3.5. 3.5 was a model that had been out for months, and from a capabilities point of view, when you just look at the evals, you're like, "Yeah, it's the same thing, but we just put the interface in here and made it so you didn't have to prompt as much." And then ChatGPT comes out, and when, when was the first sign that this thing was blowing up?
- NTNick Turley
I mean, I'm curious for- every- everyone has their slightly own recollection of that, that era, because it was a very confusing time. But for me, day one was sort of, you know, is the dashboard broken? Classic, like, uh, the logging can't be right. Day two was like, "Oh, weird." I guess, like, J- Japanese Reddit users discovered- [chuckles] ... this thing. Maybe it's, like, a local phenomenon. Day three was like, "Okay, it's going viral, but it's definitely gonna die off." And then by day four, you're like, "Okay, yeah, it's gonna, gonna change the world."
- AMAndrew Mayne
Mark, did you have any expectation about that, about-
- MCMark Chen
No, honestly, I mean, we've had so many launches, so many previews over time, and yeah, this one really was something else, right? The takeoff ramp was huge, and yeah, my parents just stopped asking me to go work for Google. [chuckles]
- AMAndrew Mayne
Wait, so wait, wait, wait a second. Up until ChatGPT, your parents were asking, like, what you were doing here?
- MCMark Chen
Yeah. No, I mean, um, they, they just never heard of OpenAI. Um-
- AMAndrew Mayne
Right
- MCMark Chen
... I think for many years, thought AGI was this pie-in-the-sky thing, and I wasn't having a serious job. So-
- AMAndrew Mayne
[chuckles]
- MCMark Chen
... it was a real revelation for them. Yeah.
- AMAndrew Mayne
Uh, what was your job title at the time?
- MCMark Chen
Um, I think just member of technical staff.
- AMAndrew Mayne
Member of technical staff?
- MCMark Chen
Yeah.
- 3:50 – 7:00
ChatGPT’s viral takeoff
- AMAndrew Mayne
it accelerate, I knew it was gonna happen, and then when it did, it was when it was on South Park. And remember that, when South Park made fun of the name, and-
- NTNick Turley
That was the first time I'd watched South Park in-
- AMAndrew Mayne
Oh
- NTNick Turley
... uh, let's just say a while. And that episode, I still think it's magic, and-
- AMAndrew Mayne
Yeah
- NTNick Turley
... it was obviously profound to watch and see, you know, something you, you helped make show up in pop culture. But there's the punchline in the end, where it's like, "Oh, this was co-written by ChatGPT?" That was so-
- AMAndrew Mayne
I think they took that off, though.
- NTNick Turley
I think they did.
- AMAndrew Mayne
I think in later episodes, 'cause it used to say, I think, "Written by, like, uh-
- NTNick Turley
Oh, man
- AMAndrew Mayne
... Trey Parker and, like, ChatGPT"
- NTNick Turley
I was not remembering that.
- AMAndrew Mayne
And then... No, it was. And then I think later, I think they may have pulled that off at some point. I don't remember, like-
- NTNick Turley
Oh, I strongly feel that you shouldn't have to give credit to... It's-
- AMAndrew Mayne
Yeah, that was-
- NTNick Turley
It's your business, whether or not you're using the-
- AMAndrew Mayne
If I had to give credit to ChatGPT for every aspect of my life, um, well, might as well just say ChatGPT maybe with Andrew.
- NTNick Turley
True.
- AMAndrew Mayne
So it's-
- MCMark Chen
Do you use it for prep for your interviews?
- AMAndrew Mayne
You know, one of my, my co-producers, Justin, probably uses it. I haven't asked him yet, 'cause I'd like to think that he's handcrafting every single question that we're thinking about here, but I am sure. You say it was a bit of a blur, and I'll tell you, like, a standout moment for me at the launch of ChatGPT was, I don't know if you remember this, but the Christmas party. And we'd had several weeks of ChatGPT out there, and Sam Altman went up and said, "Hey, this has been exciting to watch this, but the Internet being the Internet," and I think we all felt this way, "it's gonna die down."
- NTNick Turley
Mm.
- AMAndrew Mayne
Spoiler alert: It did not die down, and it just kept accelerating. What were the things you had to do internally to sort of keep this thing up and running as more people wanted to use it?
- NTNick Turley
We had, you know, quite a few constraints. And if, if, if... For those of you who remember, you know, I, I think you guys remember ChatGPT was down all the time-
- AMAndrew Mayne
Yeah
- NTNick Turley
... in the beginning. Um, and that was... Yeah, we'd said, "Hey, this is a research preview. No, no guarantees, and maybe it goes down." But the minute you had people loving and using this thing, that didn't feel super good. So, you know, people were certainly working around the clock to keep the site up. I remember, you know, we obviously ran out of GPUs, we ran out of database connections. We had, you know, um... We're getting rate limited in some of our providers. It- nothing was really set up to run a product. So in the beginning, we just built this thing, we called it the Fail Whale, and it would just tell you- [chuckles] ... kind of nicely that the thing was down, and made a little poem, I think it was generated by GPT-3, um, about being down, and, and it was sort of tongue-in-cheek.... and that got us through the winter break, 'cause we did want people to have some sort of a holiday. And then when we came back, we were like, "Okay, this is clearly not viable. You can't just go down all the time." Um, and eventually we got to something we could serve everyone.
- MCMark Chen
Yeah, and I think, you know, the demand really speaks to the generality of ChatGPT, right? Um, we had this thesis that ChatGPT embodied what we wanted in AGI, just because it was so general.
- NTNick Turley
Mm-hmm.
- MCMark Chen
And I think, you know, you're seeing that demand ramp just because people are realizing, you know, any use case that I want to, to give or to throw to the model, it can handle.
- AMAndrew Mayne
We were kind of known as the company working on AGI, and I think prior to ChatGPT, the API was certainly the first time we had a public offering where people could go use it and do it, but then it was more for developers-
- 7:00 – 9:40
Internal debate before launch
- AMAndrew Mayne
was everybody at OpenAI on board with ChatGPT being useful or being ready to launch?
- MCMark Chen
Yeah, I don't think so. You know-
- AMAndrew Mayne
Yeah
- MCMark Chen
... um, even the night before, I mean, there's this very famous story at OpenAI of, uh, you know, Ilya-
- AMAndrew Mayne
Mm
- MCMark Chen
... taking 10 cracks at the model, you know, 10 tough questions, and my recollection is maybe only on five of them, he got answers that he thought were acceptable.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
And so there's a real decision the night before, "Do we actually launch this thing? Is the world actually gonna respond to this?" And I think it just speaks to when you build these models in-house, uh, you so rapidly adapt to the capabilities.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
And it's hard for you to kind of put yourself in the shoes of someone who hasn't kind of been in this model training loop, and see that the- there is real magic there.
- AMAndrew Mayne
Yeah.
- MCMark Chen
Yeah.
- NTNick Turley
Yeah, I think to build on that, like, the controversy internally-
- AMAndrew Mayne
Mm
- NTNick Turley
... about, you know, is this thing good enough to launch? I think it is humbling, right? Because it, it's just a reminder of how wrong we all are when it comes to AI. It's why, you know, frequent contact with reality-
- MCMark Chen
Yeah
- NTNick Turley
... is so important.
- AMAndrew Mayne
Could you elaborate more on that contact with reality?
- NTNick Turley
Mm-hmm.
- AMAndrew Mayne
What does that mean?
- MCMark Chen
Yeah, I mean, when you think about iterative deployment, uh, one way I like to frame it is, you know, there's no point everyone agrees where it's suddenly useful, right?
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
Um, and I think usefulness is this big spectrum. Um, and so, you know, there's not one capability level or one bar that you meet, and suddenly, you know, the model is useful for everyone.
- AMAndrew Mayne
Were there any hard decisions about what to include or what to focus on?
- NTNick Turley
We were very, very principled on ChatGPT-
- AMAndrew Mayne
Mm
- NTNick Turley
... to not balloon the scope. Um, we were-
- AMAndrew Mayne
Mm
- NTNick Turley
... we were adamant to get feedback and data, um, as quickly as we could, so there's a lot of things that-
- AMAndrew Mayne
I'm always in Slack telling you things, by the way. [chuckles]
- 9:40 – 11:00
Evolution of OpenAI’s launch approach
- MCMark Chen
Yeah, I think over time, you know, feedback really has become an integral part of how we build the product, and it's also become an integral part of safety.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
And so you always feel the time cost of losing out on feedback. You know, you can deliberate in a vacuum, right?
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
Uh, are they gonna respond to this better? Are they gonna respond to that better? Um, but it's just not a substitute for just bringing it out there, right? Um, I think our philosophy is let the models have contact with the world-
- AMAndrew Mayne
Mm
- MCMark Chen
... and if you need to revert something, that's fine. But I think, uh, there's really no substitute for this fast feedback, and it's become one of the big levers for how we improve model performance, too.
- NTNick Turley
It's sort of funny, like, I feel like we started with shipping these models in a way that is more similar to hardware, where you make, like, one launch-
- MCMark Chen
Mm
- NTNick Turley
... very rarely, and it has to be right. And, you know, you're not gonna update the thing, and then you're gonna work on the next big project, and it's capital intensive, and the timelines are long. And over time, and I think ChatGPT was kind of the beginning, it's looked-
- MCMark Chen
Mm-hmm
- NTNick Turley
... more like software to me, where you make these frequent updates.
- MCMark Chen
Mm-hmm.
- NTNick Turley
Um, you have kind of a constant pace the world can adopt. If something doesn't work, you roll it back, and you sort of lower the stakes in doing that, and you low- you increase the empiricism, and, and of course, just operationally, too, you can innovate faster in a, in a way that is more and more in touch with what users want.
- AMAndrew Mayne
Yeah, one of the examples we had of that was the, the model becoming, uh, too obsequious or sycophantic.
- NTNick Turley
Mm.
- AMAndrew Mayne
Could you explain what
- 11:00 – 14:45
The sycophancy incident and RLHF
- AMAndrew Mayne
happened there? Well, that was where people all of a sudden say, "Hey, it's, it's telling me I've got 190 IQ, and I'm the most handsome person in the world," which I had no problem with personally, but other people did. [chuckles] And what was going on there?
- MCMark Chen
Yeah, so I think, um, one important thing is we rely on user feedback-
- AMAndrew Mayne
Mm-hmm
- MCMark Chen
... to improve the models, right? And it's this very complicated mix of reward models, which we use in, uh, a procedure we call RLHF, right? Uh, using human feedback to use RL to improve the models. And-
- AMAndrew Mayne
Could you give me just, like, a brief example what that would mean?
- MCMark Chen
Yeah, yeah. So I think, um, one way to think about it is, you know, when a user enjoys a conversation, you know-
- AMAndrew Mayne
Mm
- MCMark Chen
... they provide some positive signal, um, and-
- AMAndrew Mayne
Thumbs up.
- MCMark Chen
Yeah, a thumbs up, for instance.
- AMAndrew Mayne
Okay.
- MCMark Chen
And, uh, we train the model to prefer to respond in a way that would elicit more thumbs up, right?
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
And this may be obvious in retrospect, but, um, stuff like that, if balanced incorrectly, can lead to the model being more sycophantic, right? Um, you can imagine users might want that kind of, uh, that feeling of, you know, a model saying good things about them.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
But, um, I don't think it's a very good long-term outcome. And actually, when we look at kind of our response to-... uh, sycophancy and, and the rollout that resulted there. Um, I think there were a lot of good points about it. You know, this was something that was flagged just by a small fraction of our power users. It wasn't, you know, something that a lot of people who generally use the models noticed, and I think we really picked that out fairly early. We responded to it, I think, with the appropriate level of gravity.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
And, um, yeah, I think it, it just shows that, you know, we really do take these issues quite seriously, and we want to intercept them very early.
- AMAndrew Mayne
Yeah, it felt like there was maybe 48 hours since the model came out, and then-
- MCMark Chen
Mm
- AMAndrew Mayne
... Joanne Jiang had a response explaining exactly what happened, and I think that that's the, that's the hard part. How do you navigate that? Because the problem with social media is you're basically monetized by engagement time. You want to keep people on there longer-
- MCMark Chen
Mm
- AMAndrew Mayne
... so you can show them more ads, and certainly the more people use ChatGPT, obviously there's a cost to OpenAI. The idea is maybe use it once and stay around forever, but that's not practical. How do you weigh that, the idea of making people happy with what they're getting versus making the model, you know, be broadly more useful than just pleasing?
- NTNick Turley
I feel very lucky in this regard because we have a product that's very utilitarian, and people use it to either achieve things that they do know how to do but don't feel like doing, um, faster or with less effort, um, or they're, they're using it to do things that they couldn't do at all. Um, you know, first example is maybe, you know, writing an email that you've been dreading. Second example might be, you know, running a data analysis that you didn't actually know how to do- [chuckles] ... um, in Excel, you know. Um, um, true story. So, so, you know, th- those are very utilitarian things, and fundamentally, as you improve, you actually spend less time on the product, right?
- AMAndrew Mayne
Mm.
- NTNick Turley
Because, you know, ideally it takes less turns back and forth, or maybe you actually delegate to the AI, so you're not in the product at all. So for us, you know, time spent, it's very much not the, not the, not the thing we optimize for. You know, we do care about, um, your long-term retention-
- AMAndrew Mayne
Mm
- NTNick Turley
... because we do think that's a sign of value. Um, if you're coming back three months later-
- AMAndrew Mayne
Mm
- NTNick Turley
... that clearly means we did something right. Um, but what that means is, you know, um, I always say, um, "Show me the incentive, and I'll show you the outcome." We, we have, I think, the right fundamental, um, incentives to build something great. Um, that doesn't mean we'll always get it right. Um, the sycophancy, um, events were, were really, really important and good learning for us, and I'm, I'm proud of how we acted on it. Um, but fundamentally, I think we have the right, um, the right setup to build something awesome.
- 14:45 – 20:00
Balancing usefulness vs. neutrality in model behavior
- AMAndrew Mayne
on when, you know, ChatGPT came out, there was, like, the, the allegations, "It's woke. It's woke, and people are trying to promote some sort of, like, agenda from it." And my argument always been, like, you train a model on, you know, kind of on corporate speak, you know, average news, and a lot of academia, that's gonna kind of follow into that. And I remember Elon Musk was very critical about it, and then when he trained the first version of Grok, it did the same thing. And then he's like, "Oh, yeah, when you trained it on this sort of thing, it did that." And internally, at OpenAI, there were discussions about how do we make the model not try to push you, not try to steer you? Could you go a little bit how you try to make that work?
- MCMark Chen
Yeah. So I think, um, at its core, it's a measurement problem, right?
- AMAndrew Mayne
Mm.
- MCMark Chen
And I think it's actually bad to downplay these kind of concerns because they are very important things.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
Right? And, um, we need to make sure that the model, the default behavior that you get is something that's centered, that, you know, doesn't reflect bias-
- AMAndrew Mayne
Mm
- MCMark Chen
... um, on the political spectrum, um, or in, in many other, you know-
- AMAndrew Mayne
Mm
- MCMark Chen
... uh, axes of bias. And at the same time, you know, you do want to allow the user the capability to... You know, if you, you wanted to talk to, a, a reflection of something with more conservative values, to be able to steer that a little bit, right?
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
Um, or liberal values, right? And, and so I think the thing is, you want to make sure the defaults are meaningful, and they're centered, and that's a measurement problem.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
And you also want to give ability, some flexibility, right, within bounds, to steer the model to be a persona that you want to talk to.
- NTNick Turley
I think that's right. Um, I think, you know, in addition to neutral defaults, ability to sort of bring your own values to some extent-
- AMAndrew Mayne
Mm
- NTNick Turley
... I think, you know, being transparent about the whole thing is I think really, really important. Uh, I'm not a fan of, of, you know, secret system messages that, you know, try to, like-
- AMAndrew Mayne
Mm
- NTNick Turley
... you know, hack the model into saying or not saying something.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
Um, what we've tried to do is publish our spec, so you can go look at, you know, if you're getting certain model behavior, is that a bug? Um, you know, is it in violation of our own stated spec, or is it actually in the spec, in which case you know who to criticize and who to-
- AMAndrew Mayne
Mm
- NTNick Turley
... who to yell at, or is it just underspecified in the spec, in which case that allows us to improve it and add more specificity into that document. So by sort of publishing the rules of the AI that it's supposed to be following, um, I think that's an important step to have more people contribute to the conversation than just the people inside of OpenAI.
- AMAndrew Mayne
So we're talking about, like, the system prompt, the part of the instruction that the model gets before the user puts the input in.
- MCMark Chen
Well, I think it's more than that.
- NTNick Turley
Yeah.
- MCMark Chen
Yeah.
- AMAndrew Mayne
Yeah.
- NTNick Turley
Um, the system prompt is one way to steer the model-
- AMAndrew Mayne
Mm
- 20:00 – 22:50
Memory and the future of personalization
- AMAndrew Mayne
I, I find myself having longer conversations with it. I like the memory function.
- NTNick Turley
Mm.
- AMAndrew Mayne
I like the fact you can turn it off if you don't want. And I think about, like, you know, what's this gonna be two years from now or three years from now when it has a much longer memory, much more context with this? I like the idea to have these sort of like, you know, memento anonymous modes too-
- NTNick Turley
Mm
- AMAndrew Mayne
... where it's not gonna store this.
- NTNick Turley
Mm.
- AMAndrew Mayne
But I, I kind of wonder how much you've been thinking about two years, three years down the road. What, what's that going to be like when ChatGPT knows way more about you?
- MCMark Chen
Yeah, I mean, I think memory is just such a powerful feature.
- AMAndrew Mayne
Mm.
- MCMark Chen
In fact, it's one of the most requested features when we-
- AMAndrew Mayne
Mm
- MCMark Chen
... talk to people externally. Um, it's like, "This is the thing I really want to pay, pay more for." And I think, um, you know, you liken it to, if you've ever kind of had a personal assistant, you know, you-
- AMAndrew Mayne
No, I have not. [laughing]
- MCMark Chen
[laughing] Well, you, you do need to build up context-
- NTNick Turley
Not relatable, Mark.
- MCMark Chen
Over time.
- AMAndrew Mayne
Me neither. [laughing]
- NTNick Turley
[laughing]
- MCMark Chen
I'm sorry, guys. I'm sorry, guys. But, you know, it's, yeah, it's just like, it's, um, kind of in any kind of relationship that you have with a person, right?
- AMAndrew Mayne
Mm.
- MCMark Chen
You, you build up context with them over time.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
Um, and I think just the more they know about you, right, the richer the relationship, the more, you know, um, they can also help you, right? Uh, you can, uh, work together to collaborate on tasks together.
- AMAndrew Mayne
I, I do become-
- MCMark Chen
Yeah
- AMAndrew Mayne
... self-conscious of the fact that, like, it knows everything about me when I'm grumpy, and I've, I've, I've argued with it recently, by the way. Um-
- NTNick Turley
That's good.
- AMAndrew Mayne
Yeah.
- NTNick Turley
Um, you should be able to argue with it, and-
- AMAndrew Mayne
Yeah
- 22:50 – 29:00
ImageGen’s breakthrough moment
- AMAndrew Mayne
seemed like it preferred a certain kind of image, and a lot of the utility and the capabilities for variable binding was sort of, kind of hidden away.
- NTNick Turley
Mm-hmm.
- AMAndrew Mayne
And then ImageGen was kind of just this breakthrough moment that it caught me off guard.
- NTNick Turley
Mm.
- AMAndrew Mayne
How did you guys feel about the launch of that?
- MCMark Chen
Yeah, honestly, it caught me off guard, too. Um, and this is really props to the research team. You know, um, Gabe, in particular, did a ton of work here.
- NTNick Turley
Mm.
- MCMark Chen
Um, Kenji, many others on the team-
- NTNick Turley
So amazing
- MCMark Chen
... did phenomenal work. And, um, I think it really spoke to this thesis, that when you get a model which is good enough that in one shot it can generate an image that fits your prompt, that's gonna create immense value.
- AMAndrew Mayne
Mm.
- MCMark Chen
And I think we never quite had that before, right? Um, that you just get the perfect generation, oftentimes on the first, first try. Um, and I think that's something very powerful. You know, like, uh, people don't want to pick the best out of a grid. I think, uh, yeah, you just got very good prompt following and, you know, just great style transfer, too, right?
- AMAndrew Mayne
Yeah.
- MCMark Chen
Um, this ability to kind of put images, um, as context for the models and to modify and to change, and the fidelity that you could do that with, um, I think that was really powerful for people.
- NTNick Turley
I think, I think this ImageGen experience, um, it was just kind of another mini-ChatGPT moment-
- MCMark Chen
Mm-hmm
- NTNick Turley
... um, all over again, where-
- MCMark Chen
Mm
- NTNick Turley
... you know, you have kind of this, you've been staring at this for a while, you're like, "Yeah, it's gonna be cool. I think people are really gonna like it." Um, but you kind of, you know, you're launching, like, 20 different things, and then suddenly the world is going crazy in a way that you, you kind of only find out, um, um, by shipping. Like, I remember distinctly, you know, we had, like, t-... 5% of the Indian internet population tried- [laughs] - um, ImageGen over the weekend. And I was like, "Oh, wow, we're reaching new types of users-
- AMAndrew Mayne
Mm
- NTNick Turley
-who we wouldn't even have thought, you know, who, who might not have thought of using ChatGPT. That's really cool. And, and, um, to Mark's point, I, I think a lot of this is, um, because there's just this discontinuity where something suddenly works so well, and truly the way you expected, um, where I think it, it blows people's minds, you know? And I think we're gonna have those moments in other modalities, too. You know, I think voice-
- AMAndrew Mayne
Mm.
- NTNick Turley
-you know, it, it hasn't quite passed the Turing test yet-
- AMAndrew Mayne
Mm
- NTNick Turley
... but I think the minute it does, people are gonna, um, I think, find that immensely powerful and valuable. You know, video is gonna have its own moment, where it-
- AMAndrew Mayne
Mm
- NTNick Turley
... starts meeting the expectations that users have. So I'm really excited about the future because I think there's so many of these magical moments coming that are really gonna transform people, um, people's lives and, and also, you know, change, um, sort of ChatGPT's relevance for people. Because, um, you know, there's, um... I've always felt like there's text people and there's image people, and, like, some of them are a little bit different. Um, and now they're all using the product and discovering the value, um, across the board.
- AMAndrew Mayne
The moment when it launched, I think it kind of illustrated the, the problem that had been with image models before.
- NTNick Turley
Mm.
- AMAndrew Mayne
And, you know, when DALL-E came out, it was super exciting 'cause you're like, "I'm, I'm, like, doing pictures of space monkeys," and all these sorts of things. The moment you try to do a really complex image, and that's the, the phrase I brought up before-
- 29:00 – 33:10
Cultural shifts in safety and the freedom to explore
- AMAndrew Mayne
the technological ability to control for things, and how much of that was just saying, "We've got to push the norms?"
- NTNick Turley
I would say it was both cultural shift and an improvement-
- AMAndrew Mayne
Mm
- NTNick Turley
... in our ability to control things. The culture shift, you know, I'm, I'm, I'm not gonna deny it. I think when I joined OpenAI, there was, um, a lot of conservatism, um, around, you know, what capabilities we should give to users-
- AMAndrew Mayne
Mm
- NTNick Turley
... maybe for good reason.
- AMAndrew Mayne
Yeah.
- NTNick Turley
The technology is really new. Um, a lot of us were new to working on it, and, you know, if you're gonna have a bias, you know, biasing towards safety and being careful, it's not a bad, you know, in, in DNA to have. But I think over time, we, we learned that there's so many positive use cases that you, um, e- effectively prevent when you make arbitrary restrictions of the model.
- AMAndrew Mayne
What about faces? Why not? Why can't I make any face I want?
- NTNick Turley
Um, so this is a good example of, of, um, a, you know, capability that, that's got pros and cons, and you can err on one side or the other. But, you know, um, when we, um, first shipped, um, image uploads-
- AMAndrew Mayne
Mm
- NTNick Turley
... um, into ChatGPT, uh, we had some debates, you know, about what, what capabilities do you allow versus where are you conservative? And I think one debate do we had is, like, do we upload- uh, allow the upload of images with faces? Or rather, when you upload an image that contains a face, do you, um, you know... Um, should we just, like, gray out the face? Because you avoid so many problems, right?
- AMAndrew Mayne
Yeah.
- NTNick Turley
You can make inferences about people based on, on their face. Um, you could say mean things to people based on, uh, um, their face. Um, um, and, and, you know, you would just take a giant shortcut on all the gnarly issues if you didn't allow that. But, um, I've always felt we need to-
- NTNick Turley
... um, err on the side of freedom, and we need to do the hard work. And I think in this case, you know, there's so many valid ways. You know, if I want feedback on makeup or on my haircut or anything like that, um, I wanna be able to talk to ChatGPT about it. That- those are valuable and benign use cases, and I would prefer to allow and then study, you know-
- AMAndrew Mayne
Mm-hmm
- NTNick Turley
... where does that, um, um, fall short? Where is that harmful? And then iterate from there versus taking a default stance on disallow. And I think that's one of those ways in which our stance and posture has changed a bit over time in terms of where we set, you know, where we start.
- AMAndrew Mayne
Yeah, we- we're very good, I think, imagining worst-case scenarios. What if I use this, these faces to evaluate hires for a company or whatever? But also it's like, "Hey, is this eczema?" [chuckles] You know, like, you know, there's a lot of utility there.
- NTNick Turley
And, and honestly, I think there are certain domains of, of AI safety where worst-case scenario thinking is very appropriate.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
So I think that is an important way of thinking about risk-
- AMAndrew Mayne
Mm-hmm
- NTNick Turley
... when it comes to certain forms of risks that are existential or even just very, very bad. You know, uh, we have the preparedness framework, which helps us reason through some of those things. You know, um, can the AI let you make a, a bioweapon? It's good to think about the worst case there-
- AMAndrew Mayne
Mm-hmm
- NTNick Turley
... because it could be really, really bad. So you kind of have to have that way-
- AMAndrew Mayne
Mm
- NTNick Turley
... of thinking in the company, and you have to have certain topics where you think about, um, safety in that way. But you can't let that kind of thinking spill over onto other domains of safety, where the stakes are lower, because you end up, I think, making very, very, um, um, um, conservative decisions that, that block out many valuable use cases. So I think being sort of principled about different types of safety on different time horizons and with different levels of stakes is very important for us.
- AMAndrew Mayne
I think I want a blunt mode sometimes, and just... 'Cause, like, right now-
- NTNick Turley
Where it actually roasts you?
- AMAndrew Mayne
Well, I mean, like, yeah, 'cause I'll ask the model, like, 'cause with the, the, the voice-in, speech-out model, be like: "Do I sound tired?" And it's like, "Well, you know, I don't really wanna, you know..." And I'll be like, "Yeah, you know, just you're trying to get it to be honest."
- 33:10 – 37:45
Code, Codex, and the rise of agentic programming
- AMAndrew Mayne
Codex is somehow back [chuckles] and, you know, a new, new form, uh, same name, but the capabilities keep increasing. And we've seen Code work its way first into, uh, VS Code via Copilot and then, uh, Cursor, and then, like, Windsurf, which I use all the time now. What uh- how much pressure has there been in the code space? Because I'd say that if we asked people, "Who made the top code model?" We might get different answers.
- MCMark Chen
Yeah, and I think it reflects that when people talk about coding, uh, they're talking about a lot of different things-
- AMAndrew Mayne
Mm
- MCMark Chen
... right? I think there's coding in a specific paradigm. Like, if you pull up an IDE, and you wanna kind of get a completion on a, on a function, that's very different from, you know, agentic-style coding-
- AMAndrew Mayne
Mm
- MCMark Chen
... where, you know, you ask, uh, you know, "I want, I want this PR," and, you know, um, and I think we've done a lot of focus-
- AMAndrew Mayne
Could you- uh, sorry, could you, uh, unpack a little bit what you mean by agentic coding?
- MCMark Chen
Yeah, yeah. So I think, um, when you... You draw a distinction between more kind of real-time response models.
- AMAndrew Mayne
Mm.
- MCMark Chen
Um, you can think of ChatGPT, uh, to first order, uh, a- as you ask a, um, a, a prompt, and then you get a response fairly-
- AMAndrew Mayne
Mm
- MCMark Chen
... fairly quickly, and then a more agentic-style model-
- AMAndrew Mayne
Mm
- MCMark Chen
... where you give it a fairly complicated task.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
You let it work in the background, and after some amount of time, it comes back to you with what it thinks is something close to the best answer.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
Right? And I think we see increasingly that the future will look like more of a async kind of, uh, you know, where you're asking it very difficult, hard things.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
And, um, you're letting the model think and reason and come back to you with really the best version of what, what it can come back with. And we see the evolution of code in that way, too. I think eventually we do see a world where you'll kind of give a very high-level description of what you want-
- AMAndrew Mayne
Mm-hmm
- MCMark Chen
... and the model will take time, and, um, it'll come back to you. And so I think, uh, our, our first launch, Codex, really, um, reflects that kind of paradigm, where, uh, we are giving it PRs, units of fairly heavy work-
- AMAndrew Mayne
Mm-hmm
- MCMark Chen
... um, that encapsulate, you know, a, a new feature or, you know, a big bug fix, and we want the model to spend a lot of time thinking about how to accomplish this thing, rather than kind of give you a fast response.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
And to your question, you know, there, there's, there's, there's... Coding is such a giant space.
- AMAndrew Mayne
Mm.
- NTNick Turley
There's so many different angles at it. Kind of like talking about knowledge work or something-
- AMAndrew Mayne
Mm
- NTNick Turley
... incredibly broad-
- 37:45 – 41:45
Coding with taste
- AMAndrew Mayne
how are we going to adapt to all that?
- MCMark Chen
Right. Yeah, I mean, specifically in code, right, I think there's more beyond, did it get you the right answer? With code, you know, people care about the style of the code.
- AMAndrew Mayne
Mm.
- MCMark Chen
They care about, you know, how verbose it was in the comments. It cares about, um, you know, how much proactive work did the model do for you, right, um, on other functions? And so, I think, you know, there's a lot to get right, and users often have very different preferences here.
- NTNick Turley
Yeah, it's funny, I used to, I used to- you know, people used to ask me, "Well, what domains are gonna, like, you know, be transformed by AI?" You know, fastest, and I used to say, "Yeah, it's code," because, like, similar to math and other things, it's very, very verifiable and testable, and I think those are the domains that are particularly great to do-
- MCMark Chen
Mm
- NTNick Turley
... RL on, and, you know, you're therefore gonna see all this, this awesome, you know, agentic stuff just suddenly work. I still think that's true, but the thing that surprised me about code is that, you know, there is still so much of an element of taste in terms of-
- MCMark Chen
Mm
- NTNick Turley
... what makes good code.
- MCMark Chen
Mm.
- NTNick Turley
And there's, you know, there's a reason that, you know, people train to be a professional software engineer. It's not because their IQ gets better, because they- but rather because they learn, you know, um, how, how to build software inside an organization. What does it mean to write good tests? What does it mean to write good n- um, documentation? How do you respond when someone disagrees with your code?
- MCMark Chen
Mm-hmm.
- AMAndrew Mayne
Mm.
- NTNick Turley
Those are all actual elements of being a real software engineer that we're gonna have to teach these models, um, to do. Uh, so I, I expect progress to be fast, and I still think code-
- MCMark Chen
Mm
- NTNick Turley
... has a ton of nice properties that make it very ripe, um, for agentic, um, products. But, but, but I, I do think it's very interesting to the degree that, you know, the, the, the, the element of taste and style and, um, real-world, um, um, software engineering matters.
- AMAndrew Mayne
It's interesting, too, because y- with ChatGPT and, and the other models, you're kind of dealing with having to bridge the divide between consumer and pro.
- NTNick Turley
Mm.
- AMAndrew Mayne
I open up ChatGPT, and I tell my friends, like, "Oh, yeah," 'cause I'll plug it into whatever code model I'm working, 'cause I can actually connect it to there. And I think about, you know, well, that's a very different use case to a lot of other people. Although I've shown people, like, how to go in and use, you know, uh, an IDE, and actually have it just write documents [chuckles] for you and create folders and stuff, which people don't realize, like, yeah, you can do that. You can have ChatGPT actually control it and do that, which is cool, but then you think about, like, okay, we've got a tab now for images. There's the Codex tab, so if I want to connect to GitHub-
- NTNick Turley
Mm
- AMAndrew Mayne
... and have it work through there.
- NTNick Turley
Mm-hmm.
- AMAndrew Mayne
And there's, uh, Sora-
- NTNick Turley
Mm-hmm
- AMAndrew Mayne
... into there. So it's kind of interesting to see how all of these things are coalescing into there. How do you differentiate between a consumer feature, a professional feature, and maybe, like, an enterprise feature?
- NTNick Turley
Look, um, we build very general purpose technology, and it's gonna be used by a whole range of folks. And u- unlike m- many companies, which have this kind of founding user type, and then-
- AMAndrew Mayne
Mm
- NTNick Turley
... they use technology to solve that user's problems, we do start often with the technology, observe who finds value in it, and then iterate for them. Now, with Codex, um, our goal was very much to build for, for professional software engineers, knowing, though, that there's sort of a splash zone, where I think a lot of other people will find value in it, and we'll try to make it accessible for those people as well. Um, there are a lot of opportunities to target non-engineers. I'm personally really motivated to create a world where, you know, or help, help build a world where, um, anyone can make software. Codex is not that product, but you could imagine those products existing over, over, over time. Um, but, you know, as a general principle, it's really hard to predict exactly who the target user is, um, until we made some of these general-purpose technologies, um, available, because it gets back to the empiricism I was talking about. Um, uh, we just never exactly know, um, where the value is gonna lie.
- MCMark Chen
Yeah, and I think even, um, to, to dig deeper into that assuming, like, you know, you could have a person who's mostly using ChatGPT for coding, right?
- NTNick Turley
Mm.
- 41:45 – 43:40
Internal adoption of Codex
- AMAndrew Mayne
me, there are some tools you see that there's a lot of excitement about because there's a lot of internal demand for that.
- NTNick Turley
Mm.
- AMAndrew Mayne
How much are you using it internally, or tools like that?
- MCMark Chen
More and more.
- AMAndrew Mayne
[chuckles] Okay.
- NTNick Turley
I've, I've been really excited, um, to, to see the internal adoption. It's, it's everything from, you know, exactly what you'd expect, you know, people using, um, Codex to offload their tasks to... You know, we have a, um, um, analyst, um, workflow that will look at, you know, logging errors and automatically flag them and slack people about it. Um, so there's all these, these ways that... Or I've actually heard, heard some people are using it as a to-do, where, like, future tasks-
- AMAndrew Mayne
Mm
- NTNick Turley
... they're, they're hoping to do, they're starting to fire off Cod- Codex tasks. So this is the perfect type of thing that I think you can, you can dogfood internally. Um, and, and, you know, um, I, I'm very excited about, you know, the leverage that engineers are gonna get out of a tool like this. I think it's gonna allow us to move faster-... um, uh, with, with, with the people we have, and make each engineer that we hire, um, you know, yeah, like 10 times more productive. So, so in some ways, internal, uh, usage is, is a very good predictor of, of where we want to take this.
- MCMark Chen
Yeah. I mean, we don't want to ship something to other people that we don't find value in ourselves.
- NTNick Turley
Mm-hmm.
- MCMark Chen
And I think, you know, leading up to the launch, we-
- AMAndrew Mayne
Laundry Buddy.
- MCMark Chen
Laundry Buddy.
- NTNick Turley
Laundry Buddy is an essential partner.
- MCMark Chen
I think you and me use that today.
- NTNick Turley
Okay, [chuckles] sorry, sorry. [chuckles]
- MCMark Chen
Um, I mean, yeah, we- I mean, we had some power users, though, that, you know, hundreds of PRs a day, um-
- NTNick Turley
Mm
- MCMark Chen
... that they were generating personally, right?
- NTNick Turley
Mm-hmm.
- MCMark Chen
So I think, you know, uh, there are people internally finding a lot of utility from what we're building.
- NTNick Turley
Also, the, uh, if, if you think about internal adoption, it's also a good reality check because, you know, people are busy. You know, adopting new to- tools-
- MCMark Chen
Mm
- NTNick Turley
... tools takes some activation energy. So actually, um, the thing you find when you try to dogfood things internally is, is, is some of the reality component of how long it takes people to actually adjust to a, a new workflow, and it's, it's been, it's been humbling to, to, to watch, right?
- MCMark Chen
Mm-hmm.
- NTNick Turley
So, so I think you learn both about the technology, but you also learn about
- 43:40 – 46:45
Skills that matter: curiosity, agency, adaptability
- NTNick Turley
some of the adoption patterns when you're trying to get a bunch of busy people to change the way they write code.
- AMAndrew Mayne
As you build these tools, internally, people have to learn how to use them and are having to adapt, and there's a lot of question now about kind of what kind of skills do people need in the future.
- NTNick Turley
Mm.
- AMAndrew Mayne
You know, what kind of skills do you look for on your teams?
- NTNick Turley
I've thought about this a lot. Um, hiring is hard, especially if you want to have a small team that is very, very good and humble and able to move fast, et cetera.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
And I think curiosity has been the number one thing that I've, I've looked for, and it's actually my advice to, you know, students when they ask me, "What do I do in this world where everything's changing?" Because, I mean, for us, there's so much that we don't know. There's a certain amount of humility you have to have about building on this technology, uh, because you don't know what's valuable, you don't know what's risky until you really study and go deep and, and try to understand. And when it comes to working with AI, which, you know, we obviously do a lot, not just in code, but in kind of every facet of, of our work, um, it's asking the right questions that is the bottleneck, not necessarily getting the answer. So I really fundamentally believe that we need to hire people who are deeply curious about, um, the world and what we do. I care a little bit less about their experience in AI. Mark, um, [chuckles] presumably feels a bit different about that one. [chuckles]
- MCMark Chen
[chuckles]
- NTNick Turley
But for the product side, uh, it's been curiosity that I've, I've, um, found the most- the best predictor of success.
- MCMark Chen
No, I mean, even on research, I think increasingly less, uh, we index on, you have to have a PhD in AI, right? I think, uh, this is a field that people can pick up fairly quickly.
- NTNick Turley
Mm-hmm.
- MCMark Chen
I also came into the company as a resident without much formal AI training. And I think correlated to what Nick said, I think one important thing is for our new hires to have agency, right?
- NTNick Turley
Mm.
- MCMark Chen
OpenAI is a place where you're not gonna get so much of a, "Oh, here's... Today, you're gonna do thing one, thing two, thing three." Um, it's really about being kind of driven to find, "Hey, here's the problem. You know, no one else is fixing it. I'm just gonna go dive in and fix it." Um, and also adaptability, right?
- NTNick Turley
Mm-hmm.
- MCMark Chen
It's a very fast-changing environment. That's just the nature of the field right now, and you need to be able to quickly figure out what's important and pivot what you need to do.
- NTNick Turley
The agency thing is re- uh, is real, you know.
- MCMark Chen
Mm.
- NTNick Turley
I think we often get asked for, you know, how, you know, "How does OpenAI, you know, keep shipping, and, you know, you- it feels like you're, you're pushing something out every, every week," or something like that. It's, A, funny, because it never feels to me. I always feel like, you know, we could go... be going even faster. [chuckles]
- MCMark Chen
[chuckles]
- NTNick Turley
Um, uh, but, but, you know, I, I think fundamentally, we just have a lot of pe- people with agency who can ship.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
Um, when that comes to product, that comes to research-
- MCMark Chen
Yeah
- NTNick Turley
... that comes to policy. Shipping can mean different things. Uh, we all do very different things at OpenAI, but I think the ratio of people who can actually do things, um, and, you know, the lack of red tape, except where it matters-
- AMAndrew Mayne
Mm-hmm
- NTNick Turley
... you know, there's a couple areas where I think red tape is very, very important. But, you know, um, um, I think, I think that is what makes OpenAI very unique, and it obviously affects the type of people who we want to hire, too.
- AMAndrew Mayne
I was brought into the company- 'cause I was originally given access to GPT-3.
- NTNick Turley
Mm-hmm.
- AMAndrew Mayne
And I just started showing all these use cases
- 46:45 – 51:30
OpenAI’s “Do Things” culture
- AMAndrew Mayne
for it and making videos every week for it.
- NTNick Turley
I remember.
- AMAndrew Mayne
Yeah, and I was annoying people, I'm sure. [chuckles] But I was-
- MCMark Chen
No, it was not. It was really fascinating.
- AMAndrew Mayne
It was exciting.
- MCMark Chen
[chuckles]
- AMAndrew Mayne
It was an exciting time. I, I described it to people like, "They, you know, they... I think they built a UFO, and I get to play with it," you know?
- NTNick Turley
Yeah.
- AMAndrew Mayne
And then I make it hover, and like: "Oh, you made it hover!" I'm like, "Well, they built it."
- NTNick Turley
[chuckles]
- AMAndrew Mayne
"I just pressed the button-
- MCMark Chen
[chuckles]
- AMAndrew Mayne
... and got it to do that." But that was just what I found very empowering, was the fact that I, I- I'm self-taught. I learned to code by Udemy courses and stuff.
- NTNick Turley
Mm-hmm.
- AMAndrew Mayne
And then to be a member of the engineering staff and be told, "Just go, just go do stuff."
- NTNick Turley
Mm-hmm.
- AMAndrew Mayne
Um, nothing too critical. I didn't break anything or anybody. Um, and that's good to know that that kind of spirit is still there, and I think that is part of the reason why OpenAI is able to ship, even though, you know, it was like 150, 200 people worked on GPT-4.
- NTNick Turley
Mm-hmm.
- AMAndrew Mayne
I think people forget about that, you know?
- NTNick Turley
Totally. And, and honestly, this is how, and e- even ChatGPT, this is how, how, how it came together. You know, we, we had a research team. They'd been working, you know, um, for, for a while on instruction following, and then the successor to that, and, you know, uh, post-training these models, um, to be good at chat. Uh, but the product effort came together as a, as a hackathon. I remember distinctly we said, like, "Who, who, who, who's excited to, you know, go build consumer products?"
- AMAndrew Mayne
Mm.
- NTNick Turley
And we had all these different people. Like, we had a guy from the supercomputing team, uh, who, you know, was like, "I'll make an iOS app. I've done that-
- AMAndrew Mayne
Mm
- NTNick Turley
... in a past life." Or we had, you know, a researcher who wrote some backend code, and it was this convergence of people who were excited to do stuff, and I think the ability to do so, and I think that's how you get the next ChatGPT is, is, is running an organization where, where, where that is possible and continues to be possible as you scale.
- AMAndrew Mayne
Hackathons were my favorite thing, 'cause, one, being a performer and loving show and tell, but it was just neat to be able to see things that you knew were gonna be a product or something later on. 'Cause when you're playing with a technology this advanced and all that. Do you guys still do them?
- MCMark Chen
Yeah, absolutely.
- AMAndrew Mayne
Okay.
- MCMark Chen
Yeah. Um, we've had some fairly recently, and they are typically tied-
- NTNick Turley
Last week, actually.
- MCMark Chen
Yeah, I know. [chuckles]
- 51:30 – 55:15
Adapting to an AI future
- AMAndrew Mayne
of an optimist because I see all these opportunities or places to go in there. What advice do you give people, you know, where- whatever point they are in life, about preparing for or adapting to or being part of the future?
- NTNick Turley
You know-
- AMAndrew Mayne
I like how Mark just looked right to Nick and said, "You take this." [laughing]
- MCMark Chen
[laughing] Oh, no. I can go. Okay, I will jump in right now.
- AMAndrew Mayne
Oh, yeah, right. [laughing]
- MCMark Chen
Yeah, no, I think the-
- AMAndrew Mayne
Go, go
- MCMark Chen
... important thing is you have to really lean into using the technology, right?
- AMAndrew Mayne
Yeah.
- MCMark Chen
And you have to see how your own capabilities can be enhanced, how you can be more productive, more effective, by using the technology. I fundamentally do think that the way this is gonna evolve is you will still have your human experts, but what AI helps the most is the people who don't have that capability at a very advanced level, right? So if you imagine, right, like, uh, as these models get much better at healthcare advice, um, they're gonna help people who don't have access to care the most.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
Right? Uh, image generation, right? It's not producing, you know, an alternative for, you know, experts or, you know, professional artists. It's allowing people like me and Nick to create creative expressions, right? Um, and so I think it's kind of rising the tide that allows people to be competent and effective at a lot of things all at once, and I think that's kind of how we're gonna see a lot of these tools bootstrap people.
- NTNick Turley
The world's gonna change a lot, and I think truly everyone has a moment where the AI does something that they considered sacred and human. Um, um-
- AMAndrew Mayne
I know a guy that got vested and, or felt very threatened about his achievements in code and abilities.
- NTNick Turley
Well, that, that happened for me a long time ago. [laughing] Let's talk about someone else in the room.
- MCMark Chen
Oh, yeah. I mean, yeah.
- NTNick Turley
[laughing]
- MCMark Chen
It's definitely better than me at a lot of code problem-solving, for sure.
- AMAndrew Mayne
Yeah.
- NTNick Turley
Right. So I think it's deeply human to, to, to feel some level of-
- AMAndrew Mayne
Mm
- NTNick Turley
... um, awe, respect, uh, and maybe even fear. And I think to Mark's point, be- actually using this thing can demystify it. I think we all grew up or, you know, learned about the word AI, um, in a world where AI meant something pretty different from what we have today.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
You've got these algorithms that, you know, try to sell you things, try to do things, and... or you've got movies, you know, where the AI takes over, et cetera. You know, like, that term means so many things to different people, that I'm entirely unsurprised that, you know, um, there's fear. So-
- AMAndrew Mayne
Mm
- NTNick Turley
... actually using the thing is, is I think the best way to have a grounded conversation, um, about it. And then I think from there, the best way to prepare, I, I think there's some degree to which you need to understand the products and keep up, sure, but I think things like prompt engineering or sort of understanding the intricacies of this AI, they're kind of not the right direction. I, I, I think sort of there's fundamental human things, like learning how to delegate.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
Um, that is incredibly important because increasingly, you know, you're gonna have an intelligence in your pocket that it can be your tutor, it can be your, um, advisor, it can be your software engineer.
- AMAndrew Mayne
Mm.
- NTNick Turley
Um, it's much more about you understanding yourself and the problems you have and how someone else might help than a specific understanding of AI. Um, so I think that's gonna be important. Curiosity, I mentioned it earlier, I think asking the right questions. You'll get- you only get what you put in, right? Um, that's important. And I think fundamentally being ready to learn new things. I think the more you learn, understand how to pick up new topics, i- and, and, and domains, et cetera, um, the more you're gonna be prepared for a world where, you know, the, the nature of work is shifting much faster than it's ever shifted before. So, um, I'm prepared that my job, you know, and product is, is gonna look different or not exist at all.... but, um, I am looking forward to p- picking up something new, and, and, and I think as long as you, you bring that perspective, um, you're well set up to leverage AI.
- 55:15 – 1:01:00
The opportunities ahead: healthcare, research
- AMAndrew Mayne
for coders or people to create code-
- NTNick Turley
Mm
- AMAndrew Mayne
... however it's done. Um, and you mentioned, like, the health field, and that's one of the things I hear people like, "Oh, when, you know, when we replace everything with AI," like, well, I mean, I would be very happy having an AI diagnose me, operate on me, and probably do everything else, but I do want somebody there to talk me through the procedure and hold my hand. But also, I want people asking questions, like, like, you know, every day I take a bunch of vitamins.
- NTNick Turley
Yeah.
- AMAndrew Mayne
Is this the right time of day to take it? You know, I can't bother my doctor with all these silly little questions.
- NTNick Turley
I, I really don't think you end up displacing doctors. You, and you'd end up displacing not going to the doctor.
- AMAndrew Mayne
Yeah.
- NTNick Turley
You end up democratizing the ability to get a second opinion.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
Uh, very few people have that resource or know to, you know, take advantage of a resource like that. You end up bringing medical care into, uh, pockets of the world, uh, where that is not readily available, and you end up helping doctors gain confidence.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
You know, um, I think I oft- I've often heard from doctors that, you know, they already talk to existing colleagues, uh, to get a second opinion. In some cases, that's not possible, and I think you'd be surprised by the number of doctors that use ChatGPT. Um, now, on things like medicine, there's work to make the model really, really good, and we're excited to do that work.
- AMAndrew Mayne
Mm.
- NTNick Turley
There's also work to prove that the model-
- AMAndrew Mayne
Mm
- NTNick Turley
... is really good because I think you're not gonna trust it until there's some degree of sort of legitimacy.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
And then there's work to explain the areas where the model might not be good because increasingly, once it gets to human and then super le- human level performances, um, it's hard to frame exactly where it will fall short, which is also, um, hard, hard to sort of reckon with. But nonetheless, I think that opportunity is one of the things that gets me up in the morning. Education might be the other one.
- AMAndrew Mayne
[chuckles]
- NTNick Turley
And, um, I think there's a tremendous opportunity to help people.
- AMAndrew Mayne
What do you think is gonna surprise us the most in the next year to 18 months?
- MCMark Chen
I honestly think, um, it's gonna be the amount of research results that are powered, even in some small way-
- AMAndrew Mayne
Mm
- MCMark Chen
... by the models that we've built. And, um, one of the kind of quiet things that's taken the field by storm is the ability of the models to reason.
- AMAndrew Mayne
Mm-hmm.
- MCMark Chen
And you already see some research papers-
- AMAndrew Mayne
Could you... I'm gonna make you explain-
- MCMark Chen
Yeah, yeah
- AMAndrew Mayne
... when you say reason.
- MCMark Chen
Yeah. So this fits into the-
- 1:01:00 – 1:05:40
Async workflows and the superassistant
- AMAndrew Mayne
of view capability and then, uh, UI, was Deep Research. And Deep Research is probably the best example we maybe have of probably agentic sort of-
- MCMark Chen
Mm-hmm
- AMAndrew Mayne
-model use right now, because it used to be you would ask for a model to tell you about a topic.
- MCMark Chen
Mm-hmm.
- AMAndrew Mayne
It would-- you'd either get the data or just do a big search of the Internet, and then it would just summarize all that. Where Deep Research will go find some set of data, look at it, ask a question, then go find some new data, and come back to it-
- MCMark Chen
Mm
- AMAndrew Mayne
... and keep going on. And I think the first time I used it, other people used it, they were like: "Wow, this is taking a while." And then you added a UI change, so I can actually go away and go do something else.
- MCMark Chen
Mm-hmm.
- AMAndrew Mayne
And then the lock screen on my phone will show me this is working, which was a paradigm shift. And I talked to Sam here about that, and Sam said that was a surprise to him-
- MCMark Chen
Mm
- AMAndrew Mayne
... was the fact that people would be willing to wait for answers. And now I've seen, uh, a new metric for models, is how long a model can spend trying to solve a problem-
- MCMark Chen
Mm
- AMAndrew Mayne
... which is a good metric if it ultimately solves it, and that's-- Has this been an update to you in how you think about these things? The idea of, like, "Oh, we don't just want..." And I guess you talked about this before, about agentic, and the idea that it's not just: "Give me the answer." It's like, "Take your time. Get back to me."
- MCMark Chen
Mm-hmm.
- NTNick Turley
I think, you know, to build a superassistant, you've got to relax constraints. Like, today, you have a product that is, you know, en- entirely synchronous, you have to initiate everything. Um, that's just not the maximally best way to help people.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
Like, if you think about a real-world, um, intelligence that you might get to work with, um, it has to be able to go off and do things over a long period of time. It has to be able to p- be proactive. Um, so I think there's like-- we're, we're sort of in this process of relaxing a lot of the, the constraints on the product and on the technology to better mimic a very, very helpful, um, entity.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
Um, the ability to go do five-minute tasks, you know, five-hour tasks, eventually five-day tasks, is, like, a very, very fundamental thing that I think is gonna-
- AMAndrew Mayne
Yeah
- NTNick Turley
... unlock a different degree of value in, in the product. So I've actually not been that surprised that people are willing to do that. Um, like, I, I don't really want to be sitting around, um, waiting for my coworker either. Um, and I think if the value is there, um, I'd, I'd gladly be doing other stuff and, and come back.
- MCMark Chen
Yeah, and we really don't do it just because, right? We do it out of necessity. The model needs that time-
- NTNick Turley
Mm
- MCMark Chen
... to solve the really hard coding problem or the really hard math problem, and it's not gonna do it with less time, right? Uh, you can think about this as, I give you some kind of brain teaser, right? Your quick answer is probably, like, the intuitive wrong one, and you need the actual time to kind of work through other cases to, like, are there any gotchas here? Um, and I think it's that kind of stuff that ultimately makes robust agents.
- NTNick Turley
Mm.
- AMAndrew Mayne
We, we've seen kind of there's, like, the, the paper of the moment where somebody comes out and says: "Ah, I found a, a blocker." And I remember there was one a month or so ago-
- MCMark Chen
Mm
- AMAndrew Mayne
... and they said models couldn't solve certain kinds of problems, and it wasn't hard to figure out a prompt that you could train into a model and it could solve those kinds of problems. And we had a new one that talked about how they would fail at certain kinds of problem-solving ones, and that was kind of quickly, I think, debunked by showing-
- MCMark Chen
Mm
- AMAndrew Mayne
... that, you know, the paper kind of had flaws in there. But there are limitations. There are things that there, there might be some blockers and thing-- or things we don't know are gonna be there. I think brittleness is one of the things. There is a point where models can only spend so much time solving a problem. We're probably at a point where we're only having the model, you know, maybe two systems watch each other, and we have to think about how a third system stops, you know, to wait for things to break down. But do you see kind of any blockers between here and where I'm getting the models that are gonna be solving-- You know, doing things like coming up with interesting scientific discoveries?
- 1:05:40 – 1:07:17
Favorite ChatGPT tips
- NTNick Turley
worthwhile and part of our mission to do this all.
- AMAndrew Mayne
All right, last question, and I'll begin. It's, uh: What's your favorite user tip for ChatGPT?
- MCMark Chen
Mm.
- AMAndrew Mayne
Mine is I take a photograph of a menu, and I'm like: "Yeah, help me plan a meal," or whatever, if I'm trying to like, you know, stick to a diet or whatever.
- NTNick Turley
See, I really want that use case, but, like, I've been trying it for wine lists, and that is my eval on multimodality.
- AMAndrew Mayne
Interesting.
- NTNick Turley
It still doesn't work, like-
- AMAndrew Mayne
Really?
- NTNick Turley
It keeps embarrassing me with, like, hallucinated wine recommendations, and I go order it, and they're like, "Never heard of this" [laughing]
- MCMark Chen
[laughing] Oh.
- NTNick Turley
So I'm glad yours works.
- AMAndrew Mayne
I see.
- NTNick Turley
Uh, but for me, that's the, that's the still use case-
- AMAndrew Mayne
Well, I mean, it c- I-- maybe the wine lens is too dense.
- NTNick Turley
[laughing]
- AMAndrew Mayne
That was a problem, that was a problem with Operator-
- NTNick Turley
Yeah
- AMAndrew Mayne
... was that, like, originally-
- MCMark Chen
Mm
- AMAndrew Mayne
... was that the vision models, the too much dense text, it just loses its placement.
- NTNick Turley
Yeah.
- MCMark Chen
Yeah, I mean, speaking to DeepResearch, I love using DeepResearch. And, you know, when I go meet someone new, um, when I'm gonna talk to someone about AI, right? I just pre-flight topics, right? I, I think the model can do a really good job of contextualizing who I am, who I'm about to meet-
- AMAndrew Mayne
Mm-hmm
- MCMark Chen
... and what things we might find interesting. Um, and I think it, it really just helps with that whole process.
- AMAndrew Mayne
Very cool.
- NTNick Turley
Yeah. I'm a voice believer.
- AMAndrew Mayne
Mm.
- NTNick Turley
I, I, it's still got... Um, I, I don't think it's entirely mainstream yet because it's got, it's got many little kinks that all add up.
- AMAndrew Mayne
Mm-hmm.
- NTNick Turley
But, for me, you know, half of the value of voice is actually just having someone to talk to and forcing yourself to articulate, um, um, yourself. And I, I find that to sometimes be very difficult to do in writing. So, on my way to work, I'll use it to process my own thoughts. And, like, with some luck, and I think this works most days, I'll have sort of a structured list of to-dos by the time I actually get there, so...
Episode duration: 1:07:17
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode atXyXP3yYZ4
Get more out of YouTube videos.
High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.
Add to Chrome