GPT 4.1: The New OpenAI Workhorse

1. 0:00 – 0:57

   Introduction and Guest Welcome

   1. AFAlessio Fanelli0:00

      [on-hold music] Hey, everyone. Welcome to the Latent Space Podcast. This is Alessio, partner and CTO at Decibel, and I'm joined by my co-host swyx, founder of Smol.ai.

   2. SWswyx0:16

      Hey, and today we have a returning guest as well as some new friends. Uh, welcome Michelle and Josh.

   3. MPMichelle Pokrass0:22

      Hello.

   4. JMJosh McGrath0:22

      Hi.

   5. MPMichelle Pokrass0:22

      What's going on?

   6. SWswyx0:22

      [chuckles] Uh, both of you work on the, um... I, I guess, um, Michelle, I, I think I used to introduce you as a manager on the API team. Uh, you s- it seems like you've changed your, your role since we last talked on the, on the podcast.

   7. MPMichelle Pokrass0:35

      Yeah. Uh, now I lead a team [clears throat] on the research tr- side, specifically in post-training.

   8. SWswyx0:41

      Yeah. And Josh, you are also in post-training.

   9. JMJosh McGrath0:43

      Yep. I'm a researcher on Michelle's team.

   10. SWswyx0:46

       Yeah. And I just, uh, found an interesting commonality you guys have. You're, you're also both from Waterloo, uh, continuing the tradition of extremely cracked engineers.

   11. MPMichelle Pokrass0:54

       Oh, yeah. We talked-

   12. JMJosh McGrath0:54

       Yeah

   13. MPMichelle Pokrass0:54

       ... about that last time. That's right.

   14. SWswyx0:56

       [chuckles]

2. 0:57 – 1:54

   GPT 4.1 Launch Overview

   1. SWswyx0:58

      Uh, okay. So we're, we're gathering to talk about GPT 4.1. Uh, I, um, you, uh, launched it. I mean, we, we, we got a little preview, and it was a little bit rumored, right? It, it was r- pre-released, I guess, with OpenRouter as Quasar Alpha, and then there was also an Optimus version. Um, and, uh, I think, like, people were trying to figure out, like, why are we going back from 4.5 to 4.1? You know, there's a whole bunch of other things, but, like, what are the headline, um, facts, I guess, you guys want to emphasize about 4.1?

   2. MPMichelle Pokrass1:27

      Mm, yeah, I'll just say we released three new models today: GPT 4.1, GPT 4.1 Mini, and GPT 4.1 Nano. And the real focus on these were just making dev- models that were great for developers. Um, so we improved instruction following, coding, and shipped our first one million context models.

   3. SWswyx1:47

      Josh, anything to add? I, I don't know if there's anything else that people should really, uh, that, that are, like, is sort of in the fine print. [chuckles]

3. 1:54 – 2:53

   Developer Feedback and Model Names

   1. JMJosh McGrath1:54

      No, I think the, uh, only thing that I would touch on maybe twice is that there's actually a, a new model in the lineup, Nano, which is, uh, even faster for developers that are making, you know, low latency applications.

   2. AFAlessio Fanelli2:08

      And cheaper. Um, what's the... any fun story behind the code names? Or, you know, I got the strawberry hat as another fun time in the lore of OpenAI.

   3. MPMichelle Pokrass2:19

      Yeah. Um, yeah, we really wanted to get as much developer feedback as possible on this model to make sure it worked well in the real world. [clears throat] And so we tested it kind of through OpenRouter. Um, and it was super cool to see people latch onto the names and, and get the theories going. But the feedback we got from there was super helpful.

   4. SWswyx2:40

      Yeah, yeah. It's not even, it's not even, like, the name. It's more about just, like, the API shape. Um, once we saw, like, Chat Kumple, it was, like, very obviously OpenAI. [laughs]

   5. MPMichelle Pokrass2:49

      [laughs] Yeah. It's a, it's a good note. [chuckles]

4. 2:53 – 3:49

   Model Naming and Starry Themes

   1. SWswyx2:53

      Yeah. But like, uh, I mean, okay, is, is there, like, an emphasis on stars? Like, uh, what, what inference were we supposed to draw from, you know, quote-unquote supermassive black holes? [chuckles]

   2. JMJosh McGrath3:03

      I don't think there's anything really to draw from there. I think it was just-

   3. SWswyx3:06

      Okay. They're just cool.

   4. MPMichelle Pokrass3:07

      Just-

   5. SWswyx3:07

      They're just cool

   6. MPMichelle Pokrass3:07

      ... fun little code names. You know, they make you think of, of cool concepts.

   7. AFAlessio Fanelli3:12

      The vibes are good. The vibes are good.

   8. MPMichelle Pokrass3:13

      Things, you know? Yeah.

   9. JMJosh McGrath3:14

      Yeah.

   10. SWswyx3:15

       The other thing about the examples, uh, we j- we're just mining for lore here, right? Um-

   11. MPMichelle Pokrass3:19

       Yeah. [chuckles]

   12. SWswyx3:19

       ... tapirs, um, uh, they're a interesting animal. It comes up a few times on the live stream and on the blog posts. What's up with tapirs? Who, who likes tapirs here?

   13. MPMichelle Pokrass3:28

       Yeah. Our, our team is just a super big fan of tapirs, so-

   14. SWswyx3:32

       Okay

   15. MPMichelle Pokrass3:32

       ... they just happen to work their way into a lot of our content.

   16. SWswyx3:35

       [laughs] Okay, cool.

   17. AFAlessio Fanelli3:39

       Awesome. Um-

   18. SWswyx3:41

       Um-

   19. AFAlessio Fanelli3:41

       The, the-

   20. SWswyx3:42

       Yeah, go ahead

   21. AFAlessio Fanelli3:42

       ... yeah, go ahead. I, I, I think, like, the first thing that, yeah, we just wanna run through is obviously the 4.1 to

5. 3:49 – 4:47

   Confusion Over GPT 4.1 vs 4.5

   1. AFAlessio Fanelli3:49

      4.5. Uh, I, I think that's the first thing that everybody was maybe confused about. So, uh, and I know you're deprecating 4.5. Um, sounds like 4.1 is just, like, a kickass model, and the 4.5 size maybe is not as good of a fit. That was just a research preview. So, um, yeah, I, I don't know. Whatever you wanna, um, say to, to address that, I think it's something we've seen come up also in the Discord.

   2. MPMichelle Pokrass4:12

      Yeah, totally. [clears throat] Okay, naming is really hard, and we've tried to make this as less confusing as we can, but you know, nothing's perfect. Basically, the way we got here is that GPT 4.1 is, like, a pretty big improvement over the 4o line, and we really wanted to signify that. However, it's a model that's, like, much smaller and cheaper than GPT 4.5, and as a result, you know, doesn't achieve the same, like, AIME or other intelligence evals. So it doesn't beat 4.5 on all of the evals, and so we didn't think it made sense to increment beyond 4.5.

6. 4:47 – 5:45

   Distillation and Model Improvements

   1. MPMichelle Pokrass4:47

      Um, but we do think for most developers, they can kind of replace a lot of their 4.5 usage with 4.1.

   2. AFAlessio Fanelli4:55

      And then the mini is strictly better than 4o-mini.

   3. MPMichelle Pokrass4:59

      Yeah.

   4. JMJosh McGrath4:59

      Yeah.

   5. SWswyx5:01

      With the, with the nano. Like, um, but, like, we don't know if 4.1 is a distillation of 4.5 or there's, there's no relationship there. Like, what, what can we say about, like, the shared lineage?

   6. MPMichelle Pokrass5:13

      Um, yeah. What I'll say there is, uh, we're always using various research techniques to improve our models, and distillation is something we talked about before. It's, it's really meaningful, especially for the small models. Um, and we've kind of pulled out some of the things that made 4.5 really good, like it has a lot of the instruction following greatness-

   7. SWswyx5:35

      Yeah

   8. MPMichelle Pokrass5:35

      ... and also rolled that into 4.1.

   9. SWswyx5:38

      Awesome. Um, I think one of the... 'Cause I re- strongly remember on the 4o launch that their communication

7. 5:45 – 6:43

   Omnimodel Architecture and Future Plans

   1. SWswyx5:45

      was that we're kind of moving to a new model architecture that is omnimodel, right? That was what... That's the O in, uh, 4o. And then 4.1 is part of this subsequent trend of trying to merge everything, like the reasoning model, the omnimodel, everything. Um, and I think that there's, there's this doubt about whether 4.1... I, I think it's basically trying to be sold as a strict replacement for 4o But I don't know, i- is it going to be fully omnimodel? Is it, is it like roughly the same architecture that we, that we, that we think 4o has?

   2. MPMichelle Pokrass6:18

      So we, we already have different slugs on the real-time API and like-

   3. SWswyx6:21

      Yeah

   4. MPMichelle Pokrass6:22

      ... the responses API. So they're already, you know, somewhat different checkpoints. Um, I don't think-- we don't have any current plans to release 4.1 in the real-time API, but-

   5. SWswyx6:32

      Okay

   6. MPMichelle Pokrass6:32

      ... you know, things, things may change.

   7. SWswyx6:35

      Yeah, and then there's Image Gen and all that, right? Like, so-

   8. MPMichelle Pokrass6:37

      Right

   9. SWswyx6:37

      ... and as far as we know, no plans, maybe, but nothing announced.

   10. MPMichelle Pokrass6:42

       Not, not right now.

8. 6:43 – 7:40

   Core Capabilities of GPT 4.1

   1. MPMichelle Pokrass6:43

      The focus for 4.1 was kind of these three core capabilities for developers.

   2. SWswyx6:47

      Yeah. We, uh, uh, our Discord actually also did a launch, uh, watch party for the recent, uh, 4.5 podcast that Sam Altman did, where I think for the first time, it was basically kind of confirmed that like some-something that people already knew, like Andrej Karpathy was already talking about this, that 4.5 was like 10x the size of 4. And I think there's a question about like, do we do the linear, linear interpolation of 4.1 is like, you know, zero point [chuckles] is like, uh, I don't know, 2x the size or something?

   3. MPMichelle Pokrass7:16

      Mm-hmm.

   4. JMJosh McGrath7:17

      Uh, that's not really how we think about naming the models. There's a whole-

   5. SWswyx7:20

      [laughs]

   6. JMJosh McGrath7:20

      ... lot of different parts that go into the, the recipe. Uh, and so, you know, it doesn't really reflect on just the pre-training recipe, our version numbering. But I think the 4.1 is just, uh, because of the large jump that we have in like coding capabilities, long contexts, and so on, it's more so what it's like for the end user more so than anything about the training recipe.

9. 7:40 – 8:37

   Training Techniques and Long Context

   1. MPMichelle Pokrass7:40

      We, we can go a little under the hood on training though, and we'll say that, um, you know, Nano is obviously a new pre-train. Um, we also have a new pre-train for Mini, and then, uh, the larger version is, uh, a new mid-train. Um, but we find that actually a significant amount of the gains come from new post-training techniques. Um, [clears throat] so I think in the past the narrative is that you need to pre-train these larger and larger models, um, to get better performance, and we're finding that we're able to squeeze a lot more out of post-training now.

   2. AFAlessio Fanelli8:15

      Um, talking about how big a model is, uh, the other side of it is the context window. You have a 1 million context. I know that Sam, at DevDay last year, he said that, yeah, 1 million was like months away, so, um, right on, right on time. Um, can you talk about, yeah, how hard that was to get to 1 million, and then maybe where the end

10. 8:37 – 9:34

    Challenges in Long Context Reasoning

    1. AFAlessio Fanelli8:37

       game is in your mind? Is it 10 million, 100 million, infinite? Uh, what, what, what really matters as you start to scale this?

    2. JMJosh McGrath8:45

       Yeah.

    3. MPMichelle Pokrass8:45

       Yeah, Josh worked a lot on long context, so he's the right person to ask.

    4. JMJosh McGrath8:49

       Definitely. Um, so I think the first thing that we, that I thought was really interesting when we were going to long context is actually, uh, some of the evals that you see as like headlines on maybe other blogs where it's needle in a haystack. Actually, most of the models do really well pr- uh, right out of the box. But then we had to actually first get a lot of measurement on the longer context for long context reasonings. You know, we actually just open sourced, uh, two new evaluations that are about using the context in a more complex way. So you know, one of them you have to reason a lot about ordering, and the other is actually walking through graphs. Um, so there's a lot of reasoning that you have to do in those data sets, and that's where doing long context is actually much harder. But single needle in a haystack, we were able to saturate f- uh, pretty easily.

11. 9:34 – 10:31

    Context Utilization in Models

    1. JMJosh McGrath9:34

       Um, and then the most of the work came at this, these harder tasks.

    2. SWswyx9:39

       Yeah. I... Oh, go ahead. [laughs] Sorry, Alessio. Uh, do, do, do, you, if you have a follow-up, go, go ahead.

    3. AFAlessio Fanelli9:46

       Yeah. I was gonna say how, you know, just how you think about length of context in terms of consuming documents versus like active kinda like thinking and planning. Uh, I, I think there's obviously a whole part on the prompting side around, uh, building agentic, uh, workflows. Do, do you think that people maybe like still think too much of it about, yeah, needle in a haystack kind of document retrieval versus like traversing very long plans and kind of like iterations in context?

    4. JMJosh McGrath10:13

       Yeah. I think the mental model that I have is maybe actually has some more variables in it. So there's sing- there's the needle in a haystack where you have like some amount of distractors and some, you know, needles that you're trying to find. And I think that it's more so about how dense of the context, uh,

12. 10:31 – 11:31

    Graph Walks and Model Evaluation

    1. JMJosh McGrath10:31

       do you need to use. So like summarization, you're, you're actually just using the entirety of the context, whereas, you know, needle in a haystack, it's very sparse. And then I also generally think about, uh, orderedness. If you're going to make some sort of, uh, inference on this, do you, are you just look- looking, you know, sort of front to back, or do you need to move around in the context in order to generate a good answer while the model's sampling?

    2. SWswyx10:53

       Yeah. Is that something that you worked on with, uh, Graph Walks? Is that, uh, the thinking around that?

    3. JMJosh McGrath10:57

       Yeah. That was sort of the, the most synthetic and clean way to measure the model. And then, you know, we worked on a lot of other training techniques, um, and data to sort of test the model's ability and train in the model's ability to reason throughout the context in a sort of shuffled way.

    4. SWswyx11:13

       Yeah. You know, actually, I have the ability, I, I like to give people a little bit of visual aid with, um, with these things. Uh, so I actually went into your Hugging Face release and got a example of the graph, uh, task. And so there's, there's a few versions of this, right? There's like the BFS and DFS version, and

13. 11:31 – 12:30

    Real Life Applications of Graph Tasks

    1. SWswyx11:31

       also it's, uh, I guess it's very character specific. So I don't know, maybe could you tell us like, like, you know, design choices around this, like what was surprisingly hard, you know, anything like that?

    2. JMJosh McGrath11:42

       Yeah. Um, so the idea here is you, you take a graph and you encode it into the context by looking at the, uh, the edge lists and just putting that into the context, and then asking the model to do an operation. Um, and then, you know, under the hood, we're actually just executing the real operation and using that to then, uh, evaluate the model's ability to work. Um, one of the things that I found surprising at first was the, what the model would do when it wasn't sure how to use its context. You know, early versions of the model just sort of looping saying like, "Oh no, I can't find this edge that I think ne- should be there." Um- And yeah, I think I was actually very surprised how all models seem to have more difficulty than I would have expected on a task that, you know, we would find very simple

14. 12:30 – 13:30

    Multi-Hop Reasoning Benchmarks

    1. JMJosh McGrath12:30

       or like, you know, maybe an un- uh, an undergrad could write a Python script to, to run in a couple of minutes.

    2. SWswyx12:37

       Yeah, right. Um, okay, so like what is the per- like, no, what, what is the real-life task that this is meant to model, I guess? Um, you know, I, I feel like the other one, MRCR-

    3. JMJosh McGrath12:50

       Yeah

    4. SWswyx12:50

       ... seems a little bit more intuitive where, you know, you have like four different stories and you pick out the second one. Uh, and that's, that's a real task that the people have, but people don't really traverse graphs. Like this is a bit more theoretical, but like, you know-

    5. JMJosh McGrath13:02

       Absolutely

    6. SWswyx13:03

       ... was there, was there any sort of correlation study done?

    7. JMJosh McGrath13:05

       Uh, yeah. This is actually meant to be sort of the idealized version of like a multi-hop reasoning benchmark. So we have a lot of things where, you know, you're putting hundreds of documents into, uh, the context and then you might ask a question that you actually have to traverse 10, uh, documents for. But there the edges are... they're implicit, right? Like there is some underlying graph that's connecting all of these documents that you need to traverse

15. 13:30 – 14:28

    Agentic Workflows and Backtracking

    1. JMJosh McGrath13:30

       in order to answer the question. Um, but they're actually much harder to traverse because the edge isn't actually given to you. And so the question there was like, okay, if I actually just give you all of the IDs of these things that you need to traverse, can the model even do that? Um, where it's like a-

    2. SWswyx13:44

       [chuckles]

    3. JMJosh McGrath13:44

       ... it's actually just like a lower bound on, uh, how well the model can do. And I think it's, that's actually somewhat well, well reflected in some of the internal benchmarks we have that are using more natural data.

    4. SWswyx13:55

       Yeah.

    5. MPMichelle Pokrass13:55

       You can imagine something like a tax return, right? Where, where you like upload-

    6. JMJosh McGrath13:59

       Oh

    7. MPMichelle Pokrass13:59

       ... the entire tax code. [clears throat] Like to figure out what to put into this, you know, box, you'll need to reference all of these boxes. And so this is like a similar level of multi-hop reasoning, but again, like Josh said, all of the references are implicit.

    8. SWswyx14:14

       Yeah. I, I think that like some kind of backtracking if it's needed is also super interesting, uh, especially for agent work. Um, I... for listeners who, who've been listening to us for a while, we actually covered this paper at NeurIPS last year, uh, t- two years ago, called Geval,

16. 14:28 – 15:24

    Graph Traversals for Agent Planning

    1. SWswyx14:28

       where they actually modeled graphs for graph traversals for agent planning and, uh, uh, it, it reminds me closely of that. It's just that they never came up with this exact format that you have here, which basically is the same thing. [laughs]

    2. JMJosh McGrath14:41

       Yeah.

    3. SWswyx14:41

       Uh, I also like that you included, uh, blank answers because sometimes, uh, people do hallucinate or models do hallucinate answers, and you have a fair amount of blank ones.

    4. JMJosh McGrath14:50

       Thanks for random sampling over graphs I did, I guess. [laughs]

    5. SWswyx14:55

       Yeah.

    6. AFAlessio Fanelli14:56

       Um, is this tied also to the file search API that you released recently? Like how should p- people think about how everything kind of comes together in the API?

    7. MPMichelle Pokrass15:07

       Yeah, I think, uh, hmm, oftentimes with retrieval you, you know, might be using RAG to fill the context and a lot of this is like to get around the limitation of a short context window. So we do expect a lot of developers to start,

17. 15:24 – 16:21

    Context Usage in API and Memory Systems

    1. MPMichelle Pokrass15:24

       you know, uploading their full context more directly to the model. Um, so for smaller tasks you maybe don't need the whole, uh, vector store. Um, but we do anticipate this to play well with that paradigm as well. Like maybe you can just insert way more chunks into the context. Um, so we, we think it'll play nice.

    2. SWswyx15:43

       Yeah. A- any relationship to the memory upgrades in ChatGPT that we recently got, um, is long context just directly usable for memory or should we just always have a separate memory system?

    3. MPMichelle Pokrass15:56

       Yeah, it's a, it's a good question. So right now the dreaming, uh, feature we kind of have, uh, some of the, these memories embedded in the context. [clears throat] But you know, they are, they are separate features so 4.1 is powering the API whereas, uh, the enhanced memory is, is ChatGPT only.

    4. SWswyx16:17

       Yeah. Awesome. Um, yeah, I think, I think that's interesting.

18. 16:21 – 17:17

    Model Performance in Long Context Tasks

    1. SWswyx16:21

       Uh, I guess the, the one last thing I'll call out on long context, which is kind of, um, un- unintuitive or maybe there's an explanation which was, uh, the, you had, you had two needle for MRCR and then we had f- four and eight and everything kind of just regresses to some kind of baseline of like let's say 30% or 20% as that. But it's interesting to see where the smaller models sometimes match or outperform the larger models. Uh, I was wondering if there's anything unusual there or do you think it was like a bad roll of the dice?

    2. JMJosh McGrath16:53

       I think it's probably just a, a bad roll of the dice. I think I would probably look more so at the, the larger models-

    3. SWswyx16:58

       Narrow ones

    4. JMJosh McGrath16:58

       ... see things regress as you in- uh, increase the number of needles because there's sort of more complex reasoning that it has to do about the order of different things in its context.

    5. SWswyx17:07

       Yeah. Awesome. Yeah, cool. Happy to, happy to move on from there. Um, yeah, we have a whole bunch of other, um, evals that we can go over.

19. 17:17 – 18:12

    Instruction Following and Real World Data

    1. SWswyx17:17

       So, um, I had in my notes that we could talk over, you know, any- anything that you, that you want. There was also like Kali from Shunyu with, who we have, have on the podcast for instruction following. Um, and I, I, I realized that, you know, he joined OpenAI and I wonder if he had a role to play in that one.

    2. MPMichelle Pokrass17:35

       Uh, no, we did not collab a ton on it. Honestly, I think it's best when eval authors and model developers don't collab too much 'cause you-

    3. SWswyx17:44

       Okay

    4. MPMichelle Pokrass17:44

       ... you want to keep things, uh, you know, as objective as possible, not trying to game any evals.

    5. SWswyx17:49

       Yeah. Um, and, and then there, I, I think there was also like for the first time the announcement of the, or shout-out of the internal instruction following benchmark from API data. Um, people have had the ability to opt in to share data for a while. Um, actually I, I like published a, I, I posted a tweet out 'cause I found it in the dashboard that you can just opt in and it, and like there's o- there's basically 16 days left

20. 18:12 – 19:09

    Challenges in Grading Instructions

    1. SWswyx18:12

       for this program where you can just get free inference. Um, and like, so, so I'm just kind of curious like, um, what you found from that kind of IF eval that, that might be different from the normal IF eval that people have.

    2. MPMichelle Pokrass18:24

       Yeah, totally. A lot of the instruction following evals that are open sourced are open sourced, you know, or, or crafted in a way that are easy to craft. So for example, like graph walks is, is somewhat easy to craft. Like you can create this graph and verify it easily, but it is not exactly aligned with what the users are doing. And this is true for some of the instruction following evals, where you ask the model to output exactly four words or, you know, three paragraphs or stuff like that, things that you can verify easily in code. Um, and these are useful instructions, but we find that many of the really interesting instructions are actually challenging to grade, and so the open source evals often don't have them. And so getting this like real world,

21. 19:09 – 20:09

    Instruction Following Techniques

    1. MPMichelle Pokrass19:09

       uh, diverse set of data actually helps us find like what are the commonalities in what developers are doing, what is a really good example of like a negative instruction, and then we can go from there and figure out how to, how to evaluate it.

    2. SWswyx19:24

       Yeah. Uh, I, I think that's a, that's also an interesting question of like what domains do people use you on. And I, I wonder if like there's a way to tell you [laughs] because sometimes it can be very confusing if I... For ex- especially because maybe I'm building an app and letting people use my key, but um, other people are building apps on top of me. So you have just a lot of chaos of like multiple degrees of abstraction-

    3. MPMichelle Pokrass19:51

       Yeah

    4. SWswyx19:51

       ... where you just have to parse through the prompts. [laughs]

    5. MPMichelle Pokrass19:54

       Yeah, it's true. Well, I will say we do use our own products internally where we can, and so, um, you're, we're not manually by hand reading every prompt. After they're like anonymized, we scrub them of any identifying data, then we use our models to take passes to categorize them.

22. 20:09 – 21:05

    Prompting Techniques and Model Responses

    1. MPMichelle Pokrass20:09

       Um, and so if we get feedback that like, uh, we're not doing well on ordered instructions, then we can kind of do a pass over all of our data and find some good examples of those.

    2. SWswyx20:21

       So-

    3. AFAlessio Fanelli20:22

       Um, so there's the instruction following section in this great prompting GPT-4.1 models. I think maybe we can go through some of these examples. Uh, the first one that caught my mind that, um, it's not necessary to use all caps and other incentives like bribes or tips, but developers can experiment with this for extra emphasis. So [laughs] I, I think that second part leaves me confused. Are you saying that people should still try and do this and sometimes the model responds positively to it? Do you feel like it's still just part of the lore? I'm curious why. I, I would have loved for you to say either, "Yes, it works," or like, "No, you should stop. It looks silly." Uh, I guess the truth is somewhere in the middle.

    4. MPMichelle Pokrass21:01

       The truth is always messy. Um, reality is that our models have gotten

23. 21:05 – 22:01

    Agentic Workflows and Model Persistence

    1. MPMichelle Pokrass21:05

       a lot better at following instructions just stated once and clearly. Um, but we find honestly, developers often become the best experts at prompting our models because, you know, you're building your livelihood on this thing and, and get to know the details of it really intimately. So I will say stuff like that won't hurt the performance of the model, but we kinda always wanna leave it open to people to figure out what works best.

    2. AFAlessio Fanelli21:31

       Yep. Um, yeah, and then you had to always start with a response rules or instructions section. Are those keywords meant to be taken kinda like verbatim, like those are kinda like the tokens that work the best, um, or is it just like a, an example?

    3. MPMichelle Pokrass21:47

       More of an example. Yeah.

    4. AFAlessio Fanelli21:49

       Okay, cool. Um, yeah, this is great. I feel like until today we did an episode with like the prompt report on like all these prompting techniques, but then it's also unclear, um, for which model, uh, which ones work best. So, um,

24. 22:01 – 22:56

    Balancing Persistence and User Control

    1. AFAlessio Fanelli22:01

       it's super useful. And then you had a, in the agentic workflows one, you have a persistence thing. It's like-

    2. MPMichelle Pokrass22:09

       Yeah

    3. AFAlessio Fanelli22:09

       ... please keep going. [laughs] How much... And I, and I think I read, uh, that improves like the SWE, the SWE-bench like 20% just by having like the persistence keep going thing.

    4. MPMichelle Pokrass22:20

       But Claude, I, I wouldn't s- it's not that this one prompt improves SWE-bench 20%. It's that we found this is the most effective harness for our model.

    5. AFAlessio Fanelli22:28

       Okay.

    6. MPMichelle Pokrass22:28

       Um, and combined with all the post-training improvements, it results in the big improvement. But yeah, like the model is trying a lot to be helpful, and often it wants to check back in with the user and be like, "You know, should I keep doing this?" Like, "Is, am I on the right track?" And so a prompt like this, uh, makes sure it keeps going, doesn't bother you again, and just gets the task done.

    7. AFAlessio Fanelli22:50

       Yeah.

    8. SWswyx22:50

       Yeah, I think like, um, there's this interesting trade-off between persistence and

25. 22:56 – 23:55

    Evaluations on Model Edits and Persistence

    1. SWswyx22:56

       yielding back to the user. The more agentic a model, uh, wants to be, the, the, the more persistent it should be, but then sometimes it just goes off the rails. And I wonder how you solve this trade-off, because sometimes it just goes too far. Uh, there's been criticisms of Claude Sonnet trying to rewrite too many files at once when I just wanted to make one thing, for example. Uh, and that's a f- that's a form of bad persistence. I, I... What are the axes here in which, like you think about it?

    2. MPMichelle Pokrass23:25

       Yeah. I think one of the interesting thing that comes to mind here is that we had an extraneous edits eval, um, where you ask the model to make an edit and classify like were all of its changes related to what it was asked to do, or did it go off and, and do a little too much. Um, and we found that from 4o, which got 9%, which is pretty crazy, 9% of the time making an extraneous edit is a lot. 4.1 is at 2%, so it's a pretty big improvement. [clears throat]

26. 23:55 – 24:50

    XML vs JSON in Prompting

    1. MPMichelle Pokrass23:55

       So yeah, I, I would just say like focusing on this, we've heard feedback about this. We made an eval and we made sure to track it and improve it during training too.

    2. SWswyx24:04

       Yeah. Yeah. I mean, everything comes down to evals as, as, uh, as is no surprise to anybody.

    3. MPMichelle Pokrass24:09

       That's true.

    4. SWswyx24:10

       Um, there's another interesting eval that I think is causing some noise. Uh, for the first time, I think also that, uh, and you being the master of structured outputs should know, uh, that JSON is bad now and we should all use XML.

    5. MPMichelle Pokrass24:24

       Uh, I wouldn't say that. I don't know which eval you're talking about, but-

    6. SWswyx24:28

       It's in the prompts guide, which maybe you guys didn't write, so [laughs] we're kind of springing this on you.

    7. MPMichelle Pokrass24:32

       Yeah. Um, Noah and Julian on our team wrote the prompt guide and did a great job. Um, I do think XML is very helpful for structuring prompts, whereas for parsing outputs, uh, maybe the story is a bit different. Like, sometimes it's really useful to get outputs in JSON, so you can plug them directly

27. 24:50 – 25:49

    Instruction Placement in Context

    1. MPMichelle Pokrass24:50

       into your application. But I do think the models, uh, work particularly well with XML as inputs. Um, but Chris, anything to add?

    2. JMJosh McGrath25:01

       No, no.

    3. SWswyx25:01

       [laughs] Um, cool. I, I mean, I, I think just people always just care a lot about code tool calls and, uh, structured outputs as, as you well know, and so any updates to instructions over there is good. People also are interested in this concept of, uh, that apparently putting the instructions and user query at the top and the bottom, so duplicating it at the top and the bottom in the context is much better than... Uh, is better than putting it top only, and much better than putting it bottom only. Uh, again, this is from the prompt guide, so I don't know how aware you guys are on this.

    4. JMJosh McGrath25:35

       Yeah. Uh, I think part of that was just like, you know, empirical. We, we tried all three for when we were, uh, evaluating the model, and having that redundancy is definitely the best. Um-

    5. SWswyx25:44

       Yeah

    6. JMJosh McGrath25:44

       ... but then using the, those, the instructions at the beginning, the model's going to be able to

28. 25:49 – 26:49

    Optimizing for Prompt Caching

    1. JMJosh McGrath25:49

       then take that into account as it does processing.

    2. SWswyx25:52

       Yeah. I, I think, like, a lot of people would see this as like running counter to prompt caching, because obviously you want to put the things that change a lot at the bottom. Um, basically, is this fixable in post-training? Like, can we just tell, uh, models to take instructions or user queries only at the bottom because we want to optimize for prompt caching?

    3. MPMichelle Pokrass26:17

       When we figure it out, we will do that. [laughs]

    4. SWswyx26:20

       [laughs] I mean, it seems doable. It seems like a post-training thing. I, I, I don't know. Maybe my mental model of post-training is wrong.

    5. JMJosh McGrath26:27

       So I think actually having things at the, the beginning of the prompt, uh, you would still get prompt caching there. If you're putting in, for example, like a big needle in a haystack, uh, uh, prompt and you have the data changing each time, like be- it's per user, there's still different ways that you can be putting the prompt at the beginning and getting a lot of the, the cache hits. It sort of just depends on your use case.

29. 26:49 – 27:46

    Chain of Thought and Reasoning Models

    1. SWswyx26:49

       Yeah. Awesome.

    2. AFAlessio Fanelli26:51

       Um, the, the other thing I noticed, I know you made a note of this, Sean, too, um, is around chain of thought and reasoning and how people should think about this model versus a reasoning model. Um, yeah, what's your, uh, uh... Yeah. Should I just use 4.1 and prompt it to do a chain of thought? Should I use o1 and make a plan and then use 4.1 to implement the plan? Um, how should people think about composability?

    3. MPMichelle Pokrass27:17

       Yeah, it's a great question. Um, we have found that 4.1 is a lot better at doing planning and thinking through its steps in CoT when prompted than our previous, uh, non-reasoning models. But our reasoning models, uh, are designed to have kind of more coherent plans and be able to reason over longer horizons than these non-reasoning models. And you can see that reflected in things like intelligence bench-benchmarks, so AIME,

30. 27:46 – 28:46

    Choosing the Right Model for Your Task

    1. MPMichelle Pokrass27:46

       GPQA, stuff like that. You'll see the reasoning models do much better. Um, so in general, I would say, like the question you're really getting at is like, "I'm a developer, which model should I be using?" Um, and I think the answer is always going to be the fastest model that accomplishes your task, right? So maybe you start prompting, um, 4.1 as a starting point. Uh, if it does your task super well, then maybe you drop down to 4.1 Mini and save latency, or even Nano. Whereas if 4.1 is struggling a bit little, maybe needs more coherent reasoning over longer time horizons, then maybe you upgrade to a reasoning model.

    2. AFAlessio Fanelli28:24

       Is there a quick way to get through these heuristics? I know one thing that a lot of people do is like they use o1 for like a plan, and then they put that plan in cursor and then have the plan apply to their code base. Um, it sounds like there's maybe not a rule to when to do which, it just like task dependent.

    3. MPMichelle Pokrass28:43

       Yeah. I would say we're all kind of figuring out the best way to

31. 28:46 – 29:41

    Coding Capabilities of GPT 4.1

    1. MPMichelle Pokrass28:46

       use these models together. Um, and so I do think reasoning models for planning, uh, and using kind of more targeted models to execute is definitely a good architecture.

    2. SWswyx28:59

       Um, cool. Uh, if there's nothing else on, on, on that side, uh, I'd love to go into the coding, uh, which is something that we're emphasizing a lot. Um, it's doing super well. It's better than o1 on SWE-bench. Was that expected?

    3. MPMichelle Pokrass29:13

       Uh, not really. [laughs]

    4. SWswyx29:14

       [laughs] Yeah. Like, what's the... I mean, what's the story there? There's also SWELancer, which is a newer one, which, uh, attaches a money value to things and, um, basically, w- like what should people understand is going on here? Like, is it a better coding base model or just a coding agent model? Um, and I think there's also, there's also a question about like, you know, how important to coding is it if I'm not using a coding

32. 29:41 – 30:39

    Model Performance in Coding Tasks

    1. SWswyx29:41

       use case?

    2. MPMichelle Pokrass29:42

       Yeah. Uh, so I'll start by saying we just set out to make a model that was great at coding, both in your terminal or in your editor or wherever you want to use it. Um, and so we kind of broke that down into the problems that it, it, you know, encompasses. So like developers want the model to produce better diffs, for example, or they want the model to explore the code base correctly, uh, or they want to produce code that compiles or produce code that writes tests. And so our approach was kind of teaching the model all of these various facets. Um, yeah, there's kind of just a bunch of work streams that all coalesced around, uh, GPT 4.1.

    3. JMJosh McGrath30:22

       Yeah. I think much improved, uh, post-training all over to, uh, make for a better coding model

    4. SWswyx30:31

       Yeah, I think there's like different kinds of coding, right? Like, um, the... It's, it's interesting for me to observe that there, for example, uh, so

33. 30:39 – 31:36

    Understanding Coding Model Differences

    1. SWswyx30:39

       I-I'm just gonna pull it up on the chart here 'cause I always like to show people visuals. Um, you're at 55 on SWEBench, and o1 gets like a 41, but then on, um, oh, I, I don't think I, I don't think I have the others. Like, but, but Aider, um, is, it is less, uh, uh, it is not at o1 level. And so I, I think I, I think I struggle to get some kind of intuition of when, like, uh, like, h-how, what are the different elements of coding? I guess there is like, you know, single file edits, whether it's like a diff or, um, um, or whole file, and then there is entire project edits. Is that, um, a reasonable split? Are there more to this?

    2. MPMichelle Pokrass31:18

       Yeah, that's one way to think about it. Basically, where GPT 4.1 can kinda explore and go through a repo. Um-

    3. SWswyx31:26

       Yeah

    4. MPMichelle Pokrass31:26

       ... it's been trained to do that particularly well. Whereas, uh, you know, to just get some code and produce a, a change, um,

34. 31:36 – 32:33

    Using Smaller Models for Coding

    1. MPMichelle Pokrass31:36

       a reasoning model might do better because it, it can kind of reason over the entire file. Um, and so that's one good way to think about it.

    2. SWswyx31:44

       Yeah. Yeah, that's fair. Um, any, any in, um, understanding of like the smaller ones, the smaller, uh, models? Like basically for coding, I should only use 4.1 and forget the rest? [laughs]

    3. MPMichelle Pokrass31:55

       So you, you might like want to use, um, the smaller models, maybe if you have, like, if you have an IDE where you need an autocomplete feature, for example. Um, or if you want something super fast, uh, if you're building like, I don't know, a text-to-SQL thing, you might want the first version to populate instantly. Um, so you can see like 4.1 Mini is actually quite significantly better than 4o-mini, um, and not that far away from the old 4o. So I, I do think that model will find use case in, in a bunch of these coding niches.

    4. SWswyx32:30

       Um, and I know you might not be able to talk about this, but,

35. 32:33 – 33:28

    Future of Coding in OpenAI

    1. SWswyx32:33

       um, the clip of the, of an AI CFO talking about the agentic suite has been going viral, I think today. Um, it, it seems like every lab is putting a lot of emphasis into coding, so, um, yeah, I'm just curious if there's anything you can share about, um, how people should think about OpenAI in coding, you know? Obviously today you don't have, you know, Clodius, CloudCode, um, you don't have any, anything related to coding and, uh, I think the Windsurf partnership today, they, they're giving 41 for free-- 4.1 for free for a couple weeks. It's maybe like one of the first, um, you know, OpenAI endorsement, I guess, uh, on the live stream. But yeah, just, I know there might not be an answer that the PR team might approve, but-

    2. JMJosh McGrath33:16

       That's good

    3. SWswyx33:16

       ... uh, I, I'm curious if you have any takes and thoughts.

    4. JMJosh McGrath33:19

       I think just stay tuned. Um-

    5. MPMichelle Pokrass33:22

       Yeah, we, we... I think coding is an important use case for our users, and so that's why we

36. 33:28 – 34:26

    Internal Use and Success Stories

    1. MPMichelle Pokrass33:28

       focused it, on it a lot for 4.1. Um, we also love to use our own products internally, uh, and so making 4.1 selfishly helps us move faster as a company. And so that's where the real focus has been, uh, for this model.

    2. SWswyx33:43

       Mm-hmm. Do, do you track what percentage of code is, is written by 4.1 internally now?

    3. MPMichelle Pokrass33:49

       We do have some metrics like that. I don't have it off the top, but, uh, I was actually just talking to one of the researchers on the team who worked on something over the weekend, and he said, uh, that, uh, this model, GPT 4.1, was able to like get 4,950 of his commits on this massive PR, uh, done. So we were pretty happy to hear that.

    4. SWswyx34:10

       I'm excited to use it. Um, awesome. Yeah, I think on the, the, yeah, I, I think coding is a super exciting use case, and I think like OpenAI has always been very developer first, um, as you've been too, Michelle, so, um, it's great to see the, the convergence.

37. 34:26 – 35:25

    Vision and Multi-Modal Capabilities

    1. JMJosh McGrath34:26

       Yeah.

    2. SWswyx34:27

       Um, the other, I think the last capability that I kind of vectored in on was vision or just multi-modality in general. Uh, it is a lot better. Um, basically, I, I, I think like, uh, I, I really like these niche benchmarks like MathVista and ChartCive. Um, yeah, just any, any, any extra color on, on like the vision side that you wanted to talk about but you, maybe you couldn't fit into the blog post?

    3. JMJosh McGrath34:51

       Yeah.

    4. MPMichelle Pokrass34:52

       Yeah. Go ahead.

    5. JMJosh McGrath34:53

       Oh, I was gonna say, I think one maybe small nugget there is actually, I think the, the 4.1 Mini is really exciting on that front. Um, as we were talking about, it's a different pre-training base, and I think that really shows up in some of the vision evals.

    6. MPMichelle Pokrass35:07

       And yeah, we talked about like coding instruction following long context, a lot of gains coming from post-training, but in particular multi-modal, like, uh, basically everything you're seeing, the gains are there from pre-training. Um, so kudos to the pre-training teams there. They've done incredible work on, on perception and

38. 35:25 – 36:22

    Screen vs Embodied Vision

    1. MPMichelle Pokrass35:25

       multi-modal.

    2. SWswyx35:26

       Yeah, totally. Um, so something that we've been exploring on the podcast for a while, and I'm curious if there's any takes on, on, on your side, is, is there a strong split between like sort of what I call sc- like screen vision versus embodied vision, right? Like, um, are you taking pictures of, uh, are you training on snapshots of a computer for computer use? Um, or, uh, y- you know, a- and anything with charts, anything on, on a PDF, um, is very similar to that. Or pictures from the real world, which is more embodied, right? Like where a robot might be able to use that. Um, uh, people have argued back and forth. [laughs] I'm curious, uh, where the, the, the movement is or the emphasis is.

    3. JMJosh McGrath36:08

       I think one of the, uh, first off, I think that 4.1 is better at both of those things, uh, regardless of how it was actually trained. I think, uh, I would probably somewhat defer to the pre-training team when it comes

39. 36:22 – 37:19

    Vision Benchmarks and Model Improvements

    1. JMJosh McGrath36:22

       to which one you should be using. Um, we're using s- you know, a mixture of both, but we've improved our results across evals on both.

    2. SWswyx36:31

       Awesome. Yeah. Um, yeah, that's something that I think people should, uh, de- definitely do want to explore the more embodied stuff as well because the benchmarks tend to focus on the, the screen vision stuff, you know, um, more, more chat-

    3. MPMichelle Pokrass36:42

       It's always easier-

    4. SWswyx36:42

       ... uh, more controllable

    5. MPMichelle Pokrass36:43

       ... to look at eval that is easy to grade. Um-

    6. SWswyx36:45

       Yeah, exactly

    7. MPMichelle Pokrass36:46

       ... those are the things that get looked at the most, for sure.

    8. SWswyx36:48

       Yeah.

    9. JMJosh McGrath36:49

       I think one of the things that was really funny with both, uh, the 4.1-mini and nano is we had some strange internal eval results, and it turns out that actually the, these new, uh, vision capabilities, they were able to read like, you know, signs in the background and stuff, which was actually changing like some of the validity of our results and so we were, you know, just running into different eval problems as you actually improve the models.

    10. SWswyx37:13

        Um, is there a feature of a 4.1 image gen or is that like a completely different,

40. 37:19 – 38:13

    Model Deprecation and GPU Usage

    1. SWswyx37:19

       um, part of this vision? Like, you know, in, in some sense vision is image to text and the other, the other way around is image gen. Um, is it that simple or [laughs] is it something else?

    2. MPMichelle Pokrass37:30

       It is not. Um-

    3. SWswyx37:31

       [laughs]

    4. MPMichelle Pokrass37:32

       No, no plans right now to, to get, uh, 4.1 image gen.

    5. SWswyx37:35

       Okay. Well, you know, it's very, very popular. Um [laughs]

    6. MPMichelle Pokrass37:39

       We like the GPUs too. [laughs]

    7. SWswyx37:41

       It's like melting your GPUs. I mean, talking about GPUs, right, like, um, you know, part of this whole deprecation of 4.5 and, and moving people to 4.1 is to get back your GPUs. That's a, that's a message that both, uh, uh, Shuki and, uh, Kevin Weil have mentioned. Um, but like you are running all these models concurrently for the next three months. Like, I don't know if you get back your GPUs. I think you just grow the [laughs] usage even more.

    8. MPMichelle Pokrass38:05

       Yeah, I, I do think, you know, people get the message on deprecation and start moving over. So what-- as developers use

41. 38:13 – 39:12

    Fine-Tuning and Preference Steering

    1. MPMichelle Pokrass38:13

       this model less, we can kind of reclaim that compute. But you're right, it takes a while, and the trade-off there is really our commitment to developers.

    2. SWswyx38:22

       Yeah.

    3. MPMichelle Pokrass38:22

       Like if we have something in the API, we won't take it away on, without, you know, sufficient notice. Um-

    4. SWswyx38:27

       Yeah, with some notice

    5. MPMichelle Pokrass38:28

       ... so that's the trade-off that, that is, is right for us.

    6. SWswyx38:31

       Okay, awesome. Um, then a couple other, uh, smaller announcements, fine-tuning available day one, which is I think new for OpenAI. Usually you have to wait like a month or two for the fine-tuning capability. Uh, 41, 4.1 only and, and mini, 4.1 and mini only and nano in future. Um, any specific call-outs for, for fine-tuning? Um, I guess like there's, fine-tuning is this general discipline that always applies, but any wins that you guys can talk about?

    7. MPMichelle Pokrass38:58

       So first off, yeah, shout out to the fine-tuning team. They've w- worked really hard to get this ready on day one. Um, one thing I will say is that I think people have slept on the preference fine-tuning offering, uh, or the... I think

42. 39:12 – 40:10

    Upcoming Reasoning Models

    1. MPMichelle Pokrass39:12

       that's what we call the product. Um-

    2. SWswyx39:14

       Yeah

    3. MPMichelle Pokrass39:14

       ... so SFT is, people know it pretty well. It's the original fine-tuning we had, whereas this, uh, preference fine-tuning is super helpful for steering in a particular style. Um, and so I think not enough people are using that.

    4. SWswyx39:28

       Isn't that only for reasoning models or is that for everything?

    5. MPMichelle Pokrass39:32

       No, that's reinforcement fine-tuning is only for reasoning models.

    6. SWswyx39:34

       Right.

    7. MPMichelle Pokrass39:35

       Pre- preference fine-tuning-

    8. SWswyx39:36

       The DPO stuff

    9. MPMichelle Pokrass39:36

       ... will offer the pairs. Yeah. Um, exactly.

    10. SWswyx39:39

        Yeah. Um, and I thought it was in alpha, which is why I haven't looked into it. [laughs]

    11. MPMichelle Pokrass39:44

        I thought we did it.

    12. JMJosh McGrath39:45

        I think it's RFT that's still in alpha.

    13. SWswyx39:47

        Okay. Well, that's a lot of confusion that we, that we just cleared up. Um, yeah, I think we're going to... So, you know, I'm doing my conference again in June, and I think we're gonna do a workshop on, um, just general all the fine-tuning options and I think that will clear up a lot of things, uh, which is good. Um, okay. New models. Uh, I know that we can't talk a lot about a lot of them. Um, Noam Brown from your reasoning team just said

43. 40:10 – 41:07

    Creative Writing and Model Humor

    1. SWswyx40:10

       that there should be a follow-up on reasoning models soon. Um, what can we say about that? [laughs] And sounds like he's-

    2. MPMichelle Pokrass40:17

       Yeah. We're, we're not the right people to ask, but, but stay tuned for-

    3. SWswyx40:20

       Yeah, but like 4.1 is a good basis for like whatever comes next, right? Like...

    4. MPMichelle Pokrass40:25

       Yeah. Uh, not all of our models kind of build on each other necessarily, but, uh, we think 4.1 is a great standalone offering for developers, and we also think, you know, reasoning models are, are a good tool in the toolbox.

    5. SWswyx40:38

       Yeah. Like I more just generally, like I always want to explore the relationship between non-reasoners and reasoners, and then also like how we merge them. Are we doing routing? You know, anything, anything of that sort. Uh, obviously you have a lot of secret sauce. Um, cool. Uh, and then I think the other thing that a lot of people are demanding or asking about is the creative writing model. Um, will that ever see the light of day?

    6. MPMichelle Pokrass41:01

       We're working on incorporating kind of those improvements into the models more generally.

44. 41:07 – 42:03

    Feedback and Developer Community

    1. MPMichelle Pokrass41:07

       Um, I think that was-

    2. SWswyx41:07

       Okay. So not a separate release

    3. MPMichelle Pokrass41:08

       ... one of the things people loved about 4.5 is like the humor, the green text, the nuance. Um, so we've heard that feedback and I know, yeah, there's lots of folks working on that and trying to bring it into our next models.

    4. SWswyx41:23

       Awesome. Alessio, anything else?

    5. JMJosh McGrath41:25

       Um, no, this was great. Any requests for the developer community? Things that you want them to try out that maybe people are not doing, things you want them to build for you, uh, using the new, the new APIs? I feel like first off, send us feedback. It was really useful to look at, uh, different partners and customers who are using our models and to get this like nice wrapped feedback from them. Um, it allows us to iterate a lot faster. Um-

    6. MPMichelle Pokrass41:52

       And on that vein, you know, opt in to, to data sharing. This just helps us make the model better for you. And one kind of slept-on way to do this is the evals

45. 42:03 – 44:02

    Pricing and Blended Model Costs

    1. MPMichelle Pokrass42:03

       product. So you can upload an eval and, uh, opt in such that we'll pay for the infras- inference costs if we can also use the eval. And this is just another great way, like we'll, we'll use those evals to make sure our models are, are getting better for people over time.

    2. SWswyx42:18

       Yeah. I think the opt, uh, the evals is permanent. Uh, uh, there's no end date announced, but the opt-in in the API is at least until April 30th. Um, uh, I think a lot of people still don't know about it. We might want to extend that so that people can-

    3. MPMichelle Pokrass42:33

       Yeah

    4. SWswyx42:33

       ... do more of that.

    5. MPMichelle Pokrass42:33

       Good flag.

    6. SWswyx42:34

       Yeah.

    7. MPMichelle Pokrass42:34

       I'll, I'll raise with the team.

    8. SWswyx42:36

       Yeah. Awesome. Um, uh, and then I think the, the, the last question I had was on just on pricing. Uh, I think pricing, you know, it's basically just generally cheaper than 4o, uh, but like not a, not a ton, but like cheaper. And, uh, then you're also introducing this concept of blended pricing for the first time, uh, that I've, I've seen it, but, uh, maybe it's just been out there for a while 'cause you have caching and all that. Um, just generally, what is the cache to non-cache ratio that we should be thinking about when thinking about, um, workloads? Like, um, i- is there, is there a general rule of thumb?

    9. MPMichelle Pokrass43:10

       So one clarification, um, which is that GPT 4.1 mini is not cheaper than GPT 4o, so not... it's not just like a blanket decrease in all the models.

    10. SWswyx43:21

        Yeah.

    11. MPMichelle Pokrass43:21

        But however, 4.1 mini is cheaper than 4.1. Um, also not sure if this is widely reported, but we've increased our prompt caching discount from 50% to 75% on these models.

    12. SWswyx43:35

        Yeah, I saw that. [chuckles]

    13. MPMichelle Pokrass43:36

        So that's a, a big input, uh, you know, into figuring out what kind of application you build. Um, and then your question was on like what kind of ratios to think about?

    14. SWswyx43:46

        Uh, like, yeah, blended pricing, right? Like, um, I, I think there's this question of comparability of prices across models-

    15. MPMichelle Pokrass43:52

        Right

    16. SWswyx43:52

        ... um, and across providers because like I, you know, it's like some people are three to one in terms of context to output, and then some

46. 44:02 – 45:03

    Conclusion and Wrap-Up

    1. SWswyx44:02

       are part of that is cached. Um, uh, I, I selfishly, I predi- I make a chart that just plots all the model labs versus all the prices, and you, you, and I'm sure you guys have seen it. Um, and I don't know what numbers to plug in there. So [chuckles] what are people seeing in real life? Uh, what's the m- median, you know, caching rate? [chuckles]

    2. MPMichelle Pokrass44:20

       I don't think we have that off the top. The p-

    3. SWswyx44:22

       Okay.

    4. MPMichelle Pokrass44:23

       The blended pricing is more to just make it easier to compare. Like, so you could say something like, "GPT 4.1 is 25% cheaper than GPT 4o."

    5. SWswyx44:31

       Yeah, you want one number. Yeah.

    6. MPMichelle Pokrass44:33

       Yeah. Yeah.

    7. SWswyx44:34

       No. [chuckles] All right. Well, I'll have to figure it out. Uh, but thank you so much. That was, that was fantastic. Uh, thanks for all the work. I think people are very excited to get to work testing this out, giving you feedback. Um, and I'm sure we'll be back again for the next one, probably the reasoner.

    8. MPMichelle Pokrass44:50

       Nice. Thank you, guys.

    9. SWswyx44:53

       Thank you. [upbeat music]

Episode duration: 45:03