By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

Building the Claude API for the Frontier

Every Claude launch puts the spotlight on the model, but just as much work goes into making sure the API evolves with it.

Lucca SiaudzionisguestJake Dejnohost

Nov 12, 202618mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

Great conversations run long. Their best ideas don't have to.

uListen is free on iPhone, iPad, Android, Chrome, and Claude.

See how

Transcript

25 min read · 4,519 words

0:00 – 0:15
Intro
1. LSLucca Siaudzionis
  A lot of roads lead to prompt caching, not all, but for any customer that we talk to, that caching is the first recommendation we give.
2. JDJake Dejno
  [upbeat music]
0:15 – 1:40
How the Messages API works
1. JDJake Dejno
  Um, so we're gonna be talking about the Messages API today and, and cl- just Claude Platform more generally. Um, for folks who have not built on top of Claude Platform before, what is the Messages API?
2. LSLucca Siaudzionis
  The easiest way to see it is as the cornerstone of our API. So it, it allows you to customize it and build almost whatever you can imagine with it. So we have this very, uh, normal structure that anyone who has been building in the industry is used to by now, in which you have assistant turns and user turns and tool calls in the middle. On a bare bones perspective, you can take it from there and build whatever you want, but into the API itself, we have built a lot of tools to allow you to further customize and have knobs into Claude's behavior. One of those can change Claude's behavior pretty dramatically, like the effort. You take the same model in high effort and low effort, feels like two separate models. You can, from first principles, build anything you want, but with all those tools we have put into the Claude API, we're giving you a lot of, like, an easier path to get into the ultimate app you want to build. Some of it comes from us having a very opinionated take, so we're very close. We're in the same room of research, right? We share a building. We share a lot of Slack-
3. JDJake Dejno
  Mm-hmm
4. LSLucca Siaudzionis
  ... uh, conversations, a lot of meetings.
5. JDJake Dejno
  [laughs]
6. LSLucca Siaudzionis
  [laughs] And with it, we have a, a lot of a good take on what actually makes Claude shine. We put those as the default in the API. So if you just take a new-- to copy/paste from our documentation, the most basic request, and it's gonna carry what we think is the best out-of-the-box experience. But with it, there's also, like, all those knobs, so such as effort-
7. JDJake Dejno
  Yeah
8. LSLucca Siaudzionis
  ... that you can try to prompt Claude into having a certain type of behavior or not. But with those knobs, you can do more than any prompting outside could ever do. So it actually unlocks a level of customization that
1:40 – 4:15
Deciding what features to release
1. LSLucca Siaudzionis
  is just not available otherwise.
2. JDJake Dejno
  Talk to me about how we decide sort of like what to ship, like what knobs and dials do we allow to actually unlock the, the best Claude intelligence?
3. LSLucca Siaudzionis
  It, it's a, it's a very two-way conversation with the researchers at this company.
4. JDJake Dejno
  Yeah.
5. LSLucca Siaudzionis
  So one way comes from them to us, in which they are experimenting and building it. They start finding out things that Claude is good at, and that comes to us as a very raw thought, like, oh, you know, once upon a time, a long time ago, they started noticing that Claude is getting better at using computers.
6. JDJake Dejno
  Yeah, I remember that day. [laughs]
7. LSLucca Siaudzionis
  And then ultimately, that became the computer use tools we have in the API.
8. JDJake Dejno
  Yeah.
9. LSLucca Siaudzionis
  But sometimes it goes the other way, in which we are talking to our customers, and we're understanding what our customers want and ask us more for, and we convert those into actually asks for research to train into, obviously not the past Claudes, but the new Claudes, because there's always a pipeline. So any user ask or user need we, we have, we can eventually put in an API. And that two-way conversation keeps going. It's an ever, you know, it's a good, good chunk of my job, and it's how we actually try to get that best Claude out each time.
10. JDJake Dejno
  Yeah. And you know, what, what have we done to make a choice that we felt like really, you know, empowers developers to, to bring the best of Claude to their end users as well?
11. LSLucca Siaudzionis
  It's funny 'cause these, these two feel so small, but they are very big in practice. So being able to change effort and system prompts mid-conversation are two things for me that really show the power of being able to have a knob later on. And you could always change a system prompt, you could always change effort, but then you break prompt caching, and that to a user is, like, a deal breaker.
12. JDJake Dejno
  Yeah, that's the unique thing that we can provide there, which is-
13. LSLucca Siaudzionis
  Yeah
14. JDJake Dejno
  ... okay, how do you make these changes? How do you, uh, you know, empower the, the experience you want to while, you know, keeping costs low or, or otherwise, you know, unlocking the best intelligence?
15. LSLucca Siaudzionis
  Exactly. And you know, sometimes a user has an agentic flow that has been operating under high effort, and they know that the next request is actually easier, but if they change the effort to low, the cache is broken, and that defeats the whole purpose of their saving clause. But being able to change effort mid-conversation really gives it that power.
16. JDJake Dejno
  So we're, we're making these changes. We have, like, this tight loop with research, with inference, with a bunch of other teams, safeguards included. What does it take to get from idea to launch in the API and empowering, you know, our, our users to leverage Claude?
17. LSLucca Siaudzionis
  After the idea, the first thing you need is some sense of how well Claude works, and that usually goes around evals in each step of the process, you know?
18. JDJake Dejno
  This is what I was hoping to get into.
19. LSLucca Siaudzionis
  Yeah.
20. JDJake Dejno
  Like, you know, how do you-- Like, we have all these changes coming through. Like, what do we need to do to en-ensure that, one, workflows aren't breaking or otherwise, you know, you're getting the best from, from any of these knobs and dials we're bringing?
21. LSLucca Siaudzionis
  Yeah. And, uh, we run a lot of evals, and we recommend each customer who is adopting each of these knobs to run their own evals because, you know, ultimately we are an API. We're very generic. We're trying to serve everyone. And for a customer building on top, they have their own use case. They have their own personalization
4:15 – 8:15
Handling model versioning and migrations
1. LSLucca Siaudzionis
  that applies to them and only them in the world.
2. JDJake Dejno
  So we, we talk a lot about changing things in the API as a result of unlocking intelligence, sort of staying at the frontier with Claude. But, you know, traditionally, APIs have been versioned. There are breaking changes and, you know, we think about these things in a way of migrations. It's like, okay, user X needs to make a change in API Y and so on, and, and roll that out to other users. It takes some time. With models, it's actually been quite different.
3. LSLucca Siaudzionis
  Yeah.
4. JDJake Dejno
  Um, so describe like, you know, in, in your mind as you're thinking about new changes in the API, how do you actually, uh, ensure that, you know, migrations are relatively slow, uh, you know, that, that rollouts are relatively fast and otherwise, you know, users can ingest these changes well?
5. LSLucca Siaudzionis
  Yeah. I'm very curious about your thoughts on this as well, actually, so ask me back the question.
6. JDJake Dejno
  Want it right back? Okay. [laughs]
7. LSLucca Siaudzionis
  Yeah. [laughs]
8. JDJake Dejno
  Um, yeah, I, I think, um, one thing that's been very interesting is, you know, I, I come from o-other worlds, uh, and other roles where API versioning has been relatively, uh, important, but also relatively painful. Um, models provide such a, a, an interesting version mechanism for us, right? I mean, you have this model string, you have an API version, you have an intertwining of, of these two things. You have prompt stability, you have buckets of, of changes or switches rather that, that allow you to bring ne-new changes to, to your end users. And it's actually been very fortunate that models have been able to be really on this frontier of intelligence, and because of that, p- you know, users are more willing to, to make adjustments because the value is so high for making those adjustments. And I think that is such a powerful, uh, mechanism both for our users, but also for us to ensure that we're bringing the best intelligence forward. And so, you know, we, we think a lot about, you know, how to make sure our, our API versioning strategy keeps up with, with model intelligence, but also how model strings effectively as a version, uh, m-mean that we're bringing the, the best intelligence very, very quickly to users. But again, it comes back to what you just described, which is your harness still needs to be evaled.
9. LSLucca Siaudzionis
  Yeah.
10. JDJake Dejno
  It needs to be tested, it needs to be validated that, that these changes are, are, are, are working well. And, you know, anything from number of tokens used to intelligence to, um, outcome from the task a-are all measured and, and I think are, are very, very important for any end user to, to really think about.
11. LSLucca Siaudzionis
  Yeah. To, to elaborate on something you said there, sometimes we need to do, you know, some implicit version when you go from one model to the other, and the easy part of implicit version is when you're adding new capabilities, no one complains. It's difficult sometimes when we need to deprecate one capability.
12. JDJake Dejno
  Yes.
13. LSLucca Siaudzionis
  Someone relies a lot on that.
14. JDJake Dejno
  Yeah, I didn't go into deprecation. I did additive, right? [laughs]
15. LSLucca Siaudzionis
  Yeah. One that comes to mind is when we changed the thinking to, from regular extended thinking that everyone is used to, to adaptive thinking. We knew by then that adaptive thinking was already performing better in evals, and we wanted to roll it out, and we want it to be, you know, the way to do thinking for the future. So we did roll it out, but for a while we kept it as this other type of thinking that you could still switch back as a developer, toggle back to the old thinking that you used.
16. JDJake Dejno
  And I didn't even touch on these. We have beta headers and, and-
17. LSLucca Siaudzionis
  Yeah
18. JDJake Dejno
  ... other mechanisms for you to Roll back to, uh, previous changes if, if you need time to update your harness.
19. LSLucca Siaudzionis
  Exactly. And, uh, we try to, to buy that time. Uh, some of it at some point, we need to do a harder deprecation because at some point, models stop being trained for this old way. And then for us to put a null in API that doesn't do anything or doesn't do what it's supposed to would be even more harmful to the developer. So that's the end date of the migration, essentially. But we do try to move it out as much as we possibly can. You always have a model or two to slowly migrate and we change what is recommended, but you can still go, you know, stay at your old way.
20. JDJake Dejno
  Yeah.
21. LSLucca Siaudzionis
  Um-
22. JDJake Dejno
  So maybe stated plainly, you know, how do you think about the sort of loop of capability comes out? Like, what should an, a developer think about as they're up- upgrading to these new versions? Like, what does that sort of loop with their evals look like?
23. LSLucca Siaudzionis
  Yeah, I think in general, you should, like, apply this thinking for every single part of your workflow, even as you're using an app. If you're using Claude Code, it's always good to periodically reevaluate, uh, all the loop and setup you have around your own product when these models change, because models can be really good at one thing that makes one part of your harness different. And this is ... actually goes into a whole conversation about building harnesses in general.
24. JDJake Dejno
  Mm-hmm.
25. LSLucca Siaudzionis
  Which they're very good at leveraging this capability from the model. But at some point, if a model is 10 times smarter than the previous model was, then the harness can start to be a limitation. So I think even these forced deprecations at times are a good forcing function to reevaluate-
26. JDJake Dejno
  Mm-hmm
27. LSLucca Siaudzionis
  ... parts of the harness
8:15 – 11:22
Best practices for managing cost
1. LSLucca Siaudzionis
  that maybe don't need to be there anymore, whether this new feature can help you get better prompt caching. And if you get better prompt caching, you get better cost savings.
2. JDJake Dejno
  So, you know, we, we do a lot of things on our end and a lot of-- make a lot of changes, make a lot of updates for intelligence. We also make a lot of them for cost as well.
3. LSLucca Siaudzionis
  Yeah.
4. JDJake Dejno
  And, you know, prompt caching is one of the, the largest levers that we have in, in order to, to control for cost. What are some other, you know, ways that you think about, like, cost controls in the API a- a- and otherwise prompt caching as, as one of our unique value propositions that, that we can control and help our users with?
5. LSLucca Siaudzionis
  I think a lot of roads lead to prompt caching, not all, but [laughs] to take a second to talk about prompt caching itself, uh, for any customer that we talk to that still hasn't optimized their prompt caching, it's the first recommendation we give, because the one that can bring your cost down meaningfully.
6. JDJake Dejno
  And what, and what is prompt caching? Like, how does that work?
7. LSLucca Siaudzionis
  Prompt caching is a good way to see if you're, if you're in the middle of an injectic turn in which your second request is very similar to your first request, where it's more like an append, you're just adding new things each time. We cache those blocks that already ran once. Whenever, you know, our sales teams are talking to customers, so the first thing comes up-
8. JDJake Dejno
  Yeah
9. LSLucca Siaudzionis
  ... like have you, how much have you worked on your prompt caching rates?
10. JDJake Dejno
  Yeah.
11. LSLucca Siaudzionis
  Can you get it to up to 90% prompt caching?
12. JDJake Dejno
  Yeah.
13. LSLucca Siaudzionis
  'Cause th- those are the dream numbers to achieve, and it really translates to cost, and it's a win-win all around. The other ones are like, and you can tell me what you think about this in general, like, how do you approach cost per task versus cost per token? 'Cause some of those features we have, they don't change the cost per token of the model, but they change the number of tokens ultimately used.
14. JDJake Dejno
  It's a, it's a tough one because I, I think, you know, tokens and, and, and cost per token is just the unit of measurement that a lot of people look at. But a lot of times we've found in many evals, and so have our, our own users have found through many evals, that the cost per token, um, also needs the dimension of intelligence per token. Uh, and intelligence per token is how quickly can the outcome be achieved as a result of the model that you've chosen. So, you know, one of our higher intelligence models, maybe a Fable version versus a Haiku version, is gonna be wildly different in terms of the number of tokens used to, to achieve that same outcome. And so I think we're almost back to this evals point of, okay, great, how do you make sure that your harness, the model that you've chosen, the knobs and dials that you've pushed on, pulled on, however you want to frame it, are tuned and tweaked well enough where you have confidence in your eval story that, oh, actually, if I upgrade to this new model or change this dial, I'm thinking or, or otherwise that you're still achieving the same outcome, but potentially on a, a different cost curve, uh-
15. LSLucca Siaudzionis
  Yeah
16. JDJake Dejno
  ... one that's favorable to you. Um, we see that a lot with, you know, users using, um, you know, Haiku or Sonnet, uh, in some cases, which is great, but then not realizing, oh, actually, if I were to just use one of these higher intelligent models, that, that'd be in a, in a much better place. That's something that I'd love to see more of and, and, you know, even have us build product around and-
17. LSLucca Siaudzionis
  Yeah
18. JDJake Dejno
  ... and, and help our end users do, do that even easier than, than they can today.
19. LSLucca Siaudzionis
  The biggest though is when the, the stronger model is more expensive, uh, but then it solves a task that you couldn't solve before.
20. JDJake Dejno
  Maybe the inverse is true too, right? Where as the models on all classes get more and more intelligent, you know, there are times where you might use a model with slightly less intelligence that is now capable of, of handling that same workflow.
21. LSLucca Siaudzionis
  Yeah, exactly. So maybe you're using Fable 5 for something because it's the only model that could solve this.
22. JDJake Dejno
  Yeah.
23. LSLucca Siaudzionis
  But now Opus 5 can solve it for you.
24. JDJake Dejno
  Yeah, exactly.
11:22 – 14:41
Balancing access and safety with Mythos and Fable
1. JDJake Dejno
  Well, let's, uh, let's maybe shift gears slightly. Um, I would love to talk about, you know, we talk about the API, we talk about leveraging for intelligence, we talk about eval flows. What we haven't really talked about is how to, um, think about good actors versus bad actors potentially in, in misuse of the API, because I know, you know, a very top of mind topic for most folks, um, that are, are using these APIs in, in safety is, is our number one mission, you know, as Anthropic. So how, how do you think about the, the difference between good acting and bad acting in the models and maybe some of the changes we've been making recently, what are those intended to protect and why is that so important?
2. LSLucca Siaudzionis
  Whenever we get a new model, we start seeing how smart it is. We get the, the first feeling is the, this excitement like, wow, this model can do things that it couldn't do before. And the thing that I always tell myself, and everyone in this company tells themselves, is we have a lot of ideas, but we have such a large pool of customers that they're gonna have more ideas than we do. So we take this model, and it is our number one mission to also like get it in front of users so they can build all these amazing products. We have users who are doing science research with the model, so they need to get their access to this. And the second instinct is the, the other side of that in which you worry about like, what will a bad actor-
3. JDJake Dejno
  Yeah
4. LSLucca Siaudzionis
  ... do with this model? And, you know, navigating that balance is the focus of a lot of people. It's one of my focuses, and you see it in the very clear split of the Mythos and Fable models. So-
5. JDJake Dejno
  Yeah, let's talk about, yeah, let's talk about that
6. LSLucca Siaudzionis
  ... you know, e- every model in the API runs with a lot of classifiers, and the, a lot of these classifiers is trying to prevent bad usage of it.
7. JDJake Dejno
  Yeah.
8. LSLucca Siaudzionis
  You know, it was born of the fact that there's a lot of cyber good to be done.
9. JDJake Dejno
  Yes. Which we believe, I mean, we, like our, our mission is safety be- to bring best intelligence, you know, best outcomes to society based on that intelligence. How to cure the next disease, how to, you know, ensure that we're having a better blockage on, on cybersecurity issues. You know, there's, there's a bunch of good to come from it.
10. LSLucca Siaudzionis
  Yeah, exactly. And it is very important for our mission to actually get the right people-
11. JDJake Dejno
  Yeah
12. LSLucca Siaudzionis
  ... with access to Mythos. And then- Fable is the other side of this coin for anyone who doesn't go through this entire like KYC process can actually get and use Fable. And it's the same intelligence, the same model weights at the end of the day, and it's just about how much you can do with this model based on classifiers. The other part of the safety coin is there's a lot of distillation that happens. If we put a model out there that gets distilled by, and created into a- another model that sure will be weaker, but it'll still be a strong model. That model being un-safeguarded in the wild is an incredibly dangerous position for the world, and it is something that we work a lot on to try to stop, and we're very forthcoming and honest with users about this.
13. JDJake Dejno
  Yeah.
14. LSLucca Siaudzionis
  So we just rolled out a, a batch of changes on, uh, thinking protections, and we document it as, as measures we need to do to prevent distillation.
15. JDJake Dejno
  Yeah. How do you make sure, you know, we, we are balancing the developer experience with some of these safeguards? What are some other-- what are some mechanisms that we use to make sure that that happens?
16. LSLucca Siaudzionis
  So we run an absurd amount of evals. Everything that eventually makes it public, there were ten other variations that we experimented internally that we're just trying to see what is the actual best one. And it is a difficult problem, but an extremely data-driven solution, and it takes a lot of people [laughs] looking at a lot of numbers over a long period of time to eventually make one decision, and that's one feature that makes it up.
17. JDJake Dejno
  It's actually quite fun to, to look at all these numbers as well, right?
18. LSLucca Siaudzionis
  Yeah.
19. JDJake Dejno
  And even getting to the point where the model is maybe so good at one of these evals, even in the face of some of these changes that we're making, that we actually have to make the evals harder, uh-
20. LSLucca Siaudzionis
  Yeah
21. JDJake Dejno
  ... in order for, for the model to, you know, continue to hill climb or, or otherwise improve.
22. LSLucca Siaudzionis
  Yeah. Eval
14:41 – 18:29
What’s next for the API
1. LSLucca Siaudzionis
  saturation is a very funny, good problem to have that we keep running into. [laughs] We had an eval that at some point it stops being meaningful [laughs] 'cause every model can solve it.
2. JDJake Dejno
  Yeah. You know, maybe what's, what's one thing that you're excited about for the future when it comes to model launches generally that you can share, of course, things that, you know, maybe more broadly you- you'd love to see and, and continue to push on?
3. LSLucca Siaudzionis
  The most exciting one is always new capabilities. So there's always this drive internally whenever a new model is fresh out of the oven. It's like, how-- what can you do? Like, can you test all the prototypes that you built before that were failing? And I think ultimately it's all about model intelligence. Everything else next to that matters, but nothing matters as much as model intelligence. So what we do a lot, we, we evaluate all of our features. We re-evaluate many features that we didn't launch before 'cause they weren't good enough. We always have a few features in our set of options that we haven't launched yet because it's just not at the performance level we want to put out for users.
4. JDJake Dejno
  Yeah.
5. LSLucca Siaudzionis
  Every time we put something in the API, it needs to be, you know, it has a stamp of quality.
6. JDJake Dejno
  Yes.
7. LSLucca Siaudzionis
  We really care about it.
8. JDJake Dejno
  Yes.
9. LSLucca Siaudzionis
  And the thing that excites me the most when we have a new model coming up is rerunning evals on all of those. Like, do any of those clear the bar now? And when it does, we actually put it out there.
10. JDJake Dejno
  Mm-hmm.
11. LSLucca Siaudzionis
  And it's something that we also try to pass that same insight to our users. Like all these things that were very close but not there before-
12. JDJake Dejno
  Mm
13. LSLucca Siaudzionis
  ... are they there now?
14. JDJake Dejno
  Yeah. How are you thinking a lot about long running and evolving the API to power these autonomous flows as we see AI, you know, running into all these workflows?
15. LSLucca Siaudzionis
  A lot of our API [laughs] is designed at the end of the day about for agents. So, you know, if you open our docs, you'll very quickly see about our cloud managed agents offering, and there's a lot that can be said about that. But it's-- it gives a really easy way to just spin up and have very, very powerful agents. But then the other side being as a-- as we talked a lot about the cornerstone, the Messages API itself. We have a lot of features there, but most of them, they are, at the end of the day, about agentic flows.
16. JDJake Dejno
  Mm.
17. LSLucca Siaudzionis
  This comes from us talking a lot to customers who build agents.
18. JDJake Dejno
  Yeah.
19. LSLucca Siaudzionis
  The other tools we have are a lot of, you know, our client-side tools, and they are things that we know work well. And this is-- comes from our, like our huge battery of evals. You can always use your own batch tool or text editor tool, but you can also use ours. And the others being all those like toggles to control cloud intelligence. Obviously, you can control thinking, you can also control effort.
20. JDJake Dejno
  Mm-hmm.
21. LSLucca Siaudzionis
  And those matter because they really change your agent from like being able to do something or not.
22. JDJake Dejno
  Yeah, and I think a good iteration of this too, or, or a good showing of, of what the Messages API can do is through all of our first party apps. Um, you know, all Cloud Code, Cloud AI, but also we, we have cloud managed agents as well. You know, it's been very great to see a lot of this, you know, happening where we can take off a lot of this from your plate as a-- as an end developer using the cloud platform, making that easier and easier for you as you go, as you go into these higher order agent abstractions, which is really cool. Um, and so yeah, prompt caching, there's, you know, thinking tuning, there's consistently looking at, um, changes we're making to improve efficiency in the API and, and, and otherwise making things faster. Um, so once a lot of those things land in, in our launch, I think we're gonna continue to see this evolution more towards agents and, and leveraging the Messages API to, to make that happen.
23. LSLucca Siaudzionis
  Yeah. And actually, cloud managed agents is built on top of the Messages API.
24. JDJake Dejno
  Yeah.
25. LSLucca Siaudzionis
  So it's also another case of us just being very tightly integrated in-house. Because we are building this agents API, we, at the messages level, understand a lot of what they need. And when we understand what they need, we put it in the API available, and everyone building agents, whether through the cloud managed agents API or just straight from Messages API, gets all the same lessons to them as well.
26. JDJake Dejno
  Yeah. Makes sense. Well, it's been awesome talking with you about messages, about the API, and about just cloud platform generally. We have a, a lot of work ahead of us, uh, but, you know, it's very exciting to, to hear about the ways that you think about model launches and, and evaling and just making sure that everyone can get the best intelligence from, from Claude. So thanks.
27. LSLucca Siaudzionis
  Yeah. Great chatting. [outro music]