Before we ship a Claude model, these teams try to break it.

1. SPSpeaker0:00

   [instrumental music] Before a new Claude model ships, a small group of customers is already testing it, breaking it, and shaping what ships with it. We sat down to see what they are learning.

2. SPSpeaker0:20

   When you get something new from Anthropic, what is that energy like?

3. SPSpeaker0:23

   We know a storm's ahead, but there's something exciting about a storm because it's all hands on deck.

4. SPSpeaker0:28

   Yeah, it feels like we're moving at the speed of light.

5. SPSpeaker0:31

   That's like getting the call and jumping from whatever you're working on. We have something new, let's, let's figure out what it's like.

6. SPSpeaker0:36

   The moment we get a new model from Anthropic, we realize the grounding has changed.

7. SPSpeaker0:44

   What's it like to work at a company that's helping to shape the frontier?

8. SPSpeaker0:47

   It's insanely fun.

9. SPSpeaker0:48

   All of us are just in learning mode.

10. SPSpeaker0:50

    This moment just feels like a generational opportunity for anyone in this industry.

11. SPSpeaker0:54

    I feel very lucky and also very responsible. We need to continue to push the envelope, continue innovating, be more secure, and making things easier to build with. In a way, I love that I can unlock a new class of developers and builders.

12. SPSpeaker1:11

    What's the first thing you throw at a new model?

13. SPSpeaker1:14

    The very first thing is we will start automated evals, uh, just so that they start running in the background.

14. SPSpeaker1:19

    One use case that is kind of pipe dream that's easy to point to as a particularly complex legal task is drafting an S1. And now with agentic capabilities where these models can go out and find information that they need, synthesize it, edit documents, we're getting to larger and larger chunks of the S1 that you can kind of just send the model on its way to do.

15. SPSpeaker1:41

    Just by swapping in that one model, every question I ever wanna ask it started getting answered. You know, it went from this agent can sometimes answer questions, sometimes gets stuck, to, "Oh my God, it is answering every question quickly and accurately."

16. SPSpeaker1:54

    The dashboard of the testing agent success rate has just increased-

17. SPSpeaker1:57

    [laughs]

18. SPSpeaker1:58

    ... like by, I think it's 20%.

19. SPSpeaker2:01

    Cool. [laughs]

20. SPSpeaker2:01

    Things that don't work today are, like, the best sign for here's what the next models are gonna be way better at. Seeing evals that have never worked start working, and then start working consistently, this model is gonna be, like, something special.

21. SPSpeaker2:15

    What's it like working with Anthropic?

22. SPSpeaker2:17

    It feels like I have a conversation with you almost every other day. The engineers on the team, I feel like are, are almost on the same team.

23. SPSpeaker2:24

    It's less like we're just buying something [laughs] from you.

24. SPSpeaker2:27

    Yeah, yeah.

25. SPSpeaker2:27

    More like we can kind of build, build with you.

26. SPSpeaker2:29

    We have a very high trust bar that anything you publish is not gonna be slop.

27. SPSpeaker2:35

    What is one word or phrase that characterizes what it feels like to actually be building at the frontier?

28. SPSpeaker2:41

    Dazzling. [laughs]

29. SPSpeaker2:41

    Yeah.

30. SPSpeaker2:41

    If that makes sense. It can be kind of blinding at times.

Episode duration: 3:13