No Priors Ep. 118 | With Anthropic Co-Founder Ben Mann

Hi listeners, and welcome back to No Priors. Today we have Ben Mann, previously an early engineer at OpenAI, where he was one of the first authors on the GPT-3 paper. Ben was then one of the original eight that abandoned ship in 2021 to co-found Anthropic, with a commitment to long-term safety. He has since led multiple parts of the Anthropic organization, including product engineering and now labs, home to such popular efforts such as Model Context Protocol and Claude Code. Welcome, Ben. Thank you so much for doing this.

Of course. Thanks for having me.

So congratulations on the Claude 4 release. Maybe we can even start with, like, how do you decide what qualifies as a release these days?

It's definitely more of an art than a science. We have a lot of spirited internal debate of what the number should be, and before we even have a potential model, we, we have a road map where we try to say, based on the amount of chips that we get in, uh, when will we theoretically be able to train a model out to the Pareto-efficient compute frontier? So it's all based on scaling laws. And then once we get the chips, then we try to train it and inevitably things are less than the best that we could possibly imagine because that's just the nature of, of the business. It's, it's pretty hard to train these big models, so dates might change a little bit. And then at some point it's, like, mostly baked and we're sort of, like, slicing off little pieces close to the end to try to say, like, how is this cake gonna taste when it comes out of the oven? But, uh, as Dario has said, until it's really done you, you don't really know. You can get sort of a directional indication. And then if it feels like a major change, then we give it a major version bump, but we're definitely still learning and iterating on this process, so yeah.

Well, the good thing is that you guys are, uh, you know, no less torture than anybody else in your (laughs) naming scheme here.

Yes.

The naming schemes in AI are s- are something else, so you, you folks have a, a simplified version (laughs) in some sense. Do you wanna, um, mention any of the highlights from 4 that you think are especially interesting, or, you know, those things around coding and other areas, we'd just love to hear your perspective on that.

By the benchmarks, 4 is just dramatically better than any other models that we've had. Even for Sonnet, is dramatically better than 3-7 Sonnet which was our prior best model. Some of the things that are dramatically better are, for example, in coding it is able to, uh, not do its, uh, sort of off target mutations or over-eagerness, or reward hacking. Those are two things that people were really unhappy with in, in the last model where they were like, "Wow, it's so good at coding, but it also makes all these changes that I definitely didn't ask for." It's like, "Do you want fries and a milkshake with that change?" And you're like, "No, just do the thing I asked for," and then you have to spend a bunch of time cleaning up after it. The new models, they just do the thing, and, uh, and, and so that's really useful for professional software engineering where you need it to be maintainable and reliable.