Building the future of agentic infrastructure

1. 0:00 – 1:00

   Intro

   1. JYJess Yann0:00

      You can tell we're API nerds because we're saying agents should talk to each other through API. [laughs]

   2. KLKatelyn Lesse0:09

      [upbeat music] It's been a crazy six months, 'cause I think if we look back six months, most of what the Claude Platform was was an API that got you access to inference and tokens in and out of the model. And sure, we had started to build some interesting tools around the model that could get you more intelligence or could help you lower your costs or get more speed. But I think of late, we've started to launch some really rich features within the platform that help you get a ton more out of the model, that help take infrastructure problems off your hands, and help take harness engineering problems off your hands so that you can really get that intelligence, um, at a much lower cost for your team.

   3. JYJess Yann0:51

      Angela, what has been some of the most interesting feedback patterns you've seen from customers?

   4. AJAngela Jiang0:55

      I think the most exciting ones have been around Managed Agents, which has been really awesome, and I know you, uh,

2. 1:00 – 2:15

   Building Claude Managed Agents in production

   1. AJAngela Jiang1:00

      obviously product managed the Managed Agents. [laughs] Um, but it's been great. I think the, the feedback, um, has been really awesome, especially around concepts like memory and concepts like outcomes and dreaming. But I think my favorite one actually is a bit old school, but, uh, I did hear a developer who was using it say they just really love the abstraction levels, and, uh, that always warms my heart a little bit.

   2. JYJess Yann1:20

      Even as these workflows are evolving, um, there's still this concept of agent identity.

   3. AJAngela Jiang1:24

      Yeah.

   4. JYJess Yann1:24

      And so how do we think about the idea of workflows combined with agent identity?

   5. AJAngela Jiang1:28

      Yeah, I think agent identity will actually probably need to be almost somewhat separate, like almost the agent needs to have its own identity. I think today it's still a little bit early. People are still discovering interesting use cases, and there's still a lot of trust that, that you need to give an agent. But I think increasingly that agent identity is closer to something where the agent listens to an outcome that you want, and then it probably comes back to you and asks like, "Hey, in order to accomplish the outcome that you gave me, I need access to A, B, C, and D." And you may say, "You know, A, B, and C are okay, but I don't want you to touch D," and then the agent is able to go and see if it can accomplish that. And then when it does do that, it's able to kind of almost create a service account for itself.

   6. JYJess Yann2:03

      Mm-hmm.

   7. AJAngela Jiang2:04

      And in that world, then you can audit it, you can make sure that, uh, it's successfully doing the things that you want, and I think that that is probably closer to the, the operating model we'll see with the identity layer kind of evolving.

   8. KLKatelyn Lesse2:14

      So a lot of what's interesting

3. 2:15 – 3:00

   How agents talk to each other

   1. KLKatelyn Lesse2:15

      about how agents can talk to each other is like you can build an agent, and you can expose an API, or you can expose just some mechanism by which, like, another agent can talk to that agent the same way that a person might interact with that agent. Um, and we've seen people doing some really interesting stuff from this perspective. Like, some folks have built on Claude Managed Agents within the platform, um, and then have built like a nice thin MCP server that they can go and expose, and then you can have another agent just know exactly how to call into talking to that agent. And so I think that is extremely helpful and, um, you know, just people have been able to do really creative stuff like that.

   2. JYJess Yann2:52

      So why are these kind of workflows legitimate now? Like, what had to change at the model or at the infrastructure layer for this to be possible?

   3. KLKatelyn Lesse2:59

      Models have

4. 3:00 – 8:20

   The future of agentic infrastructure: thinner harnesses and adversarial agent pairs

   1. KLKatelyn Lesse3:00

      obviously just gotten better. Um, I think that's a huge part of it because I think previously you would've had to maybe build a whole ton of scaffolding around the model and standard operating procedure type stuff that makes sure that you have, um, this step happens and then that step happens, and then, right? Like, the nondeterminism of the model used to be a lot more problematic than it is today. I think today the models are really great and can actually figure out within some reasonable guardrails, like, what are the steps I need to take and what are the things that I need to do? I think they've also been able to run for longer. Um, like we have the right infrastructure to let an agent be ambient within a workspace, and it can get triggered by something, and then it can go off and run at some workflow for some period of time and, um, you know, come back when it's actually ready. So I think some of this is evolution of the model. I think some of this is evolution of the infrastructure and the way that people are able to actually stand up and build agents around it.

   2. JYJess Yann3:52

      Angela, I would love to hear your thoughts on how this, like, nondeterminism that Katelyn is talking about has really evolved the harness layer.

   3. AJAngela Jiang3:59

      Yeah. I think in the past, it just felt like a couple months ago really, where people would actually create these, like, very complex boxes of business processes that they'd put together. So first it needs to go through step A, and then step A can only go to B if and only if all this kind of stuff, and it's very complex web of things, and it ended up being like very fragile, I think, uh, in terms of an agent actually being able to exhibit the intelligence that you were hoping it to have, and you were obviously trying to, to automate something or make it a, you know, a little bit more useful, and we've kind of tried to box in a lot of the model. As Katelyn mentioned, with the model getting smarter and I think more and more capable, more agentic in its tool calling, and also deeper in its, like, own reasoning, it's actually able to get to a place where you can kind of start to delete some of those, uh, you know, restrictive parts of the harness. And so I see harnesses actually getting thinner and thinner over time. The other thing I see is that with harnesses getting thinner, you almost have like meta harnesses. I don't know what to call those things. Maybe they're like saddles or something in the future.

   4. JYJess Yann4:53

      [laughs]

   5. AJAngela Jiang4:53

      Um, but, uh, I do think they, they tend to combine strategies. So for example, we've seen people do really innovative stuff where it's, uh, multiple agents, and this is built into the harness in itself, but multiple agents actually compete to go try and solve a problem together. And there's another instance in which, uh, you spin up two of them, and one of them generates an idea, and then the other one's adversarial to it, and there's so many other strategies you can build on top of it. Uh, we recently did advisor strategy as an example where if the model can't figure out what to do, it actually just goes, reach out, and calls a friend, and that friend is hopefully smarter and helps it figure out. So I think things like that is increasingly where harness innovation will go, and we'll see a lot of, um, incredible ways where agents through that expansive architecture are able to solve more and more complex problems.

   6. JYJess Yann5:32

      Yeah. I think this is, uh, really getting at the heart of how we think about, um, harness evolution because-

   7. AJAngela Jiang5:37

      Yeah

   8. JYJess Yann5:37

      ... um, ultimately you have all these different composite strategies, and over time we'll be evolving to having hybrids of these strategies. Like, you could start with, um, the best of n-

   9. AJAngela Jiang5:46

      Yeah

   10. JYJess Yann5:46

       ... sort of like expansive approach, and then once you decide on the right, uh, framework, then you can iterate on that single framework. So you mentioned complex problems. Tell me about a particularly complex problem that has inspired you recently.

   11. AJAngela Jiang5:57

       There was this one really great hackathon winner that was super inspirational. I believe it was called Aria. I might be mispronouncing that. But they, you know, had this problem where inside a bunch of different manufacturing facilities, there is this need for an expert who really understands the machinery. And then you have to kind of listen to and, and monitor whether or not certain machines are going up or down, and then you also have to then read the manuals for, like, that specific component inside the machine. And traditionally, there's one person in that factory who has figured that out over the course of maybe, like, ten, fifteen, twenty years, uh, working in that plant, and then they retire very recently, and then you just, you don't have that expertise anymore. This person was actually able to take all of these types of pieces and say, "Okay, if I upload the standard operating procedure, if I build in monitoring and just attract signals from all the different types of parts of the plant, then I can actually have agents try to mimic that human judgment." And, uh, it was really cool to see them actually be able to take a significant proportion of that and then build in what is basically a redundancy in, you know, that person retiring and actually being able to return to that agentic system and say, "Okay, I can rely on this as a place to accumulate the really important, uh, you know, factory knowledge that's necessary."

   12. JYJess Yann7:06

       What about you, Katelyn? What's an interesting problem you've seen solved recently?

   13. KLKatelyn Lesse7:09

       Yeah. I think one of the cool things that's happening within engineering teams specifically or not even actually engineering teams, just, like, development teams that are trying to get work done, um, is people are coming up with agents that are just more powerful than just, "I can get some code written." Um, like, we have excellent pro- like, Claude Code is an excellent product, a whole bunch of excellent products around how can you actually write code? And I'm seeing people now go further than that and say, "Okay. If I'm starting from the very beginning of a project, what are all the things that are gonna have to happen in order for that work to get done? How do I need to think about running my development environment to actually test the code that I'm running, right? Like, how do I, um, you know, actually write the PRD up front, like, the requirements document, and then, like, later on verify all the QA testing sort of things that need to happen?" And so there's a few examples out there right now where people, um, in larger companies have actually put together full agentic systems and platforms, um, that help them do pretty custom end-to-end development. Um, and I think a really good example is, um, Shopify recently talked about doing this, um, with, I think they called it River. Um, and, you know, there's been a few other examples like this, and I think this is one of the evolutions that has been made possible by just the evolution and what agents are capable

5. 8:20 – 9:15

   Barriers to agentic adoption: security, compliance, and evals

   1. KLKatelyn Lesse8:20

      of.

   2. JYJess Yann8:20

      So it's clear that some organizations are seeing a massive amount of ceiling raising on the problems that they're able to solve, but not all organizations are feeling that. So what do you think is the biggest barrier to that?

   3. KLKatelyn Lesse8:30

      Actual boundaries, like, the things that can get in the way for people, um, security and compliance, guardrails that they need to have in place in order to feel comfortable, like, having agents do the work that they're able to do. Um, evals, I think, is a big one too in order to actually get the most out of the technology.

   4. JYJess Yann8:46

      Security and compliance is definitely a big one here because I think that a lot of teams are operating on, uh, security assumptions that prevailed, you know, twenty years ago.

   5. KLKatelyn Lesse8:55

      Yeah.

   6. JYJess Yann8:55

      And now agents are c- fundamentally changing everything, and along with that comes security. So I think that we have a pretty fundamental idea of what makes a, a safe agent with strong guardrails. I think pushing that concept uphill and re- revising the, the checklist, um, has been a bit of a, uh, a bit of a journey. [laughs]

6. 9:15 – 12:45

   How to measure agent ROI

   1. KLKatelyn Lesse9:15

      Yes.

   2. AJAngela Jiang9:15

      Yeah.

   3. JYJess Yann9:15

      Curious about how you think about ROI of agents and how enterprises can manage that.

   4. AJAngela Jiang9:20

      Yeah. I think it's a, a very top-of-mind question for a ton of different companies right now as they look at how everything is accelerating and how they can accelerate themselves. I think, uh, you know, there's, there's a couple different ways to think about this, but I try to kind of encourage a slightly more, like, simplified mental model. I think oftentimes people wanna jump to, like, "How do I agentify gigantic old school processes that I have?" And maybe they have a hundred and twenty of them, and they wanna go down the laundry list. Or they're like, "Great. This is my moment if, if consumer preferences are changing to, to fundamentally transform my entire product surface." And I think all those things are great, but it's obviously very hard to go through all of those, and there's a, a very, very, you know, wide array of things there. I do think that for an ROI point of view, it's easier to almost start from the individual actually and to be focused on how much faster that individual is getting. And it sounds really almost, like, simple and, and kind of, you know, like, not sophisticated enough, but I actually think if you start there and you are able to accelerate that one individual, you can move from that to then a team. And as you get to the team level, you start to think about the speed that you're generating there. And then, and probably only then, I would actually start then to encourage saying, "Okay. If I can make this team faster, now what is the ability, my ability to take this team and think through and process across my company?" And realistically, right, a, a process typically requires many different teams, many different individuals who don't share standard operating procedures, who don't share maybe the exact same expertise, and that's why we have all of them together, probably, you know, duct taping a bunch of things together. But if you go through that kind of phase where you think about speed at the individual layer, team and their productivity, and then lastly followed by, you know, how you're able to string these workflows together, I think you're able as a company to basically almost light up every single bar that you eventually care about, so you can eventually make your way to that kind of list of a hundred and twenty different workflows that you wish you could be agentifying. And I think if you think about it from that point of view, most of your ROI calculations should probably be on speed first and foremost and productivity. It tends to be much more leading and successful as the primary, uh, mechanism to take a look at. As that starts to, I think, flower a little bit inside of a company, then it makes more sense to kind of transition a bit to be like, "Okay. Well, you know, maybe there are financial metrics that I wanted to push. Maybe there are user metrics that will help drive the overall, like, outcome productivity that I, I care about." And I think if, if, uh, leaders and, and companies are able to follow that kind of process, they're actually able to drive more of that ROI, um, and see it stage by stage.

   5. JYJess Yann11:35

      Katelyn, you keep saying engineering teams, and I kind of wanna double-click on that and say, what is an e- engineering team anymore?

   6. KLKatelyn Lesse11:41

      So engineering team for, for us is it's a set of humans that actually doesn't look that different than an engineering team looked six months ago, twelve months ago. Um, you still kind of need that set of humans who understand the system that they are building, how to operate it, how to be on call when something's going wrong, all of these sorts of things, but each of those humans is just insanely turbocharged by agents that can help them get their work done. Um, and so what we're kind of seeing is a bit of a shift from an engineering team that was maybe, like, one technical lead who's got, you know, opinions on how we should design the system, right? And then a whole bunch of engineers who are, you know, picking up tickets and getting work done to Almost the whole team are people who have strong opinions on here's how we end-to-end build a product or build a system, here's what the technical design needs to look like, and then they're kind of orchestrating their Claudes [laughs] for lack of a better term, um, to get the work done. And so, um, for us, our engineering teams actually look pretty similar, um, but they're able to just get so much more work done, um, than they were in the past.

   7. JYJess Yann12:44

      I'm curious

7. 12:45 – 13:30

   Failure modes: hyper independence and sprawl

   1. JYJess Yann12:45

      how you think about potential failure modes of reliance on, on agents in an organization.

   2. AJAngela Jiang12:50

      I do think it does create a sense of like hyper independence in a way that is maybe slightly false. Like you think that y- of course, now everyone's a builder, so I can go and build, and yes, I can like spin up like 10 prototypes. If, uh, normally you'd be like, "Well, which of these options is the best option?" You do a little bit of thinking. You now have this ability to be like, "Well, this is cheap. Like, why don't I just like launch all 10, and then, uh, whichever one's the winner is the one that we'll pick." And so I do think it creates this like hyper independence on every single person, but oftentimes, um, you know, I think the, the quality that comes together from something that's like more systematic, more holistic tends to then be a little bit harder to coordinate. So I do think that that's a little bit of a failure mode is like by giving everyone hyper independence, but not necessarily organizing them together to a concrete direction,

8. 13:30 – 15:15

   The future: agents as an invisible substrate

   1. AJAngela Jiang13:30

      you might see like sprawl that looks a little bit like this. Um, and that can have pros and cons, but I think there's definitely failure modes that result from things like that.

   2. JYJess Yann13:38

      So where do you think the puck is going? What is the future of agentic development?

   3. AJAngela Jiang13:41

      I think it's like deeply embedded inside the organization to a point where you probably don't really use tools in the way that's like so obviously instantiated. And what I mean by that is, you know, today we all are reaching for this agentic tool or that agentic tool, and this one's good at this, and that's good at that. And I think in the future it'll be closer to like maybe there's some common substrate where we all kind of engage. It's like familiar interfaces and everything like that, but you kind of are just able to t- kind of tag an agent, spin it up and down as you see fit, and it just does a lot of like work by itself invisibly. And so... And it might even actually be proactive. Maybe it's the one coming to you and saying, "Hey, you know, we noticed this thing went down, and so I dug into the details and I figured it out and I fixed it, and here's some PR and maybe you want to review it." Um, and maybe you even told it like, "Next time in the future, uh, this, for things so small, don't bother me. Just go ahead and like ship it." And I think it'll look a little bit like that, where it almost feels like almost like an invisible like substrate that you engage with. And in that world, I think that every person ends up being at a place where they almost kind of like are able to build kind of team-oriented agents, but maybe not in the sense of like it's like another teammate, but more that like the team orientation of some workflow that, say, you and I have, there's an agent for that. Uh, and it's able to actually map to the kind of preferences that the two of us have. But then the three of us have a team-based agent, and it's like understands a slightly different preference, and it starts to kind of fill in a lot of the gaps that are necessary. And again, I think still have the common interfaces and things we have, but we might see a bit more of it as like an operating system or, or something like that instead of, uh, specific tools that we actively reach out for.

   4. JYJess Yann15:14

      And

9. 15:15 – 16:32

   What's next for the Claude Platform

   1. JYJess Yann15:15

      how do you think Claude Platform will help us get to that vision?

   2. KLKatelyn Lesse15:17

      Yeah. I think one of the bigger things that we've been kind of trying to push the boundaries on recently are some concepts like outcomes is a big one. Like we shipped in Claude Managed Agents this idea of an outcome where you tell Claude like, "Here's what good looks like," give it a rubric, how many times can it iterate to go and try and get that outcome, um, successfully before it, it stops, right? And I think as we evolve that concept, we'll probably get closer and closer to a world where you're talking to Claude and you're saying, "I want this outcome and here's a budget. Go." You know? And you don't really have to think beyond that. Um, and I think the i- the idea of what we're trying to get at with the platform is make that so easy. And being in a world where you're spinning up an agent every day because you're like, "Today I have to summarize some interview notes and put together a packet of feedback," right? Or something like that. You can say, "Okay, great. Agents, I want notes. Good looks like this, and you can spend this amount. Go." Right? Um, and then you'll get back what you want, and making it so that you don't have to work hard or think hard to actually create that is the gap that we're trying to fill, making that super easy.

   3. JYJess Yann16:22

      This has been so much fun talking about the future of agents. Thanks so much, guys. [upbeat music]

Episode duration: 16:32