EVERY SPOKEN WORD
40 min read · 7,923 words- 0:00 – 0:07
Intro
- SPSpeaker
[on-hold music]
- 0:07 – 0:14
Founders’ quick tour: three distinct Managed Agents use cases
- SPSpeaker
So all you guys are using Managed Agents in different ways, and, um, maybe, Sajjad, I'd love to hear about how you're using Managed Agents in your new meeting product.
- 0:14 – 0:52
Meeting assistant: pre-meeting briefs and post-meeting task automation
- SPSpeaker
Yeah, absolutely. So for us, we think about meetings in terms of the full journey of the meetings. There's what happens before it, during it, and after it. Almost everybody confines the thinking to literally just the period during it, but the place where agents become the most interesting are thinking about how do we help you prepare for the conversation that you're about to have? How do you make sure what you actually wanna get done happens in that conversation? And then how do you execute on the things that are supposed to happen after? And so we've been using Managed Agents to build and design a lot of new features that we're building, both on the briefs side of getting you prepared, and on the tasks and task automation side of handling your work after for you.
- SPSpeaker
And then Mihir, uh, how about yourself?
- 0:52 – 1:35
Sales platform: Watchtower cross-account agent built fast
- SPSpeaker
So our product is agents for sales teams, and the core of the product is this idea of an agent on every account that's always working and building memory and figuring out what to do to progress it. But there are a lot of workloads for sales teams that are not just restricted to one account. And so if I'm a sales rep, it's like, "What are the top five accounts I should go after today?" Or if I'm a manager, it's like, "What deals should I forecast for the next quarter?" And so on. And so we wanted to build, we call it Watchtower.
- SPSpeaker
Got it.
- SPSpeaker
We wanted to build this product that's, like, cross-account reporting and aggregation and roll-ups. And that agent is fairly different than the shape of our, like, per account agent, right? And so it needs a bunch of different tools and, and features and so on. And so we spun that up in Managed Agents in, like, two weeks, and that is now, like, one of our most used features, and also powers a lot of other parts of the product.
- SPSpeaker
And then Todd, rounding it out.
- 1:35 – 2:13
Product analytics: sandboxed code snapshots paired with real telemetry
- SPSpeaker
Yeah, so we have a product analytics solution that integrates into applications and captures what users are actually doing in it, so you can feed back data on how to make it better. Now, as people have moved faster and faster, as we can imagine, it gets harder and harder to instrument those application, understanding kind of where that analytics drifts from what's going on. So we built a new agent which sort of sits in front of our platform, and what it does is it integrates the codebase and connects what the codebase is doing to what's actually happening in reality. So we're using Managed Agents to actually snapshot a version of the codebase, compare it to what's actually happening in the application, and start proactively sending insights and even fixes to the application.
- 2:13 – 3:40
Why “Outcomes” matter: don’t show a brief unless it’s correct
- SPSpeaker
Nice. I'd love to hear more about kinda the details and, and the features you're actually using. So, uh, we were talking a little bit earlier about outcomes being really important to you, and maybe tell me a little bit about kind of how you think about outcomes and why it's such an important feature-
- SPSpeaker
Yeah
- SPSpeaker
... of Managed Agents.
- SPSpeaker
So let's talk about briefs as a feature for what we're doing, right?
- SPSpeaker
Yeah.
- SPSpeaker
Briefs are meant to surface to you who's the person you're about to talk to, why are they relevant, what did you agree to talk about, and what's important to get out of the meeting, for example, pre-context from what you might have forgotten from weeks ago. Briefs are awesome when they're correct, like, if they actually tell you who you're talking to correctly. If you're talking to a person named Tom, and it gives you information about the wrong Tom, it's actually almost destructive, 'cause the moment you end up in the meeting, you click to learn more about them, and you end up confused. And so being able to correctly identify whether the agent produced the right outcome and literally choose not to show the user anything at all if it didn't is way better than giving the user a false positive information. So Outcomes is super helpful because we can literally just take the agent and have a completely independent assessor of whether the agent actually produced the right outcome in that case in question, and then make sure the attribution of sources is actually correct. And this means that when we show things to our users, they're correct all the time.
- SPSpeaker
Mm.
- SPSpeaker
It also helps drive feedback loops, like the more you can assess that on the fly, the more you can bake into the initial agent loop, making sure it gets it right from the get-go, but it guarantees essentially a good product experience.
- 3:40 – 4:52
Designing the outcomes rubric: correctness plus scannable UX
- SPSpeaker
Yeah, maybe specify a little bit. So an outcome is basically supplying a rubric to the agent and allowing the agent to kind of iterate until the rubric is satisfied. So in this case, how do you kind of structure and set up the rubric?
- SPSpeaker
Yeah. So for the rubric, we're looking at all of the different things that are important to the end user to determine whether or not the pre-read was effective, and so this depends a lot on the type. For example, there are aspects of it where it's, did you literally just pull the correct LinkedIn for the right person? And that's, like, a very objectively verifiable thing that you can actually assess against. There are aspects where it is about the way it's presented to the user, so that information ought to be presented to you in a way that's easily scannable and readable.
- SPSpeaker
Right.
- SPSpeaker
Uh, and so the rubric will also assess for things like that-
- SPSpeaker
Right
- SPSpeaker
... where you're actually making sure it's consumable. Otherwise, you know, you get a five-page pre-read before a meeting-
- SPSpeaker
Yeah
- SPSpeaker
... like, nobody is gonna look at that. And even if it's, like, structured in the wrong order of, like, it doesn't first tell you who the person is, then why you're meeting them, then what to do, like, even then it's hard, right? So we'll look at each of the things that from a UX perspective we want the user to experience, turn that into part of the rubric, and then he'll climb against that rubric.
- 4:52 – 6:51
Runtime evaluation and independent verification context windows
- SPSpeaker
That makes a lot of sense. But one clarification, though, is so basically is this-- this is at runtime. So-
- SPSpeaker
Yeah
- SPSpeaker
... so basically, if I have a meeting, I guess, you know, say it's in an hour, your agent might kick off some period of time prior to that-
- SPSpeaker
That's right
- SPSpeaker
... and spin up, create this, for example, I guess it would be a brief, and that is graded against outcomes-
- SPSpeaker
That's right
- SPSpeaker
... iteratively at some point, in, in some time period before the meeting. Is that right?
- SPSpeaker
That's right. Yeah.
- SPSpeaker
But then how do you inject kind of ground truth? Like, you said you can verify, like, the LinkedIn, but, like, you have to go fetch that for that... Like, how are you actually doing ver-verification with-- when external information is needed?
- SPSpeaker
Yeah. Yeah, absolutely. Um, so we, we run these, for example, 24 hours before the conversation.
- SPSpeaker
Oh, okay.
- SPSpeaker
So it gives you a pretty long time window over which things can be assessed.
- SPSpeaker
Cool.
- SPSpeaker
Um, in terms of verification here, I think the interesting thing is agents that do the work and agents that evaluate the work tend to just have decoupled assessments of whether things are correct or not.
- SPSpeaker
Right.
- SPSpeaker
So even an agent that's doing independent work can look at the LinkedIn and can look at the email and can look at the calendar invite and, like, make its own judgment of-
- SPSpeaker
Sure
- SPSpeaker
... like, was this probably correct or not?
- SPSpeaker
Yep.
- SPSpeaker
And, you know, like, even I, when I show up to a cold email, cold meeting-
- SPSpeaker
Yeah
- SPSpeaker
... like, I can look up the LinkedIn and be like, "This does not look like the right person."
- SPSpeaker
Yeah, yeah.
- SPSpeaker
I'm not sure why, but it doesn't look right.
- SPSpeaker
That's right. That's right.
- SPSpeaker
And this is the value of having something independent from the tr- the trace doing the work.
- SPSpeaker
... evaluating the work.
- SPSpeaker
Yeah, and with Managed Agents, we actually use an independent context window-
- SPSpeaker
Exactly
- SPSpeaker
... independent verifier agent that's kind of baked into outcome.
- 6:51 – 9:44
Memory + “dreaming”: learning org-specific logic and answering complex questions
- SPSpeaker
That's really cool. I wanna ask you about memory. So we were talking before earlier about your Watchtower product and how it's, uh, using memory, and I'm, I'm really interested in this topic, so maybe tell me a little bit about kind of memory and dreaming and, and how you're thinking about these things.
- SPSpeaker
Yeah. So one of the things that's hard about this problem is, like, there are so many concepts within a company that are specific to that company, and so for example, like how a sales team defines ownership, or how they do their forecasting, or like whatever, right? Like how, how their deals work, et cetera. And that's especially hard because a lot of the source data is pretty messy, right? So everybody's Salesforce has like 30 different fields that mean the same thing that were like deprecated, whatever, right?
- SPSpeaker
Yeah.
- SPSpeaker
And so-
- SPSpeaker
Right
- SPSpeaker
... uh, the idea is, like, if someone asks a question, right, like if I ask a question about like what are the, what are the deals I should forecast for next quarter-
- SPSpeaker
Yeah
- SPSpeaker
... that's like a pretty, A, an important question to, like, be able to answer correctly, and there's a lot that goes into that. And so the way that we figure that out is, one, as people interact with the product, we learn those concepts-
- SPSpeaker
Right, right
- SPSpeaker
... and then store them in org-wide memory. And so that then enables us to give better and better results for, like, everyone else. And so that's one of the things that saves us, A, from like having to go in and do a bunch of like config at the beginning.
- SPSpeaker
Yeah.
- SPSpeaker
Like, we don't have to do that, but then the system can just very quickly pick up those concepts and get interaction. And we'll do some things where when you first ask a question, we'll, like, ask you, ask user a question and clarify-
- SPSpeaker
Yeah
- SPSpeaker
... and then save those.
- SPSpeaker
Yeah.
- SPSpeaker
So that's one thing. And then the second thing that we do, not just memory, but the second thing we do that I think Managed Agents has been pretty helpful for is, like, some of these queries or use cases are pretty complicated, require pretty deep work across a bunch of accounts. And so if I have, you know, like 500 deals and I wanna do some deeper work, like I can do some basic filtering-
- SPSpeaker
Right
- SPSpeaker
... but maybe the system wants to go deeper. And so it'll actually write code to be able to answer the question, and it can even do programmatic tool calling, and so it can go and narrow down a candidate set and then, like, fan out more work to independent agents that work on every single account and roll up feedback back.
- SPSpeaker
Yeah.
- SPSpeaker
And so it's both the concept of, like, learning all of these org-wide business logic, right? In a way that generalizes across, like, every team is different, and then also being able to write code to be able to handle arbitrarily complex queries across, like, relatively large, you know, sets of accounts and-
- SPSpeaker
Mm-hmm
- SPSpeaker
... things like that.
- SPSpeaker
Well, you were telling me earlier about kind of the setup of your product. So it's, it's basically I'm an account executive, and I have some number of accounts, let's say it's 100. You spin up an agent for each account.
- SPSpeaker
That's right. Yeah.
- SPSpeaker
So it's basically like a team of agents that's servicing me as like an account executive or salesperson, and as I'm interacting, like you said, you're, you're, you're learning memory, so you're learning things about maybe my preferences or org-wide preferences, but then this new product is you have this memory collected across this team of agents, and it seems like it's doing more of like kind of a-- it's doing, quote unquote, "dreaming," where it's looking across all those memories and suggesting things. So tell me, tell me about that specifically and, like, what it, what it's doing.
- 9:44 – 10:36
Two-layer memory model: per-account persistent agents plus cross-account memory
- SPSpeaker
Yeah. So there are two parts of the memory, um, that are, that are interesting. One is on each account-
- SPSpeaker
Right
- SPSpeaker
... there's an agent that runs forever, right?
- SPSpeaker
Right. Okay.
- SPSpeaker
'Cause it follows the account throughout the entire life cycle.
- SPSpeaker
Cool.
- SPSpeaker
And so it both builds context on the account and then learns, picks up these other concepts that get-
- SPSpeaker
Right
- SPSpeaker
... that can get rolled up.
- SPSpeaker
Got it.
- SPSpeaker
So that's one part of it. And the second part is, even while doing cross-account work, you'll learn about these, like, cross-account concepts, so like how forecasting works or how, like, other, other processes within an org work. Um, and so that's like a second class of memory that's less relevant maybe on the individual account but is more relevant within the context of this, like, Watchtower product.
- SPSpeaker
Right.
- SPSpeaker
And so it-- we use both, right?
- SPSpeaker
Right.
- SPSpeaker
And so e- on every level, right? You can think about almost this, like, two by two where it's like, you know, some things are you learn-- you have memory on an account and across accounts, you have memory on a user level and an org level, and so we kind of learn all of those things in different ways-
- SPSpeaker
Yeah
- SPSpeaker
... um, and then use them to enhance all different parts of the product.
- 10:36 – 12:12
Sandboxing in practice: securing source access and supporting varied sessions
- SPSpeaker
Got it. Okay. Yeah, I wanna, I wanna talk to you a little bit more about that, but I wanna talk to Todd first. Um, so yeah, Todd, we were talking about sandboxing-
- SPSpeaker
Correct
- SPSpeaker
... and how you're using it. So I think this is a really, really cool use case for sandboxing, so.
- SPSpeaker
Yeah. Well, as I shared in the description of what we're doing-
- SPSpeaker
Yeah
- SPSpeaker
... we're actually snapshotting people's source code. In general, that should be sandboxed. [laughs] As you can imagine, if people interacting with an agent, and you get access to someone's source code, you can potentially get secrets. There's just a lot of, like, interesting things. Then you also have to be-
- SPSpeaker
Yeah
- SPSpeaker
... very thoughtful about sort, what sort of tools are exposed-
- SPSpeaker
Right
- SPSpeaker
... that's interacting with that source code as well, what sort of data is sent to it, and then what sort of action's happening. Like, at the end, like, often the result of an interaction is generating a, you know, essentially generating a pull request, you know, pushing it to the actual repository. So yeah, I think, I think that was the piece where we'd, we'd spent some time rolling our own, you know, but you're just, like, covering every single edge case-
- SPSpeaker
Yeah
- SPSpeaker
... and doing all that work. It's not core to our value prop.
- SPSpeaker
Sure.
- SPSpeaker
It just needs to work. [laughs]
- SPSpeaker
Yeah.
- SPSpeaker
It needs to be safe and secure. It needs to make sure, frankly, everyone from small businesses to large enterprises will trust it. And I think we were able to get there pretty quickly with, with Managed Agents. We also have, like, a diversity of, of sessions. We have overnight sort of batch systems which are looking for issues at night. Like-
- SPSpeaker
Yeah
- SPSpeaker
... if you're trying to look at large scale funnel changes or other things, you don't wanna do that in real time.
- SPSpeaker
Right.
- SPSpeaker
You wanna do that sort of in a cron process. But then we also have sessions where if someone's asking a question, we may wanna, like, quickly look at the source code to understand the source of truth, to then answer the question more accurately. All those are happening in Managed Agents right now.
- 12:12 – 14:14
Build vs buy: core competency, speed, and early-stage iteration
- SPSpeaker
Cool. So it is interesting, we have, you know, three different examples of managed agents, managed agent with sandboxing, managed agents with outcomes, managed agents with memory. So how do you think about the decision to roll your own harness versus use a managed solution like Managed Agents?
- SPSpeaker
I think all of it comes back to whether that's part of the core competency of the feature that you're building. And so, for example, when we're building new features like briefs, that's something where the harness itself is not a core competency of that feature. The thing that really matters is are we delivering the end user the value that they need? Um, whereas for other types of things, for example, if we were thinking about building a fully voice-controlled Jarvis system, there, the harness itself might be the core competency of what we build, and it might allow us to deliver completely new capabilities. And so it's a question of basically control versus cost and time, right?
- SPSpeaker
Yeah.
- SPSpeaker
So yeah, it, it really comes back to if, if you don't need to roll your own infrastructure for it because it's not the thing that defines the quality of what you're building, choose to buy, and in the cases where it is, build it yourself.
- SPSpeaker
And Todd, you'd mentioned kind of a similar point about when you're in this kind of regime of rapid experimentation, exploration, like speed is what really matters.
- SPSpeaker
Yeah, in the early days of any product, you're trying to validate product market fit, you're trying to validate certain core features that make your product special. Like all of us are solving some hard problem and trying to solve it in sort of a 10X better way than traditional solutions. You don't need to roll your own infrastructure to solve those problems. Like n-none of us in this, this roundtable, we're, we're not infrastructure folks. Now, there may be one day when the problem we're solving necessitates building our own infrastructure because we need a little more fine-grained control, maybe cost in a certain area we can't create-- get to where we want it from an operating perspective. But in the early days, you wanna innovate, you wanna get features in front of customers fast. You wanna be able to change really, really quickly. And for that, like this is a perfect solution to make sure that you're focusing on the right things for your team. So it's, you know, not for everyone, but I think if you're in the startup phase, yeah, absolutely, it's the right way of thinking about and prioritizing your time.
- 14:14 – 17:06
Rapid shipping timelines and what teams actually tune during iteration
- SPSpeaker
And you mentioned for you is it was like two weeks from kind of experiment, experimentation with this to shipping something.
- SPSpeaker
Yeah, and look, we had, we'd had a solution obviously running that was our own homegrown solution for a while, but we started running into issues with it. And like, like [chuckles] we all know shipping, shipping something, easy, but once you start seeing a set of issues, we started to understand, okay, we have to rebuild this whole thing because it's not where we want it. Then we baked it off against, let's just use managed agents, and it was so fast to get up and running.
- SPSpeaker
Right.
- SPSpeaker
And it covered a lot of the edge cases that we were hoping to cover ourselves. Just made a lot of sense. So yeah, I mean, within a few days, we had working infrastructure. Within a few weeks, we had the whole product rolled over to it, and that's pretty compelling.
- SPSpeaker
How about for you? What was the timeframe for you kind of ramping up on Managed Agents for the, for the meeting assistant in particular?
- SPSpeaker
Yeah, I mean, it allowed us to build a first version of the feature in a day and then ramp up with like 100 to 1,000X scale up in the user base over the course of a few weeks within, again, a couple more days of iteration on that part of the system. The primary thing that required work was figuring out exactly what a user wants out of the feature, and that required a bunch of cycles of build it, dogfood it, test it with other people-
- SPSpeaker
Yeah
- SPSpeaker
... and then iterate that. But even that could be done over the course of two or three weeks.
- SPSpeaker
During the iteration phase, what services w- did you find you're modifying a lot? Was it like the prompts, the skills, tools?
- SPSpeaker
All of those. And then the orchestration of the loop really is like, for example, when are you doing search, then retrieval of different things, then ranking of the relevant parts of it, then what do you surface back to the user, then how do you orchestrate that across different sources, right? And so that also process of what's the most effective way to gather the information that you need to drive that outcome is in the gamble of the kind of thing that we would iterate on.
- SPSpeaker
Makes sense. And how about you with Watchtower? Kind of what was like the time to kind of ramping that up and, and getting it deployed?
- SPSpeaker
It was like two weeks from coming up with the thing to shipping it to customers. And like dogfood somewhere, maybe like a weekend or something.
- SPSpeaker
So similar kind of ver-
- SPSpeaker
Yeah, super fast.
- SPSpeaker
Yeah.
- SPSpeaker
Yeah.
- SPSpeaker
And within that regime of rapid iteration, what are the things, what are the levers you, you feel like you're, you're tuning like very, very frequently? It's obviously things like skills, prompts, tools. Like what are the configurable surfaces that you found to be most useful?
- SPSpeaker
It's all of those, right?
- SPSpeaker
Okay. Yeah.
- SPSpeaker
It's all of those without having to worry about handling things like compaction ourselves, all of the other aspects of building out the harness. Um, because the question that actually comes up is, depending on the kind of meeting you're entering, what's the most important information to present to the user? And so if we don't have to worry about the rest of it-
- SPSpeaker
Right
- SPSpeaker
... we can literally just iterate on that question, which comes back to the outcome, right? Like, does the outcomes feature allow us to guarantee that for this kind of a conversation, we've assessed the most important information and structured it that way, and then rapidly iterate towards it, right?
- SPSpeaker
Yeah.
- 17:06 – 27:26
Evals in the real world: from vibes-based to user-driven eval sets (and hard cases)
- SPSpeaker
And I think I remember reading something about Claude Code in the early days where like in the very, very early days, Claude Code didn't have evals.
- SPSpeaker
Mm, yeah.
- SPSpeaker
Uh, and you can iterate your way to-
- SPSpeaker
Vibes based.
- SPSpeaker
Yeah. It's, you can iterate your way to getting a thing to be sticky, vibes based, and then the moment it's sticky, then you have to have evals. Like you'll be really, really sad as a person-
- SPSpeaker
Mm
- SPSpeaker
... if you don't have evals and a systematic way to iterate towards it.
- SPSpeaker
Yeah.
- SPSpeaker
But can you figure-- like if the thing's not sticky, like no amount of evaluation is gonna make the thing sticky.
- SPSpeaker
Yeah.
- SPSpeaker
Right? You gotta figure out how to make it sticky first.
- SPSpeaker
Yeah. It's a good clarification. So kind of in this regime where, um, you're doing very rapid iteration, using a managed harness is quite nice because it's giving you managed infrastructure. It's things like prompt caching are just solved.
- SPSpeaker
Exactly.
- SPSpeaker
Um, although it's very important for cost.
- SPSpeaker
For sure.
- SPSpeaker
Tell me about how this plays into memory. So memory is one that I've, I've actually thought about a lot and worked on quite a bit. It can be quite tricky to get right, and so for Watchtower, you, you're using memory manage in Managed Agents. Tell me about how you're using it and how you've configured it. What are the configurable surfaces for memory you think, that you think are important? Um, yeah, maybe walk me through that a little bit.
- SPSpeaker
So there are parts of memory that are very like core, and we need a lot of customization over, and so that's like the memory on every account. There's a lot of like- I, I, you know, there, there are a lot of levers we need to pull there to make that really stable, r- it, it lasts forever. It's, like, pretty detailed on every account. It can get corrupted over time.
- SPSpeaker
Yeah.
- SPSpeaker
And so there are a lot of things that we do. Uh, and so that, that we, like, manage ourselves, right?
- SPSpeaker
Okay.
- SPSpeaker
'Cause that we, we have to index it in certain ways to make it efficiently queryable. And so we do a lot of work on that to, like, kind of make sure that that's really good.
- SPSpeaker
Got it.
- SPSpeaker
But then there's all this other memory on, like, you know, what your different fields mean and, like, how your org works and, like, user preferences and, like, all these sorts of things that are less core, right? Like, it's, it's, it's like they need to work and, and they need to be good, but they're not... Like, they don't need to work any differently for us than they need to work for other people. And so for us, it just, like, the, the managed memory just unblocks a lot of that. And similarly, like, part of it is just being able to iterate and get this to, to be a valuable product for people. But even longer term, right, I think it's like the... For us, a, a large part of, like, how we view our job is to, like, make tokens useful for salespeople. In our view, like, a lot of that just comes from the actual product, and like-
- SPSpeaker
Yeah
- SPSpeaker
... the product experience and the surface that you build around it and so on. And so it also lets us both, like, get it off the ground. So we built this thing in two weeks, and so it lets us, let us, like, get it off the ground pretty quickly, um, and make it good. But then also, like, longer term, we just, we haven't rebuilt it or felt the need to go off of Managed Agents, and it's been, you know, a few months because we're just focusing... There's, like, so much on top of, like, product surface that we can build that's valuable, um, that I think just creates a lot more value for-
- SPSpeaker
Cool
- SPSpeaker
... for customers. So that's how we think about it.
- SPSpeaker
Yeah, yeah. And, and just to clarify slightly, so y- you have every agent, uh, has memory, and that's what you, you internalize that.
- SPSpeaker
That's right.
- SPSpeaker
You- these agents are built-
- 27:26 – 34:41
Model migrations, then cost and attribution: where managed agents help and constrain
- SPSpeaker
That's, that's not too tractable. How do you guys feel about model migrations? So Opus drops, what do you do? Like, how do you... You know, how do you think about updating your prompts if you think that's necessary? How do you think about testing new models? Are there things you think that could be better communicated maybe from us in term- in terms of how to use models? Like, we have that prompting best practices.
- SPSpeaker
I think it's mostly that. Like, if you have a good eval suite, then you do the model migration, hopefully the way that you've done your prompting has not done historically too much hyper-optimization to a specific model family.
- SPSpeaker
Yeah.
- SPSpeaker
What I've found, though, is a lot of new models at different points have different gotcha scenarios where they do bad things in different ways. The whole five series of models tends to produce writing in a way that has actually a lot more AI telltale signs-
- SPSpeaker
Mm.
- SPSpeaker
-than previous series of models.
- SPSpeaker
More em dashes.
- SPSpeaker
More, yeah, more of all of that kind of-
- SPSpeaker
[laughs] More of that, yeah.
- SPSpeaker
-uh, the, the sentence structures that are just like-
- SPSpeaker
Yeah, yeah. Classic AI tell.
- SPSpeaker
-unreadable, unreadable sentence structures.
- SPSpeaker
Yeah.
- SPSpeaker
So if there are things where there are known failure modes of the new category of models that you're trying to hedge against versus trying to optimize your prompts to, like, get the last one percent out of the model family, like, I don't care that much about the latter thing-
- SPSpeaker
Yeah.
- SPSpeaker
-because that'll keep improving with each generation. But if, uh, a new model introduces new failure modes that are specific to it that you have to avoid, that's probably the most valuable to us.
- SPSpeaker
A few of you were mentioning this point about cost and attribution as being, like, one of the, the major challenges and pain points broadly when, when building agents. Um, tell me about that.
- SPSpeaker
Yeah. I mean, I think, uh, nobody working on agents right now is not worried about cost to some degree.
- SPSpeaker
Yeah.
- SPSpeaker
And, well, if you're fully not worried about cost, you're probably not gonna build a thing that scales long term-
- SPSpeaker
Yeah.
- SPSpeaker
-in terms of a sustainable business structure, right?
- SPSpeaker
Yeah.
- SPSpeaker
I think one of the questions that often comes up for us internally of how long do we rely on Claude Managed Agents is when does the cost trade-off versus value trade-off of control for doing things in-house-
- SPSpeaker
Yeah.
- SPSpeaker
-mean that we should do things in-house?
- SPSpeaker
Sure.
- SPSpeaker
And I think, um, on the surface, the markup of Managed Agents is pretty low in terms of the amount of CPU overhead that you're paying for for running the infrastructure.
- SPSpeaker
Yeah.
- SPSpeaker
For example, you can't run batch mode that easily today with it, and so if we're running things twenty-four hours in advance, um, we could probably save fifty to seventy-five percent on cost and, and make that possible. And so today for us, it still comes back to that question of if we have a lot more value we know we can create for our users in the short to medium term, cost optimization is something where it's be mature, be mindful, and spend well, and then come back to actually optimizing all of the details over time. And hopefully by that point, you will have built features for us to do all those things directly-
Episode duration: 34:41
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode hm8NzEd5io0
