By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
ClaudeBuilding secure agents for knowledge work
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
Great conversations run long. Their best ideas don't have to.
uListen is free on iPhone, iPad, Android, Chrome, and Claude.
See howTranscript
40 min read · 7,758 wordsIntro
Katelyn Lesse 0:00 [upbeat music]
How AI is reshaping workflows: from benchmarks to “extending” knowledge workers
Katelyn Lesse 0:07 Super excited to be chatting with you guys. I have said this to you before, but I think you guys are some of the most AGI-pilled people out there with the things that you build and the way that you talk about what's happening in AI, and so very exciting for me to get to spend some time chatting with you guys about what's going on and what you're building. So maybe actually just to start, I would love to talk about the fact that the way everybody is doing work is changing so much right now, and that means the tools that people need to be able to use to get work done are changing a lot. Um, how do you guys think about that?
Dan Shipper 0:36 We think about it a lot. Um, and it's, it's one of those, one of those questions that I, I-- it's on everybody's mind. Everybody is super afraid of it, or they're like, "Oh, it's gonna be a utopia, and, like, no one's gonna have to work at all." The space we try to occupy is, "Well, why don't we just use the models and try to figure it out?" What ends up happening is you have to reinvent your workflow from scratch, and the only way to do that is to, like, try it on the problems that you're, you're actually working on. And I think for us, we've been seeing over the last couple years, you, you, like, watch all the benchmark progress, and when benchmark progress happens, you're like, the benchmarks are sort of measuring is it, is it gonna take my job away?
Katelyn Lesse 1:13 Yeah.
Dan Shipper 1:13 Like, the, the higher it gets on a benchmark. And our perspective is if you use these tools well, they, uh, are good at extending you and your power and your influence, and there needs to be ways to both measure that and to help people understand, uh, how to do that, because yes, if you look at benchmarks, they can do pre-AI tasks, like, at or better than humans in certain cases. But if you're thinking about how do I use this for my work-
Katelyn Lesse 1:39 Mm-hmm
Dan Shipper 1:39 ... that, like, opens up a whole se-set of options. We start to see that in engineering first because there's verifiable rewards in programming, and so y- now you have, like, 10X, 100X engineers who are orchestrating teams of agents to pull forward their roadmap by, like, two years.
Katelyn Lesse 1:55 Yeah.
Dan Shipper 1:56 And I think that is coming to the rest of knowledge work, but, uh, it's taken longer, uh, A, because it's, it's just harder to know whether or not it's doing a good job.
Katelyn Lesse 2:05 Mm-hmm.
Dan Shipper 2:05 And B, knowledge work, it's a different way of thinking that maybe developers are a l- a little bit more used to.
Katelyn Lesse 2:11 Yeah.
Dan Shipper 2:12 Um, but we think that's coming. We're trying to, we're trying to, uh, both discover it and disseminate it so everyone can work that way.
What Every is: media + tools to keep companies at the edge of AI
Katelyn Lesse 2:18 So for people who don't know, I'm curious, uh, what is Every? Tell us about what you guys do.
Dan Shipper 2:23 Every is a media and technology company. We build the only subscription you need to stay at the edge of AI. So for our subscribers, we, uh, do a series of education and equipment that helps them work at the edge. Uh, on the education side, we have everything from a newsletter to courses. When new models come out, we do vibe checks. And on the equipment side, now we have an agent that allows you to work in the way that we do in public with your company.
Why build the Every Agent: spreading AI workflows across a company in public
Katelyn Lesse 2:47 Okay, so, um, you guys built an AI coworker, um, to help solve this problem. Tell us about it. What were your goals? How's it going?
Dan Shipper 2:54 So we built the Every Agent. It is an AI coworker. It's an agent that is the easiest way to AI-pill your whole company. We started basically, like, in the OpenClaude craze. We started using OpenClaude, and we were like, "Oh my God, this is the coolest thing," and it spread throughout the whole company. And then-
Katelyn Lesse 3:10 You had your Mac Minis.
Dan Shipper 3:10 We had all the Mac Minis.
Katelyn Lesse 3:12 Yeah.
Dan Shipper 3:12 Um, and then unfortunately, all the OpenClaudes died off because, like, they're just really hard to maintain.
Katelyn Lesse 3:16 Yeah.
Dan Shipper 3:16 And we started just using one company agent, and we were like, "Wow, this is actually really, really great," and, and noticed how easy it was to spread workflows that we discovered throughout the whole company because our job inside of Every is to work at the edge, but you'd be surprised how hard that is to make working at the edge even across the entire company.
Katelyn Lesse 3:33 Yeah.
Dan Shipper 3:33 And so that started to happen. We started to see these workflows start to spread. What we've also realized is there's a certain type of person that loves Every who we've been calling an AI ambassador-
Katelyn Lesse 3:42 Mm-hmm
Dan Shipper 3:43 ... who is, like, uh, has that feeling about AI where they're like, "Oh my God, I have this Codex set up or this Claude set up or whatever that is, uh, absolutely 10X-ing the way that I do my work, and I can't imagine working without it. But none of my coworkers have any idea what this is, and I'm trying to, like, explain to them all the time how they should use it, but they're, like, tuning me out." And the beauty of a Slack agent, the Every Agent in particular, is that you can show instead of tell-
Katelyn Lesse 4:05 Yeah
Dan Shipper 4:05 ... how to use this stuff to actually do really, really great work because you're doing it in Slack. You're prompting in public instead of in private. And so we built the Every Agent to have all the things you would expect in an, in an agentic coworker. You know, it can help you do recurring tasks like gather reports or do social posts or-- and any of the kinds of things you might expect, but it happens inside of your company Slack-
Katelyn Lesse 4:26 Yeah
Dan Shipper 4:26 ... so that your coworkers just automatically pick it up, and it starts to spread your best practices. We try to empower the ambassadors as much as we can with a tool like this.
High-leverage ‘small tasks’ example: automating open enrollment guidance
Katelyn Lesse 4:33 Awesome. What's one of your favorite examples of something you've seen somebody do with the Every Agent that maybe, maybe they could do it before, it would've taken them longer, maybe they couldn't have even really accomplished that thing before?
Dan Shipper 4:44 I really think that having an-- having the Every Agent, having an agent in your Slack, there's-- I mean, there's the sort of like crazy top-end stuff, but the, the stuff that I wanna focus on is actually there's, there are all these small little things that you're doing day-to-day that take up a lot of people's time and, and having a company agent makes really easy. So a simple example, we just had open enrollment. We did the entire open enrollment process through an agent, uh, through the Every Agent. The open enrollment, I-I'm sure, I'm sure you know, you get like a PDF that's like, "Here are all these plans," and, like-
Katelyn Lesse 5:14 Yeah
Dan Shipper 5:14 ... doing the algebra to figure out the plans is actually very hard, and as part of that process, the Every Agent both knew all the plans and knew who you were and, um, and helped you kind of figure out which one you should select for your particular situation, and that's something that would've been very hard to do and coordinate without a central company agent, and the Every Agent makes it really easy for, A, leaders of the company to create a set of skills that, that work in that way, and then B, get that disseminated to everyone in the org and make sure that the open enrollment process actually happens smoothly.
Katelyn Lesse 5:46 Yeah, like the amount of time you probably saved with not every single person pasting separately into Claude-
Dan Shipper 5:50 Oh my God
Katelyn Lesse 5:50 ... like, "This is the PDF."
Dan Shipper 5:51 Yeah.
Katelyn Lesse 5:52 "What do I pick?"
Dan Shipper 5:52 Yeah.
Katelyn Lesse 5:52 That saves you a lot of time.
Dan Shipper 5:53 And, and honestly, like, that's some- even for me, like, I-- as I-- anytime I get one of those things, I just, like, call my dad, and he's like-
Katelyn Lesse 5:59 [chuckles]
Dan Shipper 5:59 ... "I can't believe you're, like, thirty-five. I can't believe I have to still explain this to you."
Katelyn Lesse 6:03 [chuckles]
Dan Shipper 6:03 'Cause it is actually really complicated, and yes, I could just individually paste it into Claude, but, but knowing that this is coming from the company and the company has thought about for each person, like, "Here's what you might wanna pick," it really helps me feel confident in my choice.
Katelyn Lesse 6:17 Yeah, now you can call your dad for more fun reasons.
Dan Shipper 6:19 Yes, exactly. Sorry, Dad, I automated that task. [laughing] Um, it's not one that, not one that he relishes.
Katelyn Lesse 6:24 He didn't like it at all. [laughing]
Dan Shipper 6:26 [laughing]
Editorial automation that helps experts scale: “Kate Bench” top edits
Willie Williams 6:26 Well, we also, like, we have a lot of pretty rote tasks that go along pretty with the editorial arm. And we've spent a lot of time, like, building out something we call Kate Bench, which is named after our editor-in-chief, which is mostly like how do we do top edits before things go out? And there's still a question of like when to trigger these top edits, and so it's like we flood Kate's inbox. It's now much easier to just be- to tag the agent in Slack and say like, "Hey, just review this." And the nice part is Kate can see that-
Katelyn Lesse 6:51 Mm-hmm
Willie Williams 6:51 ... and she can kind of like judge like, okay, is the agent, like, doing the job or not?
Katelyn Lesse 6:55 Yeah.
Dan Shipper 6:55 Yeah, this is, this is a really important thing that I think, I think Kate Bench gives or gives you a, a, an, a view into how this stuff is gonna be used in knowledge work going forward. So l- like Willie said, Kate's our editor-in-chief. Like, I've been trying to automate her for many years-
Katelyn Lesse 7:09 [laughs]
Dan Shipper 7:09 ... but like, but, like, in the most loving way possible.
Katelyn Lesse 7:12 I was gonna say-
Dan Shipper 7:12 Um-
Katelyn Lesse 7:12 ... maybe don't have her watch. [laughs]
Dan Shipper 7:13 Yeah, like, like, uh, automating I think sounds scary-
Katelyn Lesse 7:16 Yeah
Dan Shipper 7:16 ... but when you actually see it in practice, you're like, "Oh, this is actually helpful for her"-
Katelyn Lesse 7:19 Yeah, for sure
Dan Shipper 7:20 ... because we publish so much. Um, and when, when Kate started, there was like four people at Every. She was like the fourth person, and now there's, now there's 30 people. Um, and she has incredible taste for copy edits, for like making sure that every single period, every single verb, every single thing is in the right place. But, um, we have 10 times the amount of things going out today than we did when she started, but she still has to do it all.
Katelyn Lesse 7:42 Yeah.
Dan Shipper 7:42 So what I did was I just, uh, took like 30,000 of her historical edits and turned it into a skill that is in the Every Agent. So now every time we are publishing something, whether it's a piece or a landing page or an email, anyone in the organization can just say like, "@Every," like, "do a Kate Pass on this," and it will take her taste and apply it, like literally make edits in the Google Doc, like with suggested changes as her, and then, uh, they can accept and reject it. And what she does is she just goes in and looks at it, and it has most of the things that she would do and even catches things she might not. She has to spend less time on each piece 'cause-
Katelyn Lesse 8:17 Yeah
Dan Shipper 8:17 ... it has done a lot of the work. And then what we do is we track what is the extra work that she added to this-
Katelyn Lesse 8:23 Mm
Dan Shipper 8:23 ... and then we compound it back into the skill so it gets better and better over time.
Katelyn Lesse 8:27 Oh, nice.
Reframing automation: compounding an expert’s impact instead of replacing them
Dan Shipper 8:27 And she gets to see a graph of here's how, here's how much time it's saving you, and here's also a rule we might wanna add to this so that it catches more things you don't, you didn't necessarily know about. And I think what that does is it, it flips the sort of like automation debate from, like, is it gonna take away my job to actually I'm an expert in an organization. My time's taken up all day by having people ask me to do things for them that only I can do.
Katelyn Lesse 8:50 Yeah.
Dan Shipper 8:51 And as the organization scales, in order to scale my impact, previously I would've had to spend more time. And if you have an agent that compounds like the Every Agent, the opportunity for experts inside of organizations is you can spread your impact and spread your, the way that you do work without having to spend more time because you're working on a system that does the work for you instead of doing all the work yourself.
Katelyn Lesse 9:12 Yeah, it's doubly valuable. Kate gets more time to do other things, um, maybe the hardest copy edits-
Dan Shipper 9:17 [laughs]
Katelyn Lesse 9:18 ... so Kate, and then everybody else gets to take advantage. I actually would love to talk about, um, Kate Bench-
Dan Shipper 9:23 Mm
Katelyn Lesse 9:23 ... for an ex- for an example. You guys talk a lot about being really, um, you know, bench pilled, right?
Dan Shipper 9:30 Mm.
Katelyn Lesse 9:30 And being very on top of here's how we do some, like, really cool use case, and then measure how well we're performing on that really cool use case. Tell us more about that mentality-
Dan Shipper 9:40 Yeah
Katelyn Lesse 9:40 ... and how, like, as you have people going out there and starting to use the Every Agent, they can kind of bring this mentality in on, like, continuous improvement.
From vibe checks to personalized evals: measuring AI by your real work
Dan Shipper 9:47 Yes. We, we have been- really started to get, to get good at this, get, get good at evals and start trying to measure these things. I think, you know, we started with everyone who's in AI and, and follows us closely knows that that doesn't really mean that much.
Katelyn Lesse 10:00 Yeah.
Dan Shipper 10:01 Um, and the question is not how good is it at a generic benchmark, it is how good is it at the kind of work I do?
Katelyn Lesse 10:06 Right.
Dan Shipper 10:06 And that's a whole- that's a totally different way of thinking about, um, whether AI is good because it's much more personal, and that's where we started with our vibe checks, where before a new model comes out, we get access to it early, and we actually test it by hand across a lot of our workflows, and then on the day it launches, we can tell you, "Here's the kind of work we do. Here's our taste. Here's what we think of this model." And people rely on that to be like, "Well, I trust Every and I trust their taste," or like, "There's a certain writer at Every that I know has similar taste to me, so like, if they like this model, I'll try it." Problem is that that's very manual and takes a long time, and so what we've started to do is capture those tasks that we do. So like anytime one of us is doing work with a model day to day and we, we notice, ooh, it did something I like, or ooh, it did something I didn't like, we can sort of capture that in, in one place, and then talk to this platform that we've built that will eventually be part of the Every Agent, talk to this platform that we built about why we liked it and why we didn't, and that turns it into this set of checks, which are basically like unit tests or rules for here's the kind of thing you look for. So for Kate Pass or Kate Bench, it would be is the headline sentence case would be an example. And, um, what, what that allows us to do is run many different models on the same scenario and then be, be able to see for our real work, did this model do a good job or not according to my taste? We've started to be able to add this sort of like quantified layer to vibe checks that I, I think looks like a benchmark, but actually measures things that you care about.
Why Slack won: consolidating into one shared agent that everyone improves
Katelyn Lesse 11:31 Awesome. Um, going back to form factor a bit, you talked about obviously being in Slack is really powerful. All the knowledge workers can have really easy access to this. Um, how did you land on Slack? How are you thinking about that form factor?
Willie Williams 11:44 Well, we s- we actually started with sort of a model of, uh, every one agent per person, you know, available in whatever surface you really wanted, uh, that agent to be there, if it was Telegram, if it was Slack. Um, and part of the reason we over time went from like a multi-agent setup to really a singular agent setup is we found that agents got better via the same dynamic the more you invested in them.
Katelyn Lesse 12:07 Yep.
Willie Williams 12:07 And so you had a lot of folks who were like, "Oh, I invest- I'm investing in my agent," but those gains weren't really like spreading out to the rest of the team. And the places where we, we really saw like agents being able to do work, real work, was when there were multiple people investing in them, making better skills, giving it better guidance, and it just made so much more sense for everyone to be investing in like a single agent rather than, uh, everyone having their own agent sort of spread around. And Slack was like- The very natural form factor for like where are we when we're having these conversations with individual agents and, and when we're doing, like, collaborative work. There's a lot of copy and paste.
Katelyn Lesse 12:43 Mm-hmm.
Willie Williams 12:43 Right? I'm like, I'm like, we're gonna have a conversation. I'm gonna, like, copy this over to my, like, Claude coworker, and I'm gonna copy the, the response back, and then you're gonna do that. And instead it's much easier to just have that conversation in the same place that the, your coworkers are and the same place your agent is.
Katelyn Lesse 12:58 Perfect and easily accessible. Let's talk a bit about how you built Everyagent.
Willie Williams 13:02 Okay.
Building on Claude Managed Agents: avoiding infra distractions and gaining primitives
Katelyn Lesse 13:02 So you built this product on Claude Managed Agents-
Willie Williams 13:04 Yeah
Katelyn Lesse 13:05 ... uh, which we're really excited about. Tell us just starting out, what were some of the main reasons why you chose to start building on Claude Managed Agents?
Willie Williams 13:11 Well, we learned a lot of pain- painful lessons with this first version of One Agent for Everyone.
Katelyn Lesse 13:15 Okay.
Willie Williams 13:15 Um, one of the main ones is, uh, infrastructure's hard, you know? Um, and a lot of what we want to focus on is like the future of work and like how to make a better product for like that future that we see coming. We don't necessarily want to spend that time like, how do we orchestrate sandboxes?
Katelyn Lesse 13:32 Yep.
Willie Williams 13:32 Right? And so we, [chuckles] we had this very painful arc where we're like, "Okay, we're going to, uh, we're gonna build one agent for everyone," and then it just got unwieldy very quickly because you're sort of like trying to improve a product and have a, have a strong, durable like infra layer underneath. And so w- we took those lessons and turned them into the Everyagent, and one of our first things was we don't want to be an infra team.
Katelyn Lesse 13:54 Mm-hmm.
Willie Williams 13:55 Right? We want to focus our energy in the place where we think, uh, the future is going. And so Claude Managed Agents was great. It wa- it was, it gave us, you know, it gave us sandboxes, it gave us memory, it gave us session control, all these primitives that really allowed us to, um, focus on the next layer of how do we do interaction design for, for an agent. You know, we can kind of just like shuffle it off.
Katelyn Lesse 14:17 Yeah.
Willie Williams 14:18 Just be like, okay, this is being taken care of. You know, I'm not getting alerted w- uh, when, when orchestrating.
Katelyn Lesse 14:23 I-
Willie Williams 14:23 Yeah, yeah. [laughs]
Katelyn Lesse 14:24 [laughs]
Dan Shipper 14:24 Thank you. Thank you for your service.
Willie Williams 14:25 Yeah, yeah, yeah, yeah. [laughs]
Dan Shipper 14:28 [laughs]
Willie Williams 14:28 Uh, and it's been great.
Katelyn Lesse 14:29 Nice.
Dan Shipper 14:29 Yeah. That's, that's one of those like classic things in startups where you're like, "Okay, we're gonna build an agent platform. Should we do the infra layer or not?" And you're like, "Oh, infra's not that hard anymore. Like, you can just... Like, we'll just have Claude spin up a bunch of servers and like manage the servers for us and like blah, blah, blah." I was saying that at least, and Willie was like, "No, no, it's, it's hard."
Willie Williams 14:46 [laughs] Yeah.
Dan Shipper 14:47 And I was like, "We're gonna, we're, we're going to try it." And then a month later I was like-
Willie Williams 14:50 Yeah.
Dan Shipper 14:50 Yeah.
Willie Williams 14:50 You're like-
Dan Shipper 14:51 [laughs] This is hard.
Katelyn Lesse 14:52 [laughs]
Dan Shipper 14:52 And yeah, Claude Managed Agents, it takes something that you don't know is hard, but when you do it, you're like, "This is so complicated," and it makes it very easy for us.
Katelyn Lesse 14:59 Nice. That's awesome. So there's infra management, that's obviously helpful. What are some of the other features? You mentioned memory.
Willie Williams 15:05 Mm-hmm.
Katelyn Lesse 15:05 Um, what are some of the things that you've used out of the box with Claude Managed Agents that maybe just helped you build agents that were more powerful than, uh, you know, you would get from just hand rolling a-
Willie Williams 15:15 Mm-hmm
Katelyn Lesse 15:15 ... a harness and going from there?
Willie Williams 15:17 I think the, the most honest answer is just it allows for the combination of primitives and allow us, allows us more time to do more experiments in the harder areas. So for example, like having the ability to isolate out we want this particular managed agent as sort of a, an edge, you know, like our staging our, our e- our frontier case, um, and being able to take, you know, the memories and the durable state that we have from production and copy it over and like run our experiments and like, okay, what happens in this scenario? What happens in this scenario? Like, how do we bring our, the token cost down for our customers? There are a million conversational edge cases that you're-
Katelyn Lesse 15:56 Yeah
Willie Williams 15:56 ... like you're trying to sort out, um, particularly, uh, around stuff that's sort of in the gray area. Like when you're building an AI coworker, there are a lot of things you take as an assumption for coworkers. Like, they're gonna make a, they're gonna make a jud- good judgment call.
Katelyn Lesse 16:09 Yeah.
Willie Williams 16:09 And it's hard to instruct models how to do that, and so you're, you're sort of running some combination of evals, but also like a, a vibe-based like does this feel like a real coworker or not sort of experiments, and being able to like in terms of a, a dev experience, be able to slice that off and run those experiments in isolation without having to impact our main production instance. But also running on the same infra as our main production instance, uh, speeds up the team and results in like a better product.
Security, isolation, and fast model updates: harness and model are now coupled
Katelyn Lesse 16:37 Yeah, that makes a lot of sense. And, uh, there's a lot going on right now in terms of security-
Willie Williams 16:42 Yeah
Katelyn Lesse 16:42 ... with agents and, um, a lot of energy is going into how do we make sure that agents are doing the things we want them to do-
Willie Williams 16:48 Yeah
Katelyn Lesse 16:48 ... and not the things that we don't want them to do.
Willie Williams 16:50 Yeah.
Katelyn Lesse 16:50 How do Claude Managed Agents help you think about security with Everyagent?
Willie Williams 16:54 Yeah. A, a lot of it is just the built-in isolation. It, it makes it very easy to, um, isolate out, uh, the sandbox, the tool calls, like what's, what's happening agent in the loop, um, and what are the environments and where are the memories and like we can, we can, because we can control all that, we can make choices around like, uh, who accesses what when. You know, session management is great for this, and that really gives us the, the nuanced tools to say like, okay, you know, if Dan requests something, does he get, uh, the same thing that I would get if I requested it?
Katelyn Lesse 17:27 Mm-hmm.
Willie Williams 17:27 Right? And underneath the hood there's, there's a lot of playing with the primitives of like what memories do we attach to that request? Like is the, are the sandboxes the same or different? Um, and so yeah, just having those primitives, uh, is just helpful from an engineering standpoint.
Dan Shipper 17:41 One thing that I do want to add, too, I think this relates to s- security, but it also relates to performance, which is there's not a clear separation between the model and the harness anymore.
Katelyn Lesse 17:50 Mm-hmm.
Willie Williams 17:51 Mm-hmm.
Dan Shipper 17:51 And a model that doesn't come with a computer that it knows how to use, like it's just not as performant.
Katelyn Lesse 17:58 Yeah.
Dan Shipper 17:59 And having all of that built in one place by one provider, uh, ensures that the model knows how to use that stuff really well, and you know, uh, the, the day we're filming this, there's a new, a new, new model launch from you guys.
Willie Williams 18:12 Yeah.
Dan Shipper 18:12 And it's one of those things Willie, Willie was telling me, if you look at the prompting guide, there are some breaking changes-
Katelyn Lesse 18:17 Mm-hmm
Dan Shipper 18:17 ... that teams would have to like care about.
Katelyn Lesse 18:20 Yeah.
Dan Shipper 18:20 But in, in Claude Managed Agents, it's just like a one-line update.
Katelyn Lesse 18:23 Nice.
Dan Shipper 18:23 And that's the kind of thing that you get for free that I think is really powerful.
Katelyn Lesse 18:27 Yeah, that makes a lot of sense. Um, some of those breaking changes and just some of the how do we get the best out of Claude, um, and make the best use of all the things that it can do- As part of re-releasing the model, we're baking all those things-
Willie Williams 18:39 Mm-hmm
Identity & authorization model: separate coworker in Slack, acting on your behalf for tools
Katelyn Lesse 18:39 ... into the Claude Managed Agents harness, and so that's cool to hear. How did you think about, uh, auth and identity? Like, identity for the Everyagent, is it acting on behalf of humans? Is it acting on behalf of itself? Um, how did you think about that?
Willie Williams 18:54 Yeah, there's a lot of nuance to this because as soon as you sort of put an agent inside of a, a, you know, basically a company in-in-internally, people think of it like a human, right? And they ask it to do things that are, that fall again into these gray zones of like, well, a human would know that, like, there's no harm in me being able to access this, like, report, you know? Um, but that's, that's hard to sort of encode into an, a piece of agent logic. Um, and so we... But, but that we settled on that as our North Star. Let-let's try and make it act as close to a human as it can within, within s- you know, Slack. And the way we do it is when it's in Slack, it acts as a separate entity, where you're interacting with it, and, uh, like, underneath the hood it has y- you know, isolated memories and I- an understanding of who you are and what you have access to within the, the, within the company. It's a conversation between two separate entities. When you ask it to go out and grab something via, like, a tool call or a connection or whatnot, it, for the most part, acts on your behalf, where it's, it's as an extension of you when it goes into, um, go grab a document or look up something. Um, and we found that model to be pretty accurate to the way people want to use it, um, where they're fine with this, this boundary around sort of like internal versus external.
Continuous improvement through ‘taste nudges’ + the hiring analogy for agents
Katelyn Lesse 20:09 Nice. That makes a lot of sense. Um, we touched a little bit on continuous improvement, memory. Um, how are you guys thinking about that? Like, what is most important to you when it comes to the agent can do certain things today really well, maybe not as well as you want. Um, how are you thinking about get that agent really great over time at all of those things better and better?
Willie Williams 20:31 Yeah. The-- One of the nice things is the agent is able to see all of the little moments where you are sort of adding your taste into requests. These little nudges of like, ah, this... You know, you didn't write this well. Uh, one today was like, it told a bad joke. I was like, "This is not a great joke. Give me another joke," you know? [laughs]
Dan Shipper 20:47 We take humor seriously.
Willie Williams 20:48 Yeah, yeah, yeah. [laughs]
Dan Shipper 20:50 Of course. [laughs]
Willie Williams 20:51 Um, and those little nudges are really the foundation of like how... what is your taste? Like, it's hard to describe your taste-
Katelyn Lesse 20:57 Mm-hmm
Willie Williams 20:57 ... uh, you know, all in one big go, but it's easy to take it from a thousand examples and sort of distill it out. And so those form the foundation of like, how do we build skills for, as Dan was saying, like particular jobs?
Katelyn Lesse 21:08 Mm-hmm.
Willie Williams 21:09 Right? If we're doing-- If you're a person who does a lot of editing, the way the agent gets better at editing is looking at your 30,000 edits and con- using that as the, the foundation for like, oh, this is, these, these, this is the, the skill, the taste I have when it comes to editing.
Katelyn Lesse 21:25 Mm-hmm.
Willie Williams 21:25 And, you know, you can think about this for like PowerPoint generation, you can think about this for email, you can think about this... It happens in coding as well, but, like, this is a new frontier for a lot of other non-coding knowledge work, where you're sort of accumulating these examples. Um, and the agent is really the repository for all of those.
Dan Shipper 21:40 I think a good way to, to think about this is... I've been talking about, a lot about benchmarks, but I think it's so, I think it's so important, and, and the way that we think about it is starting to shift, where think about hiring or, or using an agent as hiring for a job.
Katelyn Lesse 21:52 Mm-hmm.
Dan Shipper 21:52 If you're hiring a job candidate and you're looking at their SAT scores, that's a little bit like a benchmark score.
Katelyn Lesse 21:57 Yeah.
Dan Shipper 21:57 And directionally, that actually helps a lot. So if I'm hiring a job candidate and they got a 1600 on their SATs versus a 300, I would probably take the 1600 person.
Katelyn Lesse 22:06 Hope so. [laughs]
Dan Shipper 22:06 Probably. But if everyone got a 1600, which is where we are with, with these models, um, there's a lot more I need to do. What I actually want is not your SAT score. Uh, I want a re- a reference check. I want a work trial, right? And for, for these models, uh, when they come into your company on the first day, it's their first day of work. Like, they're, they're a new grad who's really smart-
Willie Williams 22:26 Mm-hmm
Dan Shipper 22:26 ... has studied a lot, but they don't know how you work.
Katelyn Lesse 22:28 Yeah.
Dan Shipper 22:28 And what you wanna try to do is create a system to, as they get experience working with you, to compound their knowledge of how you work and what you care about, so that on day 10 they're better than on day one. And what, what we are starting to build is a way to actually, uh, prove that. So like, to allow you to gather cases where the model did, did good work or didn't do good work, pull out your taste, and then allow, allow it to provably improve at those things over time. That won't be in the agent on day one, but I think that's where we're going, and I think generally where work is going, where-
Katelyn Lesse 23:02 Yeah
Dan Shipper 23:02 ... instead of doing each thing manually every single time, you're working on the system, and hopefully your agent, whether that's the Everyagent or Claude or, or any agent you're working with, enables you to do that, enables you to see how it's getting better.
Willie Williams 23:14 Mm-hmm.
What’s next: personal benchmarks, computer-use experiments, and social interaction design
Katelyn Lesse 23:14 Yeah. Makes a lot of sense. What are your next couple ideas for how you wanna improve the product?
Dan Shipper 23:18 It's a good question. [laughs] I, I think, uh, so the, the big one is this, this sort of like compounding system-
Katelyn Lesse 23:24 Yeah
Dan Shipper 23:24 ... um, that allows anyone to basically create their own little personal benchmark for here's how I work and what I care about, and what I think is good about this model's response versus another model's response, and allows you to see, uh, which one you might want to use over time, and allows you to build and optimize skills that are for you and based on your taste to do your work better over time, and allow that to spread throughout the org so that other people can use those skills and work like you can to enable more experts to get more work done without spending more time. So I think, I think that's, that's where we're going. We do tons and tons of experiments, so I have like, I have this little experiment called Hands, which is like a computer use agent on my, on my desktop that, like, the Everyagent can then go use my computer.
Katelyn Lesse 24:08 Oh, nice.
Dan Shipper 24:08 'Cause I, 'cause you're always getting-
Willie Williams 24:09 Yeah
Dan Shipper 24:09 ... like, requests where it's like, "Oh, can you do this thing?" And I'm just like, " @every use Hands" to, like, do it on my computer.
Katelyn Lesse 24:15 Nice.
Dan Shipper 24:15 And that's really cool. But yeah, we're experimenting with a lot of things.
Willie Williams 24:19 This is one of the, uh, like one of the number one questions we get from our customers is just like, how does this compare to sort of like working with Claude Code or Claude Coworker agent? And we see them as like very, um- Like, they work together really well because of experiments like Hans, where it's like sometimes you wanna be with coworkers and, and doing this thing and sort of doing it in public.
Katelyn Lesse 24:37 Yeah.
Willie Williams 24:37 And then sometimes you wanna just shift that work very seamlessly over to, like, "I just wanna do it alone." You know, just like you would, you know, as if you're a human.
Katelyn Lesse 24:44 Yeah.
Dan Shipper 24:44 I mostly wanna do it alone.
Willie Williams 24:46 Yeah, yeah. [laughs]
Dan Shipper 24:47 [laughs]
Willie Williams 24:47 Yeah. Um, and, and, and, and really it's this whole, you know, like everyone just sort of figuring it out, right? Like, where do I wanna spend my time with, with this sort of amplified intelligence? Um, and, and how do I go back if I don't? Like, like doing- like, for us, like doing context- how you do context management is both, like, gotten more invisible but also more visible as we've moved into, like... Personally, it's like I, I don't really think about context management anymore. It's- [laughs]
Dan Shipper 25:15 [laughs] Yeah, yeah, yeah. Not thank you so much.
Katelyn Lesse 25:16 [laughs]
Willie Williams 25:17 Um, uh, but with coworkers it, it really is. It's like, okay, we're trying to... Uh, uh, again, if you model it as a, as a coworker, it's like, oh, there was this conversation over here. Were you, were you part of it? And do you have, uh, enough of an understanding of human conversational norms to be like, you should, you should, you know, interject over here, or maybe you should take these two ideas from separate threads and, like, they're relevant even though one was a brainstorm and one was something a little bit more directional. And so a lot of, like, the interaction design and, like, personality design-
Katelyn Lesse 25:46 Mm-hmm
Willie Williams 25:46 ... I think is, like, a huge thing that's coming.
Dan Shipper 25:48 That is such a big thing-
Willie Williams 25:50 It's coming
Dan Shipper 25:50 ... 'cause Claude so far, and I think this is starting to change, but so far every time you- it's used to every time it sees a message, it's supposed to respond-
Katelyn Lesse 26:00 Mm
Dan Shipper 26:00 ... because you're prompting it-
Willie Williams 26:01 Yeah
Dan Shipper 26:01 ... one-on-one usually.
Katelyn Lesse 26:02 Yeah.
Dan Shipper 26:02 And Claude's in, like, a social context with other humans and probably other Claudes. There's a whole new set of norms that it doesn't necessarily know out of the box-
Katelyn Lesse 26:11 Yeah
Dan Shipper 26:12 ... that I'm sure you guys will, uh, start to improve, but we're also thinking about 'cause there's also different styles.
Willie Williams 26:17 Mm-hmm.
Dan Shipper 26:17 Like, do you wanna, do you want it to be an extrovert?
Willie Williams 26:19 Yeah.
Dan Shipper 26:19 Do you want it to be an introvert?
Katelyn Lesse 26:21 Yeah.
Dan Shipper 26:21 Like, exactly when and, and for what use cases would this agent interject versus not are all, like, really interesting-
Willie Williams 26:27 Mm-hmm
Dan Shipper 26:27 ... complex questions-
Willie Williams 26:28 Mm
Dan Shipper 26:28 ... that I think will be, uh, a lot better in six months to a year that we're all gonna have to work on making happen.
Katelyn Lesse 26:34 Yeah, and I love the idea that Every has a certain brand personality way of working-
Dan Shipper 26:39 Mm-hmm.
Willie Williams 26:39 Mm
Katelyn Lesse 26:39 ... and you're baking that into the Every Agent.
Dan Shipper 26:41 Yeah.
Katelyn Lesse 26:41 And so, um, you know, that personality will kind of come through-
Dan Shipper 26:45 Yeah
Katelyn Lesse 26:45 ... in the agent.
Dan Shipper 26:46 One of the, like, design, like, criteria for us was if you're working with the Every Agent, it should feel like you have someone who works at Every working with you now.
Willie Williams 26:56 Mm.
Katelyn Lesse 26:56 Hmm.
Dan Shipper 26:56 And that's, like, a really-
Willie Williams 26:57 Yeah
Dan Shipper 26:57 ... interesting question and a really interesting way to think about how to, like, designing a product.
Looking ahead + asks from Anthropic: fidelity, usability for knowledge workers, and personality knobs
Katelyn Lesse 27:01 Well, as very AGI pill people, I'm sure you're thinking a lot about where things go from here, where are we gonna be in three months, six months, a year, um, just in terms of how people are gonna do work, right?
Dan Shipper 27:13 Mm-hmm.
Katelyn Lesse 27:13 What do you think is gonna happen, and where do you think that Every Agent's gonna need to be? What are you gonna need to build into it so that it's ready for that moment of how work is gonna change?
Dan Shipper 27:22 It's a very good question, and I think it goes back to exactly what we were talking about in the beginning, where, um, you see developers starting to work in loops, starting to instead of, uh, you know, writing every line of code by hand, you're, you're, you're directing sometimes a whole organization of, of Claudes to do your work for you. Um, and your job is to make sure that even if you're not doing every unit of work, you've created a system that feels like it- it's an extension of you that... I've been thinking about words for this. I think one word is, like, fidelity. Like, the agents have fidelity-
Katelyn Lesse 27:51 Hmm. Yeah
Dan Shipper 27:52 ... for what you want, what you care about-
Katelyn Lesse 27:54 Love that
Dan Shipper 27:54 ... so that you don't have to be involved in every single thing. That's starting to be the case in how we do programming. Obviously, like, with s- with all the security stuff going on, like, sometimes it's not the case-
Katelyn Lesse 28:05 Yeah
Dan Shipper 28:05 ... and that's an interesting one.
Katelyn Lesse 28:06 Yeah.
Dan Shipper 28:07 It's also something that benchmarks don't really measure that well because they have- they're, by, by definition, they are more generic. They're not about you and what you want.
Katelyn Lesse 28:14 Right.
Dan Shipper 28:14 Um, but I think over the next year or so, uh, in order to take that next leap in, in knowledge work, um, we're going to have to make agents that have fidelity to what you want and what you care about, and I think what that will unlock is for individual knowledge workers, the same kind of thing that you see in coding where you're extracting yourself out of, out of the, uh, the, the loop. You're working on the loop. You're working on the system, and making sure the system operates according to what you care about and, and what your taste is and what your vision is, and you're intervening at certain, at certain times, just like a manager would, to make sure things are on track. And what that will do is allow you, A, as an individual person, to do way more than you could have before, and then, B, allow other people, whether it's in your organization or your clients, to use that, use your taste, do things like you would, without taking up more of your time.
Katelyn Lesse 29:06 Yeah, that makes a lot of sense. And what are you gonna need from us-
Dan Shipper 29:10 [laughs]
Katelyn Lesse 29:10 ... from Anthropic, from the Claude platform, in order to make Every Agent what it needs to be at that time?
Dan Shipper 29:16 We're starting to reach the point where successive gains in model intelligence are not necessarily able to be consumed-
Willie Williams 29:23 Mm-hmm
Dan Shipper 29:23 ... even by power users.
Willie Williams 29:25 Mm-hmm.
Dan Shipper 29:25 I think it's so amazing that there are in- these advances in frontier math or, or now, like, now biology, and those are super, super important, but, um, I don't want you guys to forget about us knowledge workers.
Katelyn Lesse 29:35 Yeah. [laughs]
Dan Shipper 29:36 Um, because I truly do think ev- uh, e- even as those gains in those other areas are happening, there's s- there's still so much low-hanging fruit-
Katelyn Lesse 29:44 Yeah
Dan Shipper 29:44 ... in terms of knowledge work. Like, when I get a response back from Claude, like, do I understand it?
Katelyn Lesse 29:48 Yeah.
Dan Shipper 29:48 Does it reference things that, like, are at my level of knowledge-
Willie Williams 29:51 Mm
Dan Shipper 29:51 ... or that I, I would remember? All the, like, all those kinds of little things I think are so critically important to making a tool that people can use every day-
Katelyn Lesse 29:59 Yeah
Dan Shipper 29:59 ... that I think you will improve on those dimensions, and I expect if you do, there will be a lot m- more that opens up that al- allows knowledge workers to get more done.
Katelyn Lesse 30:08 Yeah, exactly. And it's, it's the intelligence, but it's also the speed-
Dan Shipper 30:11 Yeah
Katelyn Lesse 30:11 ... and the cost-
Dan Shipper 30:12 Yeah
Katelyn Lesse 30:12 ... and all these things that are paired.
Willie Williams 30:13 And the, I was gonna say, the personality. Uh, we do a lot of thinking and work around, like, personality design. Like-
Katelyn Lesse 30:18 Yeah
Willie Williams 30:18 ... how do we, particularly for us, like, keeping it because we're, we sit between managed agents and, and our users.
Katelyn Lesse 30:24 Yeah.
Willie Williams 30:24 Like, how do we keep the personality, like, both consistent but also configurable?
Katelyn Lesse 30:27 Yeah.
Willie Williams 30:27 Any tooling there or any little knobs that we could turn to be like, okay, you know, one, one customer really wants an aggressive personality, one customer really wants a- Patient personality. It's very standard in the way we can kind of like turn Claude into that.
Katelyn Lesse 30:42 Yeah. Nice. I'm excited to go back to the team and talk about-
Willie Williams 30:45 Yeah
Katelyn Lesse 30:45 ... new API parameter-
Willie Williams 30:46 Yeah, yeah
Katelyn Lesse 30:46 ... personality aggression. [laughs]
Willie Williams 30:49 [laughs]
Katelyn Lesse 30:49 Whoa, me do. [laughs]
Willie Williams 30:50 [laughs]
Hot take and ‘magic moments’: more agents can mean more work—and more delight
Katelyn Lesse 30:51 Um, awesome. What's your hot take?
Dan Shipper 30:52 My hottest take is there's all this discussion about how AI is changing work, and there's all this fear of it, like, making work totally go away. And m- my experience or our experience, I think of, I think of Every as being like a little bellwether for how knowledge work might, might change because everyone is trying to use agents as much as possible to get as much of their work done as, as possible. And what we've found is the more we use agents, the more work there is to do. And, um, certainly, the work changes, and certainly, like, the way this will play out in different parts of the economy and different companies is going to be different, but I, I think that that is a sort of general truth that when you reach this horizon of work, you expect there to be nothing, but actually there's ... It just opens up a whole new horizon in front of you. And, um, and, and that provides an opportunity for everyone to actually just do better work at a higher quality that feels like them. Like, automating your work doesn't mean that it's done in a robotic way or that you don't pay attention to it at all. It just means that where you're working in the system has changed.
Katelyn Lesse 32:02 Okay, last fun question. I think a lot of people have this, like, magic fun moment that they sometimes feel with agents-
Dan Shipper 32:09 Mm-hmm, mm-hmm
Katelyn Lesse 32:09 ... that can be really powerful. One of my favorite ones is the emoji that I love to use the most is the blue colored heart. And one time, I was working, and I was working with Claude, and Claude showed up and reacted to one of my messages-
Dan Shipper 32:21 Mm
Katelyn Lesse 32:21 ... with a blue heart.
Dan Shipper 32:22 Mm-hmm.
Willie Williams 32:22 Mm-hmm.
Katelyn Lesse 32:22 I was like, "Whoa."
Dan Shipper 32:23 Mm-hmm.
Katelyn Lesse 32:24 "How did Claude know?" I'm so curious for you guys with Every Agent, what's been just the most fun, magic moment that you've experienced in working with it?
Willie Williams 32:32 Our CO, uh, keeps track of, uh, basically beer COs folks.
Katelyn Lesse 32:38 Yeah. [laughs]
Willie Williams 32:38 Like, "Oh, my bad. Uh, I... My bad. I did this, you know. Like E- Every, remind me that, that-
Katelyn Lesse 32:44 [laughs]
Willie Williams 32:44 ... I owe Dan a beer, uh, for this." Uh, and the agent will insert that back in occasionally, like, in, in the appropriate conversation where it's just like-
Katelyn Lesse 32:53 Oh, that's really cool
Willie Williams 32:53 ... "Oh, yeah, remember when Dan, like, hooked you up? Like, are you still... Is that... Have you, have you reclaimed that beer? Have you claimed that beer yet?"
Katelyn Lesse 32:59 [laughs]
Dan Shipper 32:59 The lead on the Every website, his name is Andre, uh, and he does these, like, really, really funny announcements-
Willie Williams 33:06 Mm-hmm
Dan Shipper 33:06 ... um, that are just, like, they're in all caps, and they're like, "I- You must turn on two-factor authentication-
Willie Williams 33:11 Yeah, yeah, yeah
Dan Shipper 33:12 ... in all of your logins or whatever." But he, like, spends a lot of time, like, making them just, like, really funny, and I just made a skill that was like, have-
Willie Williams 33:18 Yeah
Dan Shipper 33:18 ... the Every Agent, like, do announcements in his voice.
Katelyn Lesse 33:20 Nice.
Dan Shipper 33:20 And so now people are just using his announcement skill to, like, do all of their announcements-
Katelyn Lesse 33:24 Nice
Dan Shipper 33:24 ... and I'm just like, it's just very pleasing that-
Willie Williams 33:26 Yeah
Dan Shipper 33:26 ... that just works, you know? [laughs]
Katelyn Lesse 33:27 Poor Andre.
Willie Williams 33:28 Yeah. [laughs]
Dan Shipper 33:28 [laughs]
Katelyn Lesse 33:28 All the people take his seriously when everybody else is doing it. Nice. That's awesome. Okay, well, I'm excited for the day that the Every Agent orders you guys some beers-
Willie Williams 33:37 Yeah
Katelyn Lesse 33:37 ... to, to pay the debt. [upbeat music] This was awesome. Thank you guys so much for spending the time.
Dan Shipper 33:41 Thank you.
Willie Williams 33:41 Thank you so much. [upbeat music]
Episode duration: 33:46
In the uListen app, the transcript lights up word by word as you listen. Get uListen
Transcript of episode z7cNbsr3b5s