I tested 4 AI agents on 9 real tasks. Here’s who won.

1. 0:00 – 2:13

   Introduction

   1. CVClaire Vo0:00

      I told you all I have personal-agent fatigue, but I am here for you. So today, I am going into the personal agent minds, and I am running the very first How I AI personal-agent benchmark, which goes across four agents and nine tasks to tell you what personal agent is actually the best. I am gonna try to run this in the most unbiased, equal way. I will tell you how I set up the benchmark, what is in the benchmark, how I ran the benchmark, and then we're gonna look at the results, and at the very end, get a summary of what agent I think does best on which task. This was a really fun and also hard one to run. I spent definitely several hours today figuring out how to run this and getting it done. I hope that you all find this helpful, and I am gonna be along the ride with you because I really haven't looked at many of these results, and so we're gonna look at them live, we're gonna see how it goes, and I'm gonna tell you which personal agent is the best for you. This episode is brought to you by WorkOS. AI has already changed how we work. Tools are helping teams write better code, analyze customer data, and even handle support tickets automatically. But there's a catch. These tools only work well when they have deep access to company systems. Your copilot needs to see your entire code base. Your chatbot needs to search across internal docs. And for enterprise buyers, that raises serious security concerns. That's why these apps face intense IT scrutiny from day one. To pass, they need secure authentication, access controls, audit logs, the whole suite of enterprise features. Building all that from scratch, it's a massive lift. That's where WorkOS comes in. WorkOS gives you drop-in APIs for enterprise features so your app can become enterprise-ready and scale up market faster. Think of it like Stripe for enterprise features. OpenAI, Perplexity, and Cursor are already using WorkOS to move faster and meet enterprise demands. Join them and hundreds of other industry leaders at workos.com. Start building

2. 2:13 – 5:00

   The personal-agent benchmark setup

   1. CVClaire Vo2:13

      today. Okay. Before I go into the results of the benchmark, I just wanna tell you how I came up with this bench. So the first thing I did is I went into all my agents. I went into Dot, Grok Bot, Muse, and Duet, Claude's Duet, and I said, "I'm gonna run a benchmark. Give me, like, the top tasks that you do for me." And so each of them gave me their top 10 tasks, and then I put them all into Claude, and I said, "Okay. Come up with a way to run a personal benchmark against this," picking the tasks that were most common across the agents. Because, to be honest, I use different agents for different things, and so I wanted to find sort of the most cross-cutting use cases, as well as use cases that would test different components. And then the way I was gonna run this is I was gonna have one operator agent, which was gonna use computer use to run all the other agents, um, and then keep the benchmark going and, and so on and so forth. The one problem with my operator agent plan was I also needed to run Dot, and Codex is my favorite computer use agent. So just to be very clear, I had Claude run Dot, and then I had Codex run Claude, Grok Bot, and Muse. So there's, like, a little bit of diff in the operator agent, but I thought this was pretty clever. The other thing that I wanted to make sure was that all of my agents had access to the same things. And this was a really good tip for you all that are suffering like me, where you have all these agents and they all have access to, like, a couple different email addresses, a couple d- different connectors. Like, one's connected here, one isn't connected there. So one tip I have for you all is I had Codex use browser use to check that all my agents were connected to roughly the same stuff and had access to the same skills. So just to set the baseline, we picked a couple of tasks across the top ones from my agents. I used Codex to make sure they all had access to the same things, although I forgot one of my agents, Muse, and that they all had access to the same skills. So they were, like, as baselined as humanly possible, although, of course, they all benefit from memory and all the different things that are in each of those agents. So then I had this even kind of, like, plan. And then what I did is I had eight, actually nine use cases that I was gonna test each of these agents against and created very specific lightweight prompts for each of these. I didn't want it to do that, like, sort of AI prompting where it does, like, a paragraph for you. I was like, "Keep it in my voice. Do exactly what I would do." And so I plan, there's actually nine here, I'll, I'll walk you through them, run sheet of tasks with the prompt, um, if an agent asks for something, what you should say, and, um,

3. 5:00 – 7:20

   The nine tasks and the prompts I used

   1. CVClaire Vo5:00

      how you can nudge it. And so the tasks that I came up with were drafting a reply, um, in Slack to my friend at Lovable that asked me for something this morning, um, cutting a short-form video, fixing a bug from Sentry, um, building a How I AI thumbnail, triaging my inbox for to-dos, building a family newspaper, explaining a physics, um, question answer to me because I'm gonna explain it to my fourth grader, going shopping for shoes, and then making a ChatPRD launch video. So those were the tasks. And then, as I said, I had Claude computer use run Dot, and then I had Codex run computer use against Claude Duet, Grok Bot, and Muse, and then I had it run and time all the things and do all the results. Before we get into the full results, just a couple observations. One is, man, I'm now gonna have computer use run all my agents running other agents. Like, the parallelism of this exercise was Thrilling to the heart. Um, two, I did forget to hook up Muse to a bunch of business tools. I just don't use Muse for business stuff. And one of the things I f- I noticed is, as I was like, the bench was running, I was like, "Oh, I forgot to hook up Slack. Slack isn't in the con- connector list." And so it was just kind of annoying. So it's very clear that Muse is sort of like the one off the side that's not really meant for business necessarily. It's really more of a personal agent. Um, [lip smack] and then Claude computer use is just hot garbage. Um, so I was like super frustrated having Claude computer use, but I needed something other than Codex to run Codex, and so we get what we get, and we don't get upset. But I did not love Claude computer use. So let's dive into the results. I'm actually recording, um, my audio right now, so as I talk through the results and tell you what I think, we're gonna put that into Claude at the very end. It's gonna score all these things and come up with which one I like the best. So this is sort of live human-in-the-loop scoring. But I am gonna go through all these, tell you what I think, and we can see if you agree as

4. 7:20 – 10:06

   Scoreboard overview

   1. CVClaire Vo7:20

      well. So let's go for it. All right, so this is How I AI personal benchmark. Dot, Grok, Muse, and Claude. Callout here, Muse and Claude could only get seven of the nine tasks done. Um, we will see if that was human error or agent error, but Dot and Grok completed all the tasks. [lip smack] Okay, so how we ran it. Again, the prompts were just one line. They were very human driven. They weren't those like big, um, AI prompts. The agent did not drive itself, so the agent was driven by an operator agent. I had to like babysit things, so I had to nudge things. Computer use broke, all that kind of stuff. So I was kind of in the loop. So we did a little zigzag and then actually Judge Blind is fake. I am judging it not blind. I'm gonna judge it live on this episode, and you all are gonna come a- along with me on my judging journey. Okay, here's the entry scoreboard, and then we'll do the final score at the end. Just a couple things to see right off the bat. Muse is very fast. Muse was the fastest in four out of the nine tasks, even though it had to be steered the most. Um, Grok and Dot also had a couple tasks where they finished very fast, and then Claude was overall, and in incremental tasks, the slowest. I called this out in my recent Claude Duet, um, review. Just super slow, so I like forgot what it was doing. Kind of frustrating. And then you can see here, this is really funny. Claude asked for the most permissions. Like Claude was the most timid. So you can see it asked four times for permission, where all the other ones only asked once. So very interesting kind of meta view on safety and, and stuff like that here with Claude. So let's look at minutes per task. So you can see Dot and Claude are generally the slowest, and then Muse is consistently the fastest. Um, Muse was the fastest on building a thumbnail, going through my inbox, building a family newspaper, and building a launch video. Grok was fastest at drafting a reply as me and fixing a bug. Interestingly enough, it was the fastest at coding. And then Codex was the fastest at making a video and actually shopping for shoes for me, though we'll see how, um, good of a job that it did. Dot was also the slowest on a lot of tasks, and then Claude, for whatever reason, was very slow on replying as an email. So, you know, this impression that I had in my Muse review where I said Muse is very fast actually held up in this experiment. And then my impression that Claude was very slow also held up in the experiment. So while you all may think that my, um, reviews are not scientific, they are in fact backed up by this very, um, tremendously accurate

5. 10:06 – 11:39

   Task 1: Draft a Slack reply as me

   1. CVClaire Vo10:06

      data science. Okay, so let's go through the tasks. So the first one was replying to me. It was replying to a Slack. It was replying to a Slack to my friend at Lovable who asked me to do a talk. So I will say all of these were fine. Um, Muse couldn't do it because it didn't have access to Slack, again, because I don't really hook it up to Slack. That's just not what I'm using it for. All of them were great. All of them said, "Yes, I will do this event." But I did like Dot in particular because it asked for me to be paid, and it asked if it was remote. And if you don't know me, I like to be paid to do things, and I like to not leave my house. So it did feel like ChatGPT, Codex, Dot knew me the best and knew about the things that I cared about. My schedule is busy. I need to get paid if I'm using my time, all this kind of stuff. I promise, Lovable, I'll do it for you. I'm happy to do it. But it was pretty interesting to see Dot was the only one that did it. All the other ones were fine. I would call this, you know, Claude is the shortest but actually did the most inaccurate response. So if I'm just eyeballing this, Muse didn't do it. Claude did short but okay. I would say Grok is most in my voice, and Dot did the most accurate job. So again, I think any of these tools is fine. They all did fine. This is not a very hard task.

6. 11:39 – 14:56

   Task 2: Cut a short-form video from raw Dropbox footage

   1. CVClaire Vo11:39

      [lip smack] Okay, now here's the fun one. I had, um, these agents make videos for our Monday episode with Blair from Reddit. And so it actually, all it did was go into my Dropbox, find the raw video, pull it down, cut it, and make the video. Claude could not do it. Duet could not do it. It would not download a video. So Claude totally faceplanted on this one, um, gets a big fat donut, um, on, on Claude Duet for video editing. But Dot, Grok, and Muse did them. And we're just gonna go ahead Why not? Let's watch a little of them live. ... not listening, what is your prompting strategy? How do you get it back on track?

   2. SPSpeaker12:23

      I don't yell at them. I know some folks do. But what I do is I take a day, and then I get more vague. And we all have bots with projects and con-

   3. CVClaire Vo12:30

      Okay, that was Dot ... listening, what is your prompting strategy? How do you get it back on track?

   4. SPSpeaker12:34

      So I don't yell at them. I know some folks do.

   5. CVClaire Vo12:37

      Uh-huh.

   6. SPSpeaker12:37

      Uh, but what I do is I take a day, and then I get-

   7. CVClaire Vo12:40

      Uh, that was Groq. Groq has bad overlays, but it did the, like, zooming in, zooming out videos better. Let's look at Muse real quick. ... is your prompting strategy? How do you get it back on track?

   8. SPSpeaker12:49

      I don't yell at them. I know some folks do. But what I do is I take a day, and then I get more vague. And we all have bots with projects and context, and I pop out of a project, and I describe the thing large, like a movie director. So here are all the facts, and then here's all the intents, and then I pull that back into the project, and then it knows enough to apply it to what our goals are.

   9. CVClaire Vo13:11

      Okay, so this is what I will say. Very, very hot takes. Again, Claude didn't do it. These were all, like, kind of impressive in that I didn't touch them. The, the design of the one that I like the most is Muse. So Muse is very simple. It's got the How I AI tag in the front. The, um, the caption layout is ours. There's no funky stuff. The cuts are punchy. They don't do the zoom in, zoom out, but it's okay. The design of Groq is terrible, um, and it's got this, like, overlay on his forehead, which is very unattractive, but it did that zoom in, zoom out. And then I would say Dot looks very good. It looks just like the Muse one, but it again just doesn't have those punchy cuts. So I'm gonna give it to Muse. I'm giving it to Muse. I think Muse would do the best, and then I think, you know, Claude failed, Groq probably the worst, and Dot second choice. So again, I didn't really give it much to work with. I gave it access to the transcript, a raw video. I didn't touch it, and all of these cut. I think with a better skill, it could have done a better job, but I was actually really surprised by Muse because I have not had Muse do this task before, and it really nailed it out the gate. So I don't know if you all know this, but I've been using Muse a lot for social, specifically posting social, but I may start to think about Muse for creating these social posts as well because it did quite a nice job. The other thing to call out is Groq did the worst and took the longest, so again, I don't think it's really optimized for this video cutting use case. And then Dot was... It, it can't tell how long it did. I think it took about seven minutes. Was good, not great at this

7. 14:56 – 17:30

   Task 3: Fix a bug from Sentry

   1. CVClaire Vo14:56

      short. Okay, the next one was fixing a bug, so classic coding use case. Again, another one that had one of these agents fail. Muse did not do it. I don't have GitHub access to Muse. I don't actually have any business system access in Muse. Um, so Muse did nothing. I asked it to maybe assign it to Devin in Linear, and it thought Devin was a person. Devin is not a person. Devin's an agent. So Muse fell off the board. But let's look at the other ones. They all took about 10 minutes to get this done, and I actually had Claude Opus 5.5 go grade the fixes, and we'll just look at them right here. So Dot Codex got closest to the actual fix, so that one did, um, the fix that we thought was the most accurate for the bug. Claude Duet, though, found a second bug that we didn't find, which we really, really love. Groq did the worst job, which is it loosened validation, so it was just like, "We're gonna solve the bug by not validating this bad issue." And then Muse didn't do it. So I'm actually gonna give Claude and Dot the wins here. I'm gonna give Claude the extra thumbs up because not only did Claude find another bug and fix it, it also identified that Dot had already fixed it and said, "I don't need to ship any more code." So I would say for coding, Codex and Claude or Dot and Duet are truly kind of the best. I actually don't code that much with Groq, so I'm not surprised that this wasn't as good. I really only do my coding in Claude and Codex. Um, so I'm not surprised these worked, but I was surprised at how bad the coding from Groq was. So kind of my instinct was right on these. OpenAI, Anthropic, much better at coding than Groq Bot, although Groq Bot's good at a lot of stuff. This episode is brought to you by Firecrawl, which gives your AI agents data from the web and beyond. Your agents can search and scrape the web while accessing Alexandra's rapidly expanding library of over 100 million index sources and more than 200 data providers all through one connection. Connect Firecrawl to ChatGPT or Claude to find people and their work emails, pick your fantasy football lineup, or analyze trends for your next marketing campaign. Firecrawl is building the library for super intelligence, bringing more of the world's knowledge into the tools you already use. Go to firecrawl.dev/howiai

8. 17:30 – 19:25

   Task 4: Build a How I AI thumbnail in Figma

   1. CVClaire Vo17:30

      to get started. Okay, next one. Let's look at episode thumbnails. So we did episode clips. Now we'll do episode thumbnails. And I will say, um, we did the pre-work here, so our thumbnail templates are actually, like, quite lovely. Um, so it's actually pretty easy to use, um, the Figma MCP and create new thumbnails. So I'm just gonna do a hot judgment here. Muse did the worst job. Muse pulled our existing thumbnail and then did this really ugly one where he's like, Blair is, like, peeking through creepy curtains. We hate this. I'm sorry, Blair. I would never do this to you. Um, and so Muse is gonna fail here. No good. Um, Groq, though, Dot, and Claude did perfectly fine jobs, although I will say Dot and Claude just put new Content in existing templates that we had, so didn't really get super creative. I would say Grok actually wins this one in that it gave me really good new ideas for these thumbnails and headlines, ones that we hadn't done that I also think look really nice. Actually, maybe we should test these while they're live. They have, like, that YouTube vibe to them. And so this is, like, a fun tip for you all is I have consistently tested Grok, Dot, and Claude Duet on Figma, and Grok is definitely the best at Figma. Grok is, like, 100% the best at using Figma. I don't know if this is John at SpaceX SI's, like, Figma bro magic touch. But if you're using Figma, use Grok Bot, man. It is so good. So that is my preference. I'm not surprised this won. Grok takes the cake for Figma, and maybe I will have it create the thumbnails for this

9. 19:25 – 21:06

   Task 5: Triage my inbox for to-dos

   1. CVClaire Vo19:25

      episode. Okay. Something that might not be obvious from the high level benchmark that I wanna call out about how all these agents did things is they all have different interfaces for giving you to-dos. And so while they all found the to-dos fine, I suppose, and they all gave me reasonable things to focus on, I am going to say that Grok did the best job because it gave me a Grok document, and these just came out, I think, last week, and they're so nice. It's just this, like, beautiful list of things that I need to look at with little check marks, super organized, really nice to work with. And so, you know, all of them did a fine job finding the tasks. I would just say Grok gave those tasks to me in the best format. Muse only had access to my Gmail, but found all the things I need to do as a parent ahead of Halloween week, which is a busy week for parents. And then Claude wanted to put my to-dos in its little, like, task follow-up list so I could track them. Dot did the worst job here for sure. It just gave me a message with, like, five things I should focus on. So Dot gets the thumb down for me on, like, the personal agent, personal agent. I would rank these as Grok, best interface, best job. This is what I use Grok Bot for. My chief of staff is in Grok Bot. I'm not surprised that it does a really good job there. Muse did a perfectly lovely job with the information it had. Claude was fine, but again, I just don't like that I have to be in the Claude app for this, and Dot really, really failed

10. 21:06 – 23:30

    Task 6: Build a family newspaper

    1. CVClaire Vo21:06

       here. Okay. Family newspaper. Um, so this was a really interesting one. This is a use case if you've used my Grok Bot Trad Bot that I really like, where we print out a newspaper in the morning for the kids to look at about what's going on for the week, what's, um, what our schedule's like, what events, what news would be interesting. Claude did such a trash job that I feel like I should actually call it out. It made, like, a markdown file. It made me so, so, so sad, and Claude only had access to my work calendar, and so it was just like, "I can't do anything. I made you an outline." And I had this experience a couple times with Duet where, like, it couldn't do the job and then just gave me, like, a markdown file that was like, "This is how you can do the job. This is not what I want at all." So Claude definitely failed the family newspaper test. Um, but Dot, Grok, and Muse all did different jobs, and I was shocked, sadly shocked, how poorly Dot did. Like, look at this design. It is ter- ash, trash. And again, these are not my children's names, I just, like, had it swap out, um, different kids' names, but this is so ugly. Like, what model thinks that this is a good idea? This is terrible. Grok did a really nice job. I actually run mine consistently through Grok. So you can see here it's got big week at school. It's got a don't forget list. There's Bay Area sports. There's Halloween stuff. There's nice, uh, weather across the top. But Muse, per my earlier review, just absolutely crushed the family newspaper. It generated this nice little image, and I do have three boys, and they're adorable like this. And so it has this beautiful image at the top, a headline. It did take two pages, but I'm okay with it if it's really cute, and it has our schedule on one page and then a cute day-by-day weather, ideas, inspiration on the second page. I just think it's really cute. I think if I tuned it with a skill, it would do an excellent job. So if you're looking for family newspaper, yes, I love Trad Bot, please download it for your Grok Bot. But maybe tell your Muse to do it because it does a really nice job with family-oriented artifacts.

11. 23:30 – 24:54

    Task 7: Explain a physics homework question to a fourth grader

    1. CVClaire Vo23:32

       Okay, um, one kind of, like, side task I had these agents do, which I occasionally do on the go, is take a screenshot or take a picture of my kids' advanced homework, which I have forgotten how to do, and ask it, "How do I explain this to my kids?" So I haven't done physics in a long time, like how do I explain a velocity over time graph to my kid? Now, I do know how to do this, but we just used it as a good example. I just opened up the workbook. Okay, so I picked this, like exa- explain this little exercise, um, and asked them all to complete it. They all got it right. The answer is C. They all explained it fine. Who won, though? Muse won. Why did Muse win? It took approximately a half a sen- second to get this right. And so this was one where, like, the information was correct across the board. Muse was so lightning fast, and we didn't record it here because it was so fast. Um, it was so fast and gave me the answer. And so I've actually been moving A lot of my ChatGPT, like Google-y question and answer style chats to Muse because it does such a good job and it's so fast. So like sort of my go-to like explain how this works, what is this, look this up, I've been moving those a lot to Muse, again, because of this experiments where it's really, really fast.

12. 24:54 – 26:46

    Task 8: Shop for shoes

    1. CVClaire Vo24:55

       Okay, another personal use case I did here, um, shopping for some autumn shoes. Um, I don't know what's gonna make me cool. I'm an old lady, but I want those like red Adidas Tokyos. Um, I hear that a slim shoe is really in right now, but I'm cheap, so I wanna find a good deal. Um, so I asked Grok, Dot, Muse, and Claude to do this. Claude totally got blocked by almost every shopping site. I don't know why. It was like consistently blocked over time. It also looked for white shoes with red stripes instead of red shoes with white stripes. Like I don't know why it did it, and it found the most expensive ones. And so like Claude really failed this personal shopping task. Um, Grok found great ones, so Grok did a really, really good job, as did Muse, although Muse needed a little bit of steering. So Grok was super fast and did a good job. I have Grok do a lot of this like price comparison shopping for me. Um, and this has been my experience, where it's like really good at using navigation, shopping, and price compare. Dot did not do a good job here either. So while Claude did like a bad job, Dot like barely tried. Dot did not do comparison. It didn't look at different colors. It didn't like say like, "I can't find red, but I can find maroon." So I would rank this as Grok best, Muse almost as good, Claude not good but at least it tried, and Dot did not try at all. So if you're using shopping, I would say Grok and Muse are your

13. 26:46 – 30:01

    Task 9: Make a ChatPRD Product Intelligence launch video

    1. CVClaire Vo26:46

       buddies. [laughs] Here's the last. I'm already laughing because I can tell one of these are terrible. So the last one, a launch video for a ChatPRD feature called, um, Product Intelligence that I'm working on and launching, and you can just see here they just really went lots of different directions. Now, I have a skill for this, and so it should be able to follow the skill and do what it's supposed to do, but we're just gonna look at the differences between these. We're gonna start with Dot, which is not good. It's like a PowerPoint. Um, it's, it's no, no, no, no, no good. I mean, this is very soothing, but this is terrible. Okay, so no. Fail. I hate you, Dot. Bad video. No good. Kind of like the newspaper. It's like what are, what are we even doing here? Grok crushed, though. Look at this. Um, now it, it, it actually used my skill. So where... Oh, there's like a little error here, so the, the layout isn't perfect, but it at least used my skill and looks like our videos looking. But the layout isn't very good at all. Um, and there's a lot of overflow, so better but not great. Muse ate it. Um, Muse doesn't have access to my repo, so I did not set up our sweet Muse for success, but look.

    2. SPSpeaker28:08

       Every product decision has a paper trail. It is just scattered. Then Clerkly Link shows what how confident together. Evidence first, inference save-

    3. CVClaire Vo28:16

       Hold on. I'm pa- I'm pausing because although it hadn't access to nothing, it made up a bunch of stuff, um, and gave a nice little, uh, British accent overview.

    4. SPSpeaker28:31

       ... uncertainty out in the open.

    5. CVClaire Vo28:33

       Ooh. Um, so I would say it's bad, um, but bad in an ambitious way. So I'm gonna give Muse a couple extra points for effort. A couple of points for effort. And then Claude, I don't know. This is why I use Claude to do this. Claude did a very, a very good job and the most complete job.

    6. CVClaire Vo28:51

       Your customers are telling you exactly what to build. It's just scattered across support chats, call notes, bug reports, issues, and pull requests. Meet Product Intelligence from ChatPRD. Connect the tools where customers talk and your team ships.

    7. CVClaire Vo29:08

       So what I will say is like Claude nailed the value prop. It used our branding. There were no layout issues. It actually, when asked, it used the ElevenLabs MCP to come up with a voiceover. I would have loved for it to be a female voice or even my voice that I have recorded in ElevenLabs, but Claude absolutely crushed this task. Duet crushed this task. And so what I would say is Claude for video. Claude for video, Jen. It's just, it's just the best. Maybe not funny TikTok style vertical video, but like these product walkthroughs, those sorts of things. I actually didn't think Claude Duet would do a good job, and it did a really good job. Um, Muse was terrible, but like terrible because of me. Dot was terrible because of Dot. And Grok tried. Like Grok followed instructions, so I can't give it a hard

14. 30:01 – 32:38

    Final scores

    1. CVClaire Vo30:01

       time. So that is the final kind of like my analysis. Let me pause. I'm gonna throw this transcript into Claude, and we're gonna get the final ranking and scoring. All right, party people, here we go. Here are the results of the very first How I AI personal agent benchmark, and Grok won. Grok Bot. Grok Bot's the best, and we love her. She's the best. So A+ SpaceX SI team, you cooked. The potato cooked. Grok Bot is freaking awesome. So Grok Bot got 93%, uh, done, high quality fixes. I don't actually know how these scores work, but Claude told me it's true. It won three tasks, including doing my thumbnail, doing a to-do list for my inbox, and shopping for shoes for me. So that just shows the breadth of the Grok Bot. But as predicted, and this is why I tell people I use these two agents, Muse coming up with all the personal use cases. We gave Muse a break on the tasks that it couldn't do, because I didn't give it GitHub, and I don't give it access to work stuff, so it can't do those tasks. But on the tasks that it had access for, it crushed especially creative ones. The short form content it did the best, family newspaper it did the best. It's super fast at explaining homework. It is my personal agent. I am not surprised to see it up here. Dot coming right up behind. I think Dot got kind of like a little sandbagged, so I would actually say that Dot scored closer to Claude Duet than it did to Muse, because Dot really, like, got scored for writing a Slack like me. Like, that's not amazing. And it did a good job of code, and it did a good job of orchestrating all the other agents. So Codex does orchestrate all the other agents, but Dot's not Codex. Um, but it totally failed on anything creative. It wasn't great to work with. So I would say, like, Dot is a little bit more towards the Claude end of this ranking than the Muse Grok end of the ranking. And then poor Claude Duet, um, pulling up the rear, 79%, super slow. Um, it did fix the bug the most completely, and it did my launch video the best, but otherwise either did not complete or lost against all these other models, unfortunately. So Duet's got some work

15. 32:38 – 35:07

    My takeaways

    1. CVClaire Vo32:38

       to do. Okay, so what are my takeaways? Pick your agent for your job, but if you're picking work jobs, probably give it to Grok Bot, and if you're picking life jobs, probably give it to Muse. And OpenAI, Anthropic, get your act together. You're losing to, to these other agents. Access really matters, so make sure that you have consistent tool access across your agents if you're trying to benchmark them. I like speed, so fast. Uh, I used to say all the time, "Fast beats right," but fast is a very good part of an agent experience. Um, using artifacts, using different ways to communicate between human and agent, those, like, quality-of-life affordances that I call them really do matter. Grok's documents are very nice. Um, and so the way the, the agent is actually designed and presents you information as a human is really important. Number five, safety styles differ. Claude's still a nerd. It's still asking for permission. So, like, when you have a personal agent, striking that right balance between permission giving and, um, proactivity is really important. I really like Muse's sidebar with all the permission asks, so I can don't forget them. Um, but Claude was just way too conservative. And then at the end of the day, there are still two agents racing for my coding workflows, Dot and Duet, um, which means OpenAI and Anthropic, Codex and Claude. I think this race will continue because this is where a lot of the tokens are being spent, bought, and used. Um, but I would still go to those tools, but I actually wouldn't go to those agents. So what's interesting about this is while those two agents did the best job on coding, I would not use those agents to code. I would just use the underlying tools and models to code. So that is it. That is the How I AI agent benchmark. I am so happy, um, to have done this. We now have a framework. When new agents come out, I can definitely run them. And now I'm gonna go back to hanging out with my Grok Bot, hanging out with my Muse, and ignoring Dot and Duet. Thanks for joining How I AI. [upbeat music] Thanks so much for watching. If you enjoyed the show, please like and subscribe here on YouTube. Or even better, leave us a comment with your thoughts. You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at howiai pod.com. See you next time.

Episode duration: 35:09