CHAPTERS
- 0:00 – 2:01
Claire’s “personal-agent benchmark”: four agents, nine real tasks
Claire sets up the episode as a hands-on benchmark to compare four personal agents across nine everyday tasks. She explains she’ll walk through setup, prompts, and results, then share which agent is best for which kind of work.
- •Benchmarks four agents across nine tasks to determine “what personal agent is actually the best.”
- •Plans to keep the comparison as unbiased and equal as possible.
- •Will review results live and then summarize which agent fits which tasks.
- •Notes she spent hours figuring out how to run the benchmark.
- 2:01 – 3:32
Benchmark design: how tasks were chosen and how the run was orchestrated
Claire explains how she collected suggested “top tasks” from each agent, then consolidated them into a benchmark. She also describes her operator-agent approach and the constraints of running one agent through another.
- •Asked Dot, Grok Bot, Muse, and Claude Duet for the top tasks they do for her, then consolidated common use cases.
- •Used Claude to help pick cross-cutting tasks that test different capabilities.
- •Planned to use one operator agent with computer use to run the others; ended up splitting operators.
- •Had Claude run Dot, and Codex run Claude, Grok Bot, and Muse due to tool constraints.
- 3:32 – 5:33
Baseline access and prompt style: “one-line, human-driven” prompts
Claire emphasizes that tool access parity matters, so she checked connectors across agents and tried to baseline skills. She also clarifies her prompting philosophy: lightweight one-liners, with an operator agent steering when needed.
- •Used Codex browser/computer use to verify agents had similar connectors and skills (but forgot some Muse business tools).
- •Tried to baseline access, acknowledging differences like memory and built-in agent features.
- •Created nine use cases with short, specific prompts rather than long “prompt engineering” paragraphs.
- •Operator-driven execution required nudges and babysitting when computer use broke.
- 5:33 – 7:03
Early observations: parallelism, missing connectors, and “computer use” friction
Before scoring task outputs, Claire shares operational takeaways from running the benchmark. She highlights the benefit of parallel execution, the impact of missing tool access, and frustrations with one computer-use setup.
- •Loves the parallelism of having computer use run agents running other agents.
- •Notices Muse lacked connectors like Slack, affecting performance on business tasks.
- •Expresses strong frustration with Claude computer use performance.
- •Sets up that results will be scored live with human-in-the-loop judgment.
- 7:03 – 10:05
Scoreboard meta: speed, permissions, and completion rates
Claire previews the scoreboard and calls out high-level patterns across agents. She contrasts speed and interface behavior (like permission prompts), and notes some agents couldn’t complete all tasks.
- •Muse and Claude could only complete seven of nine tasks; Dot and Grok completed all tasks.
- •Muse appears fastest across multiple tasks; Claude is slowest overall.
- •Claude asks for the most permissions and is described as timid relative to others.
- •Time-per-task trends match Claire’s prior impressions from earlier reviews.
- 10:05 – 11:36
Task 1 — Draft a Slack reply “in my voice”
Claire evaluates how agents draft a Slack reply to a friend requesting a talk. She notes the task is easy overall, but highlights differences in voice, accuracy, and whether agents asked the right clarifying questions.
- •Muse couldn’t do the task because it lacked Slack access.
- •Dot stands out by asking about being paid and whether the talk is remote, which Claire values.
- •Claire judges Grok as most in her voice, and Dot as most accurate overall.
- •Claude’s response is short but she considers it less accurate than others.
- 11:36 – 14:43
Task 2 — Cut a short-form video from Dropbox footage
Claire compares how the agents download raw footage from Dropbox and cut a short-form clip. She watches outputs and critiques editing choices like overlays, punchiness, and zoom effects.
- •Claude Duet fails the task and will not download the video.
- •Muse produces the cleanest design with the show tag and caption layout; Claire prefers it overall.
- •Grok’s output includes unattractive overlays but better zoom-in/zoom-out behavior.
- •Dot looks similar to Muse but lacks punchier cuts; Claire ranks it behind Muse.
- 14:43 – 17:15
Task 3 — Fix a Sentry bug: grading quality and tradeoffs
For the coding task, Claire compares bug-fix attempts and uses another model to evaluate fixes. She credits two agents with the strongest performance, while calling out a risky approach from another.
- •Muse can’t complete the task due to missing GitHub/business access and misunderstands “Devin.”
- •Dot gets closest to the best fix; Claude finds an additional bug and identifies Dot’s fix as already sufficient.
- •Grok performs worst by loosening validation rather than properly addressing the issue.
- •Claire gives the win to both Claude and Dot for coding quality, with extra credit to Claude.
- 17:15 – 19:16
Task 4 — Build a How I AI thumbnail in Figma
Claire reviews thumbnail outputs using the team’s Figma templates and compares creativity versus template-filling. She concludes one agent consistently excels at Figma work based on repeated testing.
- •Muse’s thumbnail concept is strongly disliked and treated as a failure.
- •Dot and Claude produce acceptable results but mostly swap content into existing templates.
- •Grok provides stronger new creative ideas for thumbnails and headlines, and wins the task.
- •Claire states she has repeatedly found Grok to be the best at using Figma.
- 19:16 – 20:48
Task 5 — Triage inbox into actionable to-dos (and the importance of interface)
Claire evaluates agents on finding inbox to-dos and, crucially, how they present them. She argues that artifact format and human-agent affordances significantly affect usefulness.
- •All agents find reasonable to-dos, but differ in how they present them.
- •Claire prefers Grok’s document format with organized checklists.
- •Muse, limited to Gmail, still finds relevant parent-related tasks for Halloween week.
- •Dot performs worst by returning a simple message list rather than a structured system.
- 20:48 – 23:20
Task 6 — Create a family newspaper: artifacts for kids and weekly planning
Claire compares agents on generating a printable “family newspaper” covering schedules, reminders, and kid-friendly content. She critiques weak outputs and praises one agent for a charming, family-oriented artifact.
- •Claude disappoints by producing a markdown outline and citing limited calendar access; Claire calls it a failure.
- •Dot’s design is described as extremely ugly and performs poorly.
- •Grok produces a solid newspaper format with sections like reminders, sports, and weather.
- •Muse is praised for a cute two-page artifact with an appealing image, schedule, and day-by-day elements.
- 23:20 – 24:52
Task 7 — Explain a physics homework question to a fourth grader
All agents answer the question correctly, so Claire differentiates them by speed and usability. She uses this to explain how she’s shifting certain “Google-y Q&A” chats to one agent.
- •All agents get the correct answer (C) and provide a reasonable explanation.
- •Muse wins primarily due to being extremely fast on this task.
- •Claire says she’s been moving quick Q&A/explanations to Muse because of speed.
- •Frames this as a practical, on-the-go use case (e.g., photographing homework).
- 24:52 – 26:24
Task 8 — Shop for autumn shoes: site blocking, price comparison, and effort
Claire tests agents on finding a deal for a specific style/color preference and compares navigation and comparison behavior. She notes one agent gets blocked frequently and another barely attempts to compare options.
- •Claude is repeatedly blocked by shopping sites and searches the wrong color direction; it also finds expensive options.
- •Grok performs well at price comparison and navigation, matching Claire’s past experience using it for shopping.
- •Muse also does well but requires more steering.
- •Dot performs poorly and does minimal comparison or exploration.
- 26:24 – 30:08
Task 9 — Make a ChatPRD Product Intelligence launch video
Claire reviews launch video outputs and reacts to very different interpretations of the prompt and skill usage. She praises one agent for branding, completeness, and voiceover integration while criticizing others’ results.
- •Dot produces a PowerPoint-like result that Claire strongly dislikes.
- •Grok follows the skill but has layout/overflow issues; better than Dot but not great.
- •Muse lacks repo access and invents details; Claire calls it bad but gives some credit for ambition.
- •Claude delivers the most complete, on-brand video and uses an ElevenLabs MCP voiceover; Claire calls it the best for this task.
- 30:08 – 35:09
Final ranking and takeaways: Grok wins, Muse for life, and why access/speed matter
Claire shares the final scores generated after feeding her judgments into Claude: Grok wins overall. She closes with lessons about matching agents to job types, ensuring consistent tool access, valuing speed and artifact design, and noticing different safety/permission styles.
- •Final winner is Grok Bot; it wins tasks including thumbnails, inbox to-dos, and shoe shopping.
- •Muse is credited for excelling at personal/creative outputs (short-form content, family newspaper, fast homework explanations).
- •Dot is framed as stronger for some worklike tasks but weak on creative outputs; Claude Duet is slow but strong on bug-fixing and the launch video.
- •Takeaways include the importance of consistent access/connectors, speed, quality-of-life interfaces/artifacts, and differing safety/permission behaviors.
