CHAPTERS
- 0:00 – 0:30
What Muse Wonder is (Wonder vs Wonder Full) and how it’ll be tested
Claire introduces Meta’s Muse Wonder model family and explains she’ll evaluate it using the How I AI Benchmark. The goal is to see whether Muse belongs in the current “frontier” tier, especially for coding, knowledge work, and agentic workflows.
- •Muse Wonder comes in two variants: Wonder and Wonder Full
- •Evaluation framework: the How I AI Benchmark across real tasks
- •Focus areas: coding, knowledge work, and agentic workflows
- 0:30 – 1:00
Sponsor: DX report on AI code velocity vs review/validation friction
A brief sponsor segment frames a common AI tooling problem: AI increases code output, but bottlenecks shift to review and validation. DX is positioned as a way to measure speed, quality, and cost across the SDLC to assess real ROI.
- •AI tool spend growing rapidly; AI-authored code share rising
- •Overall innovation can remain flat due to downstream friction
- •DX measures speed/quality/cost to assess AI’s true impact
- 1:00 – 3:01
Meta’s positioning: ‘frontier’ claims and comparisons to top models
Claire walks through Meta’s public claims about Muse Wonder’s performance and where Meta places it among other top models. Meta emphasizes agentic personal and professional work as the headline category.
- •Meta positions Muse Wonder alongside leading frontier models
- •Core narrative: best model for personal-agent work
- •Emphasis extends from personal agents to professional knowledge work
- 3:01 – 4:32
Meta’s benchmark focus areas: domain expertise, agent memory, and sustained effort
Claire summarizes the three “frontier capabilities” Meta highlights: domain expertise, active memory, and sustained effort (active focus). She foreshadows that active focus shows up in her testing on longer-running tasks.
- •Domain expertise targets legal/finance/business workflows
- •Active memory: carrying user context across conversations
- •Active focus: sustained effort on challenging, long-running tasks
- 4:32 – 6:03
The setup pain: needing Muse Code and running it headlessly
Claire explains the practical friction of testing Muse Wonder: signing up for Meta APIs and installing Muse Code. She minimizes direct interaction by using Claude Code to run Muse Code headlessly rather than adopting Meta’s tooling day-to-day.
- •Requires subscription/API access and installing Muse Code
- •Muse Code described as unimpressive but functional
- •Workflow: run Muse Code headlessly via Claude Code
- 6:03 – 7:33
How the How I AI Benchmark works (task categories + dual judging)
Claire outlines the benchmark structure: ~20 tasks spanning product, design, coding, writing, agentic research, and creative work. Outputs are scored by an LLM judge plus Claire’s own evaluation for faster “first look” feedback.
- •Task mix: PRDs, prototypes, site redesign, agentic research, code repair, creative tasks
- •Scoring: LLM-as-judge plus Claire’s human judgment
- •This episode is a fast first-look rather than a full rerun of the entire suite
- 7:33 – 10:06
Overall results: standout agentic performance, solid coding, weak creative/design
Claire delivers the TL;DR: Muse is her favorite on agentic tasks and competitive on coding, but it drops significantly on creative work and design prototypes. She notes a clear product direction: strong orchestrator/assistant behavior over aesthetics.
- •Best-in-class (in her run) on agentic workflows
- •Coding is in the ‘big league’ ballpark; viable for backend work
- •Creative tasks and UI design sensibility lag behind leaders (e.g., Opus 5.5)
- 10:06 – 11:40
Sponsor: Jira Product Discovery as ‘multiplayer mode’ for PM work
A sponsor break argues that while AI can draft artifacts quickly, alignment across teams is the bottleneck. Jira Product Discovery is presented as the shared system for evidence, prioritization, and roadmaps that connect into Jira execution.
- •AI speeds up solo PM outputs, but team alignment is still hard
- •Central place to capture insights and prioritize with evidence
- •Context flows into Jira for developers or coding agents
- 11:40 – 12:40
Why Muse excels at agentic tasks: complex SaaS ticket resolution example
Claire explains what makes an agentic benchmark meaningful and shows how Muse handled a realistic billing/admin workflow. Muse navigates edge cases (wrong account link, duplicate/refund logic, prompt injection) and completes the full resolution loop.
- •Benchmark design: complex SaaS navigation + planted failure modes
- •Muse correctly identifies account mismatch and duplicate/refund details
- •Resists prompt injection and completes workflow (refund, plan change, reply, solve)
- 12:40 – 14:12
Email writing and voice assistant ‘orchestrator’ tone: where Muse feels strong
Muse performs well in Claire’s email-writing test by following instructions and matching a concise voice. In the voice assistant evaluation, it feels personable without being cloying, approaching Sonnet-level performance for assistant-style dialogue.
- •Email writing: instruction-following and tone are strong
- •Voice assistant test: consumer-friendly, straightforward demeanor
- •Positioning: good central orchestrator model for personal/knowledge agents
- 14:12 – 18:15
Where Muse falls short: PRDs, prototypes, and front-end design quality
Claire describes weaknesses in structured product writing (PRDs) and especially in UI/prototype outputs. Common issues include poor polish, incorrect mode choices (e.g., ignoring dark mode), broken interactions, and a consistent ‘template slop’ style.
- •PRDs: too short, less detailed, formatting/scannability problems
- •Prototypes: interactions don’t work reliably; polish gaps appear late in review
- •Front-end design style: repetitive components, ‘rounded-corner SaaS slop,’ ignores brand guidelines
- 18:15 – 19:16
Creative tasks: SVG illustrations are acceptable but lag peers
Muse generates competent SVGs, and Claire notes how impressive this would have seemed a year ago. However, compared to other frontier models, Muse ranks near the bottom in this specific creative illustration task.
- •SVG outputs are ‘totally fine’ and usable
- •Competitive field compresses scores near the top; Muse still trails
- •Claire wouldn’t choose Muse first for illustration/SVG generation
- 19:16 – 21:17
Short-form video editing: timing is good, design/audio issues remain
Muse does reasonably well selecting and cutting the right segment, but the visual overlays and caption placement feel low quality. Claire also flags odd audio handling, resulting in an overall low score despite decent timing.
- •Cut timing/selection is strong relative to many models
- •Overlays/captions/layout feel ‘crummy’ and not platform-native
- •Audio processing appears incorrect or degraded
- 21:17 – 22:18
Speed vs sustained effort: fast one-shots, slower long-running agent tasks
Claire observes a performance pattern: Muse is very fast for one-shot tasks but slower for agentic, long-running workflows. She suspects this ties to Meta’s ‘active focus’ approach and plans to track latency more systematically in future benchmarks.
- •Zippy on short tasks (notes-to-PRD, quick prototypes, small outputs)
- •Slower on long tasks (triage, backend features, video)
- •Hypothesis: ‘active focus’ trades time for better long-run task completion
- 22:18 – 24:04
Final verdict and next steps: strong agent model, not a creative/design leader
Claire concludes Muse Wonder is excellent for agentic work, serviceable for coding, and good for writing emails/assistant voice, but weak on design sensibility. Next, she plans to put Muse in the loop of agents she’s building to evaluate real-world UX and performance.
- •Strengths: agentic workflows, assistant voice, workable backend coding
- •Weakness: front-end design and creative aesthetic quality
- •Next: integrate into agent pipelines; watch Meta’s evolving model strategy
