How I AIHow Mozilla Uses Claude Mythos to find Firefox bugs before hackers do
CHAPTERS
Why Firefox bug-finding can’t be “one shot”: scale, context, and tedium
Brian Grinstead frames the core challenge: Firefox’s massive codebase makes it impossible to ask an LLM to find all bugs in one pass. The conversation highlights why agents excel at tedious, exhaustive exploration that humans struggle to sustain.
The viral spike chart—and the real story behind it (Mythos vs. system design)
Claire and Brian unpack the “Firefox security fixes by month” spike and challenge the simplistic narrative that a single model (Mythos) caused it. Brian argues the harness and pipeline improvements were at least as important as model advances.
What a “harness” is: turning an LLM into a tool-using bug hunter
Brian defines the harness as the orchestration layer that gives an LLM tools (shell, build, browser, fuzzing infra) and a goal. This moves from “brain in a jar” chat to an agent that can measure outcomes and iterate.
Inside Mozilla’s agentic loop: file targeting → exploit attempts → fuzzing signal → structured output
They walk through the flow: prioritize files, run an analyzer agent with a target file, generate HTML test cases, and validate via existing fuzzing/ASan infrastructure. A verifier agent filters out “wonky” agent behavior to drive false positives near zero.
Goal loops and guardrails: making relentless iteration safe and useful
Claire connects Mozilla’s approach to goal/outcome loops and emphasizes guardrails to prevent shortcut solutions. Brian explains how prompt tuning and verifier feedback evolved from log review and engineer feedback.
Building it in practice: Claude Agent SDK, Codex options, and why vendor harnesses matter
Brian details implementation choices: starting with the Claude Agent SDK (Claude Code JSON streaming mode), exploring Codex integrations, and considering model-agnostic frameworks. He argues vendor harnesses are often best-tuned but defenders should test across models to mirror attacker diversity.
From detection to fixes: patching agent, verification, and why humans still review
They describe extending the system to generate patches, rebuild, and confirm the crash is gone. Despite automation, expert engineers review because agents can be overly narrow and miss related fixes across the codebase.
Real-world wins: reproducible HTML exploits and a 15-year-old bug archeology
Brian showcases the kind of reports that now matter: not just analysis, but a working repro HTML file that triggers a real crash. He also highlights an “archeology” win—using agents to trace when an old vulnerability was introduced despite file renames and history complexity.
Open-sourcing the tooling and the importance of a crisp verification signal
Mozilla open-sources parts of the tooling so researchers and defenders can test and adapt it. Brian stresses that many teams lack a binary verification signal like ASan crashes, so defining measurable success/failure criteria becomes essential.
Live demo: from “just a prompt” to a full harness run with structured outputs
Brian demonstrates a minimal V1 approach—running Claude Code with a prompt and JSON streaming—and a more capable V2 harness with tool access and verifier output. The demo illustrates how approachable the first step is, then why richer tooling is needed for non-trivial bugs.
How Mozilla prioritizes where agents look: LLM scoring + simple heuristics
They explain the practical constraint: you must choose targets. Mozilla uses a lightweight LLM “judge” to score files on likelihood of memory safety issues and web reachability, then combines it with operational heuristics (duplicates, prior runs) to allocate compute effectively.
Mobilizing Mozilla: incident-response intensity and high-actionability reports
Brian describes a rapid, high-volume push resembling incident response: many engineers coordinated to land fixes quickly. The key enabler was not just quantity but the high actionability of verified, reproducible reports.
Lightning round: model vs harness, security optimism, and prompting style
In closing, Brian estimates the impact as roughly “both” (a pragmatic 50/50) and expresses cautious optimism: AI improves bug discovery for defenders too. He shares pragmatic prompting habits—hands-on control for writing, tone-shifting for creative vs. ops tasks, and simple corrective nudges for bad code.
