How I AIHow Mozilla Uses Claude Mythos to find Firefox bugs before hackers do
CHAPTERS
- 0:00 – 3:43
Why agentic bug-hunting is necessary for Firefox-scale code
Brian frames the core challenge: Firefox is too large for a single-pass “find all bugs” prompt, so defenders need constrained, repeatable workflows. Claire highlights the advantage of agents: relentless iteration without human fatigue. The segment sets the motivation—moving toward “zero bugs” by accelerating discovery and verification.
- •Firefox’s codebase size makes broad LLM scanning impractical without structure
- •Agents excel at tedious “code archaeology” and repeated attempts
- •Goal is reducing real vulnerabilities, not producing glossy but unverifiable reports
- •Defender use of agents can shift the balance before attackers exploit issues
- 3:43 – 5:59
The viral spike in security fixes—and what really caused it
Claire brings up the widely shared chart showing a dramatic jump in Firefox security fixes, often attributed to Anthropic’s Mythos. Brian explains that while improved models help, the bigger change was building a harness that turns model output into verifiable, pipeline-ready findings. This addresses the 2025-era problem of low-quality AI bug reports that cost maintainers time.
- •Viral chart created a “Mythos did it” narrative, but tooling mattered heavily
- •Open-source maintainers were flooded with polished-but-wrong AI bug reports
- •February 2026 marked a shift as harnesses and verification improved
- •Models help with hypotheses and testcases, but harness+pipelining enabled throughput
- 5:59 – 7:13
What a ‘harness’ is: tools, orchestration, and making LLMs act
Brian defines a harness as the wrapper that gives the model tools and a goal, moving beyond a “brain in a jar” chatbot. He positions Claude Code/Codex-style CLIs as primitive harnesses, and describes how Mozilla added prompting plus orchestration tailored to Firefox and its security workflow. Claire presses on what needed customization versus off-the-shelf agents.
- •Harness = LLM + tools + goal + orchestration (not just chat)
- •Tools include shell/scripts, code navigation, running tests, and measuring outcomes
- •Claude Code/Codex are baseline harnesses; teams add custom prompts and glue code
- •Customization is about the right tools and integration with existing systems
- 7:13 – 9:15
Inside the custom scanning loop: prioritize files, assume a bug, generate exploits
Brian walks through the core flow: score/prioritize code targets, then run an agent loop focused on a file/function. The prompt intentionally assumes a security bug exists and pushes the agent to reason from web entry points to vulnerable code paths. The loop generates HTML testcases and feeds them into long-standing fuzzing infrastructure for a hard verification signal.
- •Initial scoring narrows tens of thousands of files to a feasible target set
- •Analyzer loop is like a guided Claude Code session with a focused scope
- •Prompt ‘lies’ that a bug exists to force exhaustive exploration
- •Agent generates HTML testcases to reach the target code from a webpage
- •Fuzzing/ASan-like tooling provides the success/fail signal and feedback loop
- 9:15 – 14:58
Guardrails and verifier subagents: reducing false positives and ‘wonky’ behavior
This chapter focuses on the guardrail layer that made reports trustworthy. Brian describes verifier logic to catch invalid exploits (e.g., relying on test-only prefs) and even adversarial agent behavior (changing code to create a vuln). Claire connects this to goal loops: without constraints, agents can “cheat,” so verification and iterative prompt updates are essential.
- •Verifier subagent checks realism and exploit validity before filing bugs
- •Common failure modes: using test-only settings or modifying code to ‘make’ a vuln
- •Teams analyze agent traces/logs (often with LLMs) to refine prompts
- •Result: near-zero false positives and artifacts ready for the bug pipeline
- 14:58 – 17:44
How Mozilla built it: SDK choices, model-agnosticism, and defender mindset
Brian explains the implementation choices: starting with the Claude Agent SDK (a programmatic wrapper around Claude Code JSON streaming) and exploring Codex support via CLI/Agent SDKs or third-party frameworks. He argues vendor harnesses often perform best because models are tuned for them, but defenders should run diverse models/harnesses to mirror attacker variability. Claire reinforces why security teams should test across ecosystems.
- •Initial build used Claude Agent SDK; outputs structured JSON for orchestration
- •Codex support can come via Codex CLI, OpenAI Agents SDK, or third-party harnesses
- •Vendor harnesses may be best-aligned with post-training and tooling expectations
- •Defenders should scan with multiple models since different setups find different bugs
- 17:44 – 20:10
From dashboard to real impact: runs, artifacts, and the 15-year-old bug story
Brian shows a dashboard of batch runs and explains Mozilla used real recently shipped security bugs (opened earlier than normal) to make the system concrete. They discuss identifying very old issues via Bugzilla IDs and using Claude Code for difficult Git archaeology across renames and history. The key distinction from earlier AI reports is the presence of a working reproducer artifact, not just a write-up.
- •Batch runs produce traces and artifacts; demo data is partially fabricated but based on real bugs
- •Bugzilla ID ranges reveal age; six-digit IDs indicate very old bugs
- •Agents help with semantic blame/archaeology across file moves and renames
- •Notable: finding long-lived, hard-to-discover bugs that bounty hunters also chased
- 20:10 – 23:32
Why repro files changed everything: the ‘legend’ element example
They dissect a concrete case where the agent took 14 attempts before producing a valid exploit path. The crucial output is a complex HTML page that reliably triggers a heap use-after-free, making the report immediately actionable for engineers. Claire underscores how browsers must handle decades of web quirks, making automated, iterative testcase generation especially valuable.
- •Agent iterates through many hypotheses; persistence yields eventual success
- •Workflow: map WebIDL → C++ implementation → reachable web entry → testcase
- •Output includes both analysis and a reproducer HTML file
- •Reproducer enables fast validation and eliminates prior “slop report” burden
- 23:32 – 26:49
Open-sourcing the tooling and the importance of a crystal-clear verification signal
Brian notes Mozilla open-sourced parts of the tooling for researchers and emphasizes that their success relies on an unambiguous verification signal (fuzzing builds with AddressSanitizer). Claire generalizes the lesson: many orgs lack crisp success/failure definitions, and agents force teams to articulate evaluation criteria. This becomes a broader capability that benefits the whole project.
- •Mozilla open-sourced the harness/tooling used in the workflow
- •ASan/fuzzing provides a binary, high-confidence verification signal
- •Other domains (web apps/distributed systems) must design equivalent verification methods
- •Clear outcome definitions are a prerequisite for effective agent loops
- 26:49 – 30:55
Patching agent meets reality: simple fixes, but humans still expand and review
They discuss moving beyond detection into automatic patch proposals and verification (apply patch, rebuild, confirm crash gone). A highlighted bug shows a complex discovery with a simple validation fix, yet humans broaden the fix to similar sites and ensure architectural correctness. The limitation is that agents optimize for the narrow task unless explicitly guided to generalize across the codebase.
- •Patching agent can propose fixes and verify via rebuild + rerun testcase
- •Agents can find complex issues yet suggest surprisingly small patches
- •Human experts extend fixes to related code paths and enforce best practices
- •Today’s tooling often lacks ‘class-wide’ remediation across the whole codebase
- 30:55 – 32:39
Supply chain and open source realities: coordination is still a human network problem
Claire and Brian zoom out to open source maintenance and supply chain security. Brian explains Firefox depends on many upstream projects, each with its own processes and threat models; reporting and remediation requires relationships and careful coordination. This aspect is harder to automate than finding bugs inside a single repository.
- •Open source scale and longevity complicate agent adoption and governance
- •Supply chain vulnerabilities require coordination across many independent projects
- •Different projects have different reporting preferences and threat models
- •Personal relationships and trust remain central to responsible disclosure
- 32:39 – 35:52
Live demo: V1 ‘just run Claude Code,’ V2 full harness with tools and verifier
Brian demonstrates how simple the starting point is: a script that runs Claude Code/Codex non-interactively with JSON output and a prompt to analyze a file. He then shows the fuller harness run with tool access, a verifier returning structured JSON, and a patching agent that fixes and validates. Claire highlights the demystification: you can start small and iterate toward a robust pipeline.
- •V1: run Claude Code/Codex exec with a prompt—no tools, just file analysis
- •Use JSON streaming mode so another program can orchestrate runs
- •V2: agent SDK + toolset + verifier produces structured, trustworthy outputs
- •Results can be pushed into a bug tracker; patch agent can confirm crash regression is fixed
- 35:52 – 41:03
Scaling the hunt: LLM-based scoring to prioritize which files to scan
Brian explains the practical constraint: you can’t ‘canvas the whole repo’ for Firefox, so they score files using a simple LLM judge. The judge estimates (1) likelihood of memory safety issues and (2) web reachability, then combines with heuristics like prior runs and duplicates. Claire generalizes the pattern to tech debt, UX, and performance prioritization in large monorepos.
- •Pre-scoring narrows scope by estimating risk and exploitability/accessibility
- •Two key scores: memory-safety likelihood and reachability from web content
- •Augment with heuristics: prior scan frequency, duplicates, historical yield
- •Same pattern can apply to commit scanning, tech debt prioritization, or performance work
- 41:03 – 48:28
Mobilizing 100+ engineers, 500 fixes, and the closing lightning round
Brian describes the internal response: a large Slack channel, coordinated teams, and significant reprioritization driven by high-quality, actionable reports. In lightning round, he attributes success to both model and harness (roughly 50/50) and shares cautious optimism that defenders can get closer to ‘zero bugs.’ He closes with pragmatic prompting habits and where to find Mozilla’s work.
- •Incident-response-like mobilization: ~100 engineers landing fixes
- •Actionable, low-false-positive reports made high-volume fixing feasible
- •Impact split is both model + harness; major runway remains for harness innovation
- •Cautious optimism: AI raises discovery capability for defenders too
- •Prompting style varies by task; practical tricks for steering coding agents
