How I AIEvals, error analysis, and better prompts: A systematic approach to improving your AI products
CHAPTERS
Hamel Husain’s core premise: quality comes from looking at data
Hamel frames AI product quality as a data analysis problem, not a “prompt magic” problem. The twist with LLMs is that the data is messy, stochastic, and often multi-step, but the fundamentals of product analytics still apply.
- •High-quality AI products start with examining real data, not vibes
- •Traditional PM skills (SQL, metrics, spreadsheets) still matter
- •AI adds complexity: non-determinism and more unstructured inputs/outputs
- •Systematic improvement requires instrumentation and review loops
Case study setup: Nurture Boss virtual leasing assistant and why it’s hard to scale
Hamel introduces a real client, Nurture Boss, an AI assistant handling inbound leasing communications across channels (SMS, email, web chat). The team’s pain: prompt tweaks felt risky because they couldn’t tell if changes improved overall behavior or broke something else.
- •Product context: AI assistant for property managers/leasing workflows
- •Multiple channels and tasks (questions, appointments, tours, handoffs)
- •Early prototype worked in demos but failed in real-world edge cases
- •Main challenge: no reliable way to measure improvement beyond gut feel
Traces and observability: capturing what actually happened in an AI interaction
Hamel explains “traces” as the log of end-to-end AI behavior: system prompt, user message(s), tool calls, tool responses, and final assistant output. He shows how observability tools (e.g., Braintrust, Arize Phoenix) make these sequences inspectable.
- •A trace includes prompts, multi-turn context, retrieval/tool calls, and outputs
- •Essential for debugging chatbots/agents where behavior is a chain of events
- •Tools differ, but the goal is the same: reviewable, queryable interaction logs
- •Traces turn fuzzy failures into concrete artifacts you can analyze
Reality check from user logs: vague, typo-filled, ambiguous prompts
Reviewing real conversations reveals how differently users interact than builders expect. Hamel and Claire highlight an example where the user’s message is unclear, and the assistant responds with something plausibly helpful but likely wrong for the user’s intent.
- •Real users write ambiguous text (typos, slang, incomplete questions)
- •Builder testing is biased: we ask “good” questions with clear intent
- •Seeing real distributions changes what you should test and simulate
- •Synthetic data may miss messy real-world patterns unless guided by logs
Error analysis, step 1 (open coding): write quick notes on the first upstream failure
Hamel introduces error analysis as a simple, high-leverage process borrowed from classical ML practice. The first step is “open coding”: sample ~100 traces and write short notes describing what went wrong, stopping at the earliest (most causal) error in the chain.
- •Process is accessible: manually review a sample and jot one-sentence notes
- •Stop at the most upstream issue to avoid downstream noise
- •Upstream errors propagate (bad intent → bad tool call → bad response)
- •This makes the problem tractable and yields fast insight
Error analysis, step 2: categorize notes and count—turn observations into priorities
After collecting notes, you bucket them into issue categories (optionally with LLM help) and then quantify frequency. Counting transforms qualitative review into a prioritized roadmap of fixes, replacing paralysis with clarity.
- •Use LLMs to help cluster notes, but iterate and correct categories
- •Quantify by counting category frequency—classic PM leverage
- •Outcome: a ranked list of the biggest failure modes to tackle first
- •Creates confidence in what evals and fixes are worth doing next
Building custom annotation UIs: reducing friction for faster review and labeling
Hamel explains that off-the-shelf observability UIs are helpful but sometimes too generic or slow for high-throughput review. For Nurture Boss, they quickly “vibe coded” a tailored annotation tool with filters, channel views, and lightweight labeling to speed analysis.
- •Goal: remove friction so humans can annotate quickly and consistently
- •Custom UI can be simple: filters by channel, annotated/unannotated, stats
- •Annotation needs to be human-readable and workflow-aligned
- •Fast labeling becomes the foundation for better evals and iteration
Impact of the process: clients get immediate quality gains and clearer next steps
Hamel notes many clients find error analysis alone transformative—sometimes enough to ship meaningful improvements. It also prevents premature obsession with eval tooling by clarifying which evals matter and what “good” looks like for real failures.
- •Even a few hours of review can surface major systemic issues
- •Teams stop guessing and start targeting the biggest sources of failure
- •Error analysis informs what evals to write (instead of infinite possibilities)
- •Reduces reliance on user thumbs-up/down feedback as the main signal
Choosing evaluation types: code-based checks vs reference-based vs subjective judging
With prioritized failures in hand, the next step is writing evaluations that match the problem type. Hamel distinguishes deterministic/code-based evals (unit-test-like), reference-based evals (known right answers), and LLM-judge evals for subjective or nuanced criteria.
- •Code-based evals catch concrete issues (formatting, leakage, policy rules)
- •Reference-based evals work when “correct answer” is known/derivable
- •LLM-as-judge is for subjective tasks like handoff quality or tone constraints
- •You still need representative test cases from traces or synthetic generation
LLM-as-a-Judge done right: binary, task-specific, and validated against human labels
Hamel critiques vague dashboards (helpfulness/truthfulness scores) as hard to interpret and easy to misuse. Instead, he recommends judge prompts that output binary pass/fail for specific failure modes, and validating judge agreement with hand-labeled examples to build trust.
- •Avoid abstract scalar scores that don’t map to actionable fixes
- •Create problem-specific binary outcomes (e.g., ‘handoff correct: yes/no’)
- •Validate the judge by comparing to human labels to measure agreement
- •Without validation, dashboards can destroy stakeholder trust in evals
Improving prompts and system instructions: fix obvious gaps, then iterate systematically
Once evals reveal failure clusters, teams decide what to change—prompting, retrieval, examples, tool specs, or eventually fine-tuning. Hamel emphasizes there’s no universal prompt trick; progress comes from targeted experimentation guided by measured errors (e.g., missing today’s date causing scheduling mistakes).
- •Use eval results to decide whether it’s prompting, retrieval, tools, or data
- •Common quick wins: missing context (like current date) and unclear rules
- •Prompt edits are “bug surface area”—small wording can cause big behavior shifts
- •Fine-tuning becomes easier/cheaper once eval + data curation pipelines exist
Evaluating and analyzing agents: tool-to-tool handoffs, transition matrices, and workflow insight
For agentic systems, Hamel highlights advanced analysis techniques like mapping transitions between steps and identifying where failures cluster. Claire adds that the same telemetry helps product discovery—seeing where users seek value and where workflows bottleneck.
- •Agent evaluation must consider sequences, not just single responses
- •Transition matrices help locate where chains break (handoff points)
- •Debugging overlaps with product analytics: what are users trying to do?
- •Notebook-style, AI-assisted analysis can speed iteration for PMs and teams
Hamel’s personal AI workflows: Claude Projects, Gemini for video, and a monorepo for prompts
Hamel shares how he runs his business with AI: Claude Projects for repeated tasks (proposals, copywriting, course FAQs, legal), Gemini for turning videos into consumable notes, and a GitHub monorepo that centralizes prompts, rules, content, and tooling to avoid vendor lock-in.
- •Claude Projects as reusable ‘work cells’ with instructions + examples
- •Proposal automation: feed call transcripts, then lightly edit outputs
- •Gemini excels at video/transcript-to-text artifacts (slide summaries, notes)
- •GitHub monorepo stores prompts/tools/content + agent rules for portability
Who should do annotation & a practical writing prompting tip: outline → draft → edit inline
In the closing lightning round, Hamel argues subject matter expertise is central—often PMs, sometimes ops/function experts, and data scientists as analysis scales. For writing quality, he recommends stepwise workflows and tools that support inline editing to create better examples and preserve human voice.
- •Best annotators are SMEs with ‘taste’ for what good looks like
- •PMs often act as SMEs; ops teams can contribute domain expertise
- •Data scientists become useful as evaluation/analysis sophistication grows
- •Writing tip: outline first, draft small sections, edit inline to teach the model
