How I AIEvals, error analysis, and better prompts: A systematic approach to improving your AI products
CHAPTERS
- 0:00 – 1:39
Why AI product quality starts with looking at real data (not vibes)
Claire tees up the core challenge of AI products: non-deterministic outputs that are hard to debug and improve systematically. Hamel argues the single most important habit is the same as pre-AI product work—look at data—then shows how that mindset adapts to LLM systems.
- •AI product teams often “vibe check” prompts without knowing if fixes improve or regress other behaviors
- •The fundamental skill is still data analysis; the twist is the data includes conversations, tool calls, and multi-step traces
- •Goal of the episode: a repeatable process for debugging, evals, and prompt/system improvement
- 1:39 – 3:10
Sponsor break: GoFundMe Giving Funds (DAF)
Claire shares a sponsored segment about GoFundMe’s donor-advised fund product, focusing on tax benefits and zero fees. She then transitions back into the episode with Hamel.
- •Donor-advised fund with zero admin/asset fees
- •Immediate tax deduction; can donate later to many nonprofits
- •Simple hub and single tax receipt; option to transfer an existing DAF
- 3:10 – 5:33
The fundamentals for AI PMs: applying classic analytics skills to LLM products
Claire and Hamel align on why PMs’ existing analytics toolkit (SQL, metrics, spreadsheets) is still essential. Hamel frames the rest of the conversation as translating that competence into the new “AI-shaped” data surface area.
- •Data literacy is table stakes for modern PMs—AI raises the stakes further
- •AI systems require analysis beyond standard metrics because behaviors emerge in interactions
- •Systematic improvement starts with observable evidence, not intuition
- 5:33 – 6:35
Case study setup: Nurture Boss virtual leasing assistant and the “prompt tweaking” trap
Hamel introduces Nurture Boss, an AI assistant for property managers, and the problems they faced after prototyping. The team felt stuck because prompt changes were guesswork and could break other parts of the experience.
- •Product context: top-of-funnel leasing support (tours, questions, scheduling, handoff)
- •Common scaling pain: fixes feel local and unmeasured; teams don’t know if quality is improving overall
- •Need for a method to find failure modes and prioritize what to fix first
- 6:35 – 13:12
Tracing real interactions: system prompts, tool calls, and messy user messages
Hamel explains traces—logged, end-to-end records of chatbot turns and internal actions—and walks through real examples. The traces reveal surprising user behavior (typos, ambiguity) and show how tool calls and hidden context affect outputs.
- •Traces capture multi-turn conversations plus retrieval/tool calls and their results
- •Real user inputs are often vague or malformed, unlike developer test prompts
- •Observing tool selection and intermediate results helps diagnose why the assistant responded incorrectly
- 13:12 – 14:31
From overwhelming logs to a method: error analysis as the bridge
Claire highlights why this data is hard to query with traditional SQL-style questions, given ambiguity and scale. Hamel introduces error analysis as a long-standing ML practice that makes the problem tractable.
- •Classic analytics questions (“find ambiguous prompts”) are hard without labeling
- •LLM failures can look “kind of helpful” while still being wrong for the task
- •Error analysis provides a simple workflow to turn messy qualitative data into actionable categories
- 14:31 – 17:05
Error analysis step 1: open coding (write short notes on the first upstream issue)
Hamel details the first step: randomly sample traces, read until you hit an issue, and write a one-sentence note about the most upstream error. This “journaling” approach reduces complexity and focuses on causal root problems that create downstream failures.
- •Open coding = quick human notes describing what went wrong
- •Sample ~100 traces; stop at the earliest/root issue rather than chasing downstream symptoms
- •Upstream focus is a pragmatic heuristic for faster, higher-leverage fixes
- 17:05 – 17:35
Error analysis step 2: categorize and count to get a prioritized fix list
After collecting notes, Hamel recommends bucketing them into categories—often with LLM help—then using simple counting to identify the biggest issues. The result is clarity: teams know what to address first and can write targeted evals.
- •Use LLMs to help cluster notes into categories, with human iteration to refine buckets
- •Counting remains powerful: frequency turns qualitative notes into quantitative priority
- •Example outputs: handoff/transfer problems, tour scheduling errors, missing follow-ups, incorrect info
- 17:35 – 23:26
Building custom annotation tools to remove friction and speed review
Hamel shows why off-the-shelf observability UIs can be too slow or awkward for rapid human review. For Nurture Boss, they “vibe coded” a lightweight annotation app with filters (channel, annotated status) to accelerate labeling and analysis.
- •Goal: maximize throughput and readability when reviewing many traces
- •Custom UI can mirror existing tools but tailor workflow (filters, summary stats, fast note entry)
- •Faster annotation enables more reliable categorization and better downstream eval datasets
- 23:26 – 29:33
From error analysis to evaluations: choosing the right eval types (code vs LLM judge)
Hamel argues error analysis should come before eval design—otherwise there are infinite evals and no grounding. He then distinguishes reference/code-based tests from subjective evaluations that need an LLM-as-a-judge, plus the role of synthetic data.
- •Error analysis tells you what evals to write (e.g., tour scheduling correctness, handoff quality)
- •Code-based/unit-style evals work for objective checks (e.g., UUID leakage)
- •Subjective tasks may require an LLM judge; test cases can come from traces or synthetic generation
- 29:33 – 34:39
LLM-as-a-Judge done right: binary, task-specific, and validated against human labels
Hamel critiques generic dashboards (helpfulness/truthfulness scores) as non-actionable and trust-eroding. He outlines a more reliable approach: create narrow, binary judges for specific failure modes and validate them against hand-labeled data to ensure agreement.
- •Avoid vague, scalar “helpfulness” scores that don’t map to concrete fixes
- •Design judges with binary outcomes for a specific behavior (“pass/fail tour scheduling”)
- •Calibrate and audit judges by comparing to human annotations to prevent loss of stakeholder trust
- 34:39 – 38:15
Improving prompts/system instructions (and when fine-tuning becomes worthwhile)
With evals in place, Hamel describes how teams decide what intervention to try—prompting, examples, retrieval fixes, or occasionally fine-tuning. He notes many early wins are simple (like adding today’s date) and that solid eval infrastructure makes fine-tuning far more practical.
- •No universal “magic prompt”; improvement is iterative and guided by measured failures
- •Common low-hanging fixes: missing context like today’s date causes scheduling errors
- •Retrieval is often the Achilles heel; fine-tuning is usually later—but becomes easier once eval + data pipelines exist
- 38:15 – 41:35
Evaluating agent workflows: mapping step-to-step failures with transition matrices
Hamel shares a more advanced lens for agentic systems: analyzing where errors occur across tool/tool or step/step transitions. Claire also highlights how the same analysis can support product discovery by showing where users seek value.
- •Agent evals can focus on handoffs between steps (e.g., “generate SQL” → “execute SQL”)
- •Transition matrices help locate concentrated failure points in multi-step workflows
- •Beyond debugging: workflow analytics can reveal user intent patterns and roadmap opportunities
- 41:35 – 47:58
Hamel’s personal AI stack: Claude Projects, Gemini for video-to-notes, and a monorepo for prompts
Hamel walks through how he runs his business with AI: multiple Claude Projects for writing and operations, plus a toolchain that converts YouTube videos into annotated slide summaries using Gemini. He also organizes prompts, content, and tools in a GitHub monorepo to avoid vendor lock-in.
- •Claude Projects for repeatable workflows (proposals, course FAQ generation, copywriting, legal support)
- •Proposal workflow: feed call transcript → draft proposal → quick human edit
- •Gemini excels at video/transcript processing to produce slide-by-slide annotated notes
- •GitHub monorepo centralizes prompts, content, and tools; AI agents are pointed at the repo with guiding rules
- 47:58 – 54:48
Lightning round: who should do annotation, and a practical writing prompting tactic
Hamel explains annotation needs “taste” and domain expertise, often coming from PMs or SMEs, with data scientists helping as analysis scales. He closes with a writing workflow: provide examples, outline first, draft iteratively, and use tools that let you edit outputs inline to teach the model your style.
- •Best annotators are SMEs; PMs often act as the SME proxy for product quality
- •Role specialization still matters at scale (PM, AI engineer, data scientist)
- •For writing: outline → draft sections → careful edits; inline editing creates immediate examples the model can follow
