Aakash GuptaHow to Build AI Evals in 2026 (Step-by-Step, No Hype)
CHAPTERS
- 0:00 – 2:21
Step-by-step promise: building application-specific AI evals on real production data
Aakash sets the stage with Hamel Husain and Shreya Shankar: this will be a concrete, end-to-end walkthrough of AI evals using real traces, not theory. They preview the core ingredients—observability, trace review, and error analysis—as the foundation for improving AI features in production.
- •Goal: teach evals "step by step" using real, messy data
- •Evals framed as a critical skill for PMs shipping AI to production
- •Preview of why traces matter more than generic scores
- •Set expectations: practical process over hype
- 2:21 – 3:11
Do you really need evals? Debunking the 'no evals' narrative
They address a common controversy: some claim certain AI products (e.g., coding agents) can skip evals. Shreya argues that most real applications—especially those beyond naïve foundation-model usage—still require evals or an equivalent rigorous improvement loop.
- •Why the 'Claude Code doesn't use evals' claim is misleading
- •Upstream testing may exist in some domains (e.g., code), but doesn’t generalize
- •Most real apps have app-specific requirements that need app-specific evals
- •Evals are fundamentally about systematic product improvement
- 3:11 – 5:12
Case study setup: Nurture Boss and the realities of production AI (tools, RAG, multichannel)
Hamel introduces Nurture Boss, an AI assistant for property managers handling tenant/prospect conversations. The product is a strong evals example because it includes real-world complexity: tool calls, RAG, multi-turn dialogue, and multiple interaction channels.
- •What Nurture Boss does (leasing, scheduling, answering questions)
- •Why it’s a realistic eval target: tool calls, RAG, multistep workflows
- •Multi-channel UX: voice, SMS, web chatbot
- •Objective: move beyond vibe checks to systematic improvement
- 5:12 – 7:34
Start with observability: what traces are and how to log them (tools optional)
They explain that the first practical step is capturing traces—what the model saw, did, and returned—so you can evaluate real behavior. Hamel emphasizes you don’t need a fancy platform: CSV/JSON logging can be enough if it’s easy to start and review.
- •Observability first: instrument the app to capture traces
- •Tooling options (BrainTrust, LangSmith, Arize) but not required
- •You can log to Datadog, CSV, JSON—simplicity matters
- •Traces enable diagnosis of failures that metrics alone miss
- 7:34 – 14:39
Trace review in action: finding UX failures humans catch (and LLMs often miss)
They read a real SMS trace and identify multiple subtle failures: misunderstanding user constraints, tool usage gaps, and formatting issues like markdown appearing in texts. They show why a human product lens is essential—LLMs can give a 'looks fine' verdict while missing key UX problems.
- •Example failures: ignoring/contradicting user request (bathroom configuration)
- •Tool-follow-through issue: says it will check but doesn’t act
- •Channel mismatch: markdown formatting in SMS output
- •Why generic helpfulness/conciseness scores won’t capture product-specific issues
- 14:39 – 16:48
Why 'just ask ChatGPT if it’s correct' isn’t enough
Shreya describes a common anti-pattern: dumping traces into a model and asking for correctness. The model may spot one obvious issue but miss nuance, misinterpret constraints, or recommend the wrong fix because it lacks your product context and preferences.
- •LLMs can validate superficially while missing nuanced UX requirements
- •Examples: missing SMS/markdown problem; overreacting to “cherry-picking” listings
- •Need for a human-in-the-loop product judgment
- •Evals should encode your team’s taste and product intent
- 16:48 – 19:47
Open coding: quickly annotating traces with notes (fast, imperfect, high leverage)
Hamel demonstrates the core workflow: scan traces and write short notes about what went wrong—without getting stuck debating root causes. They stress speed and consistency: you can review ~100 traces quickly and build a high-signal set of 'open codes' for analysis.
- •Write short notes per trace: capture what’s wrong, then move on
- •Don’t aim for perfection—catch the most important issues
- •Avoid rabbit holes and trace-by-trace debates
- •Outcome: a set of open codes that represent observed failure modes
- 19:47 – 25:30
More failure patterns from real traces: lead nurturing gaps and scheduling breakdowns
They review additional traces showing common product failures: robotic dead-ends instead of helpful follow-up, false promises (virtual tours that don’t exist), and buggy 'reschedule' flows that create duplicate bookings. These examples illustrate how eval categories should reflect real operational risks.
- •Failure mode: poor lead nurturing / no helpful next step
- •Mismatch between assistant claims and tool capabilities (virtual tours)
- •Reschedule bug: creates a second tour rather than modifying the first
- •These patterns become candidates for systematic evaluation
- 25:30 – 27:01
Error analysis begins: from open codes to themes you can act on
Hamel frames error analysis as the most skipped but most valuable evals step, especially for PMs. With a spreadsheet of notes, you already understand the system deeply; now you organize those observations to decide what to fix and what to evaluate.
- •Error analysis is the central skill, not an optional extra
- •PM/domain expertise is essential to judge real product quality
- •A sample of 100 traces can reveal the dominant failure modes
- •Next step: categorize notes to make them measurable and actionable
- 27:01 – 34:56
Axial coding: turning messy notes into clear, non-vague categories (with AI assist, then refinement)
They introduce axial coding—grouping open codes into categories—and show how an LLM can propose categories as a starting point. Shreya warns that AI-generated categories are often too vague; the team must refine labels so they’re understandable, consistent, and useful for later labeling or automation.
- •Axial coding = categorizing open-code notes into error types
- •LLMs can bootstrap categories, but initial suggestions may be too broad
- •Refine categories to be specific enough for consistent labeling
- •Add "none of the above" to reveal missing categories and iterate taxonomy
- 34:56 – 38:28
Counting issues with pivot tables: turning qualitative review into prioritization power
Once notes are categorized, they count frequency via pivot tables to identify the most common failure modes. They highlight that prioritization isn’t purely frequency-based—PMs should weigh severity and business impact—but counting breaks paralysis and creates a roadmap grounded in reality.
- •Use counts to escape "what do we fix next?" paralysis
- •Pivot tables reveal frequent clusters like conversational flow issues
- •Severity matters: rare but catastrophic issues can outrank frequent minor ones
- •Counts create evidence for prioritization and stakeholder conversations
- 38:28 – 41:31
From findings to evals: code-based checks vs LLM-as-judge evaluators
Hamel explains that not every issue requires an LLM eval—some can be caught with cheap deterministic checks (e.g., markdown in SMS). For more subjective or contextual issues (e.g., human handoff), an LLM judge can be appropriate, but only when it supports iteration on a real problem you’ve observed.
- •Not all failures need an eval; some are straightforward fixes
- •Two eval types: code-based assertions vs LLM-as-judge for subjective criteria
- •Choose evals that you’ll iterate against (avoid evals-in-the-dark)
- •Example: formatting issues → code; handoff failures → LLM judge
- 41:31 – 47:33
Building a binary LLM judge: rubric design, iteration, and avoiding Likert traps
They show a simple LLM-judge prompt for detecting handoff failures and argue for binary outputs (true/false). Shreya explains binary is easier to align and maps to real product decisions, while range/Likert scales often create ambiguous, misaligned scoring.
- •LLM judge prompt includes: what counts as failure vs non-failure
- •Return only true/false to simplify alignment and validation
- •Binary matches real-world ship decisions (act vs don’t act)
- •Iterate on rubric; examples help but aren’t required to start
- 47:33 – 56:08
Measuring the judge: why agreement/accuracy can mislead (TPR/TNR mindset)
They warn that many teams stop after producing judge outputs, which leads to stakeholder distrust when metrics don’t match lived product quality. The fix is to validate the judge against human labels and avoid naive agreement when failures are rare; you need to understand performance on positives vs negatives.
- •Don’t stop at judge outputs—validate against human-labeled traces
- •High agreement can be meaningless if the failure rate is low
- •Ask: does the judge catch true failures and avoid false alarms?
- •If alignment is weak, revise the judge prompt and category definitions
- 56:08 – 1:04:17
Roles, workflows, and common mistakes: PM ownership of error analysis + keeping prompts accessible
They discuss how PMs, engineers, and small teams split responsibilities, emphasizing that domain experts should drive error analysis. They share examples of building internal trace-review tools and admin prompt editors, plus common pitfalls: skipping error analysis, outsourcing it, or separating the prompt from the PM.
- •Best owner for error analysis: domain expert/PM (engineers may lack context)
- •Build lightweight internal tools to reduce friction reviewing traces
- •Expose prompts in an admin UI so non-technical domain owners can iterate
- •Common mistakes: skipping error analysis, outsourcing it, over-relying on vendor metrics
- 1:04:17 – 1:07:00
Wrap-up: where to learn more, and how to start if you’re pre-production (dogfooding + synthetic data)
They close with resources (evals.info) and a practical note: if you don’t yet have production traces, you still need data. Start by dogfooding with friends; if that’s not possible, generate synthetic user interactions carefully to bootstrap your evaluation loop.
- •Resource: evals.info + finding the hosts online
- •If not in production: recruit testers and dogfood to create traces
- •Alternative: synthetic user simulation at scale (done carefully)
- •Core message: real data + error analysis is the fastest path to better AI UX
