Aakash GuptaThe ONE AI Skill Every Product Manager NEEDS in 2026
CHAPTERS
- 0:00 – 5:25
Why AI evals are a must-have PM skill: taste, iteration, and scale
Hamel lays out why product managers need to master AI evaluation: it puts PM judgment directly into the build loop, enables rapid iteration with structured feedback, and scales product taste across many AI workloads. He reframes evals from “monotonous annotation” into high-leverage product influence.
- •Evals operationalize PM taste and customer context in the engineering critical path
- •Structured evals enable faster iteration than ad-hoc “vibe checks”
- •Evals let you scale judgment across multiple AI systems and teams
- •Trust in evaluators is essential—otherwise dashboards become ignored
- 5:25 – 7:37
Defining “evals” and why they’re the hardest part of AI products
Shreya defines evals as systematic measurements of quality with a chosen criterion and measurement method. The group argues that getting evals right often solves the hardest part of building AI products because the process of creating them forces clarity, data review, and disciplined iteration.
- •Eval = criterion of quality + method of measurement (human, heuristic, model-judge, etc.)
- •Real products need a suite of evals (often 3–10), not a single metric
- •The value is in the process of building evals, not just the numbers
- •Without evals, most AI products can’t survive multiple iterations
- 7:37 – 8:53
Why “vibe checks” don’t scale—and turning judgment into rubrics
Shreya explains why relying on PM/engineer intuition doesn’t scale: it’s hard to teach, reproduce, or onboard. Evals translate subjective taste into explicit rubrics with pass/fail criteria and examples, and teams can measure alignment across reviewers.
- •Vibe checks matter but are unscalable and hard to transfer to others
- •Convert taste into explicit rubrics with examples of pass vs fail
- •Operationalizing judgment enables repeatable iteration and delegation
- •Alignment measurement across reviewers becomes part of the system
- 8:53 – 14:11
Binary evals over 1–5 ratings: clarity, calibration, and LLM-judge reality
Hamel and Shreya argue that binary pass/fail is usually superior to 1–5 scales because it forces decisive shipping thresholds and avoids ambiguous averages. They connect this to LLM-as-judge constraints: off-the-shelf judges align more reliably to binary criteria than fine-grained scales without extensive tuning and calibration.
- •1–5 ratings are hard to interpret and calibrate; averages can be meaningless
- •Binary forces teams to define “good enough to ship”
- •LLM-as-judge works better with clear pass/fail examples than multi-level scales
- •Using 1–5 often signals unclear thinking about requirements
- 14:11 – 16:26
Skepticism + scientific method: avoiding whack-a-mole iteration
They stress that evals should be grounded in experiments, measurement, and skepticism rather than trust in generic metrics or vendor dashboards. Hamel uses a “whack-a-mole” analogy: prompt tweaks without evals lead to circular progress, while evals create systematic improvement loops.
- •Don’t trust claims; validate via experiments and recorded measurements
- •Generic vendor hallucination/toxicity scores often fail in specific domains
- •Without evals, teams fix one issue and create another (whack-a-mole)
- •Evals enable structured iteration rather than ad-hoc prompt hacking
- 16:26 – 19:34
Case study: hallucinated sales emails—and why domain-specific evals work
Aakash shares an Apollo.io example where GPT-generated emails hallucinated details. Hamel explains why generic hallucination metrics fail and how domain-specific labeling plus an iteratively validated LLM judge creates scalable, trusted quality control.
- •Start by observing and labeling real failure modes in your domain (what “hallucination” means here)
- •Craft an LLM judge prompt tailored to those domain failure modes
- •Iteratively validate judge agreement with human labels to build trust
- •Untrusted dashboards kill AI products because teams stop using evals
- 19:34 – 22:19
From classic ML to LLMs: what Airbnb taught about evaluating stochastic systems
Hamel connects LLM evaluation to longstanding ML evaluation practices: stochastic, non-deterministic systems require measurement discipline. He describes major Airbnb ML applications (fraud, search ranking, LTV) and positions the course as a shortcut to the evaluation mindset without a full ML curriculum.
- •LLMs behave like ML systems: stochastic outputs demand evaluation frameworks
- •Airbnb ML examples: fraud detection, search/recommendations, LTV marketing allocation
- •Many PMs haven’t been exposed to ML-style evaluation discipline
- •Goal: teach evaluation skills focused specifically for LLM products
- 22:19 – 25:22
Evaluating search and RAG: changing metrics when the consumer is an LLM
Shreya and Hamel discuss how search evaluation ideas transfer to RAG. The key shift: the consumer of retrieval results may be an LLM with long context windows, changing tolerances (e.g., recall@500). Retrieval can be evaluated like classic IR, while generation needs its own evals.
- •RAG splits into retrieval eval (classic search/IR) and generation eval
- •LLMs can consume many more retrieved results than humans (long context)
- •Metrics like recall@k may use much larger k (e.g., 500) for LLM consumers
- •Human-facing outputs still need precision/quality measurement
- 25:22 – 31:25
Code generation evals: verifiable domains, test harnesses, and why dev tools lead
Hamel explains why developer tools were early AI winners: developers are the domain experts and code is relatively verifiable. He describes how large-scale test harnesses and benchmarks enabled rapid iteration toward Copilot-like quality, and how his earlier work contributed to code-related benchmarks.
- •Domain expert must be in the loop; for dev tools, that expert is the developer
- •Code is verifiable via tests, enabling strong automated eval harnesses
- •Upfront eval infrastructure unlocks rapid iteration and quality gains
- •Benchmarks/datasets (e.g., code retrieval) shaped early model progress
- 31:25 – 35:12
Hill climbing needs PM-defined metrics—and the risk of overfitting
They explain “hill climbing” as engineers optimizing toward metrics—only possible once metrics are defined. PMs often must define those metrics based on product context. Hamel warns that naive hill climbing can overfit to evaluation data, requiring ML-style safeguards.
- •Engineers can optimize quickly once metrics/targets exist; PMs must often define them
- •Traditional ML had well-defined metrics; LLM products often don’t
- •Overfitting = improvements that don’t generalize beyond the eval set
- •Common pitfall: leaking test cases into prompts (directly or indirectly)
- 35:12 – 38:08
Anti-overfitting toolkit: holdout sets, suspiciously high scores, and hard tests
Shreya and Hamel share practical guardrails: reserve test data you never consult during development, treat near-perfect scores as a red flag, and design tests that discriminate between good and bad. They add nuance: keep both basic regression tests and aspirational benchmarks.
- •Create labeled data and reserve a test set you never look at while iterating
- •Treat 95–100% results as suspicious; investigate leakage or weak test sets
- •Hard tests should differentiate quality; otherwise they provide no signal
- •Maintain both “must-pass” regression tests and aspirational benchmarks
- 38:08 – 42:42
What “good evals” look like in the wild: interfaces, scoped judges, and product metrics
Shreya avoids naming a single best-in-class company, noting everyone struggles, but highlights converging best practices. These include custom annotation/labeling interfaces, narrowly scoped LLM judges embedded in products, and clear success-correlated metrics like next-token prediction in coding tools (e.g., Cursor).
- •Teams gain leverage by building labeling/annotation interfaces for trace feedback
- •Deploy multiple well-scoped LLM judges rather than one generic evaluator
- •Coding tools benefit from well-defined success metrics (next-token prediction)
- •Most non-code domains lack verifiable ground truth, increasing PM responsibility
- 42:42 – 46:11
Foundation benchmarks vs business evals: why OpenAI can’t define your quality
They distinguish general-purpose model benchmarks (MMLU, HumanEval) from domain-specific business evals. OpenAI and other labs focus on the former, but cannot encode each product’s unique taste. The discussion uses writing style as a vivid example of “average taste” vs bespoke preference.
- •Foundation model benchmarks measure general capability, not product-specific success
- •Business/domain evals differ by company vision and user expectations
- •Models reflect average labeler taste; power users (e.g., writers) often dislike outputs
- •Competitive differentiation comes from encoding your taste via eval pipelines
- 46:11 – 56:04
Evals as the moat—and how fine-tuning fits the lifecycle (after evals)
Shreya argues evals (and the surrounding pipeline) are the core defensible asset in AI products. They position fine-tuning as a later-stage tool—after evals, model choice, and task decomposition—due to operational complexity and ongoing maintenance costs. They also introduce Shreya’s “Three Gulfs” framing to decide between prompting, RAG, and fine-tuning.
- •Evals + pipeline (not just a single metric) form the durable moat
- •Evals unlock faster iteration, easier model swaps, and eventually fine-tuning readiness
- •Fine-tuning should be last resort due to MLOps/maintenance burden and upgrade friction
- •Three Gulfs: prompting solves specification; RAG/fine-tuning solve generalization limits
- 56:04 – 1:15:27
Learning roadmap: error analysis, grounded theory coding, judge calibration, and productionization
They outline a structured curriculum for mastering evals, emphasizing error analysis as the central bottleneck. They describe a grounded-theory-inspired method (open coding → axial coding → saturation) for discovering failure modes, then turning them into automated evaluators (LLM judges, code checks). They also cover advanced topics like multi-turn evals, agents, CI/CD, and annotation interfaces.
- •Error analysis is the highest-leverage step: “look at your data” with structure
- •Use grounded theory methods to derive failure modes from traces (open/axial coding)
- •Iterate and validate LLM-as-judge against labeled data; measure annotator agreement
- •Advanced eval topics: multi-turn chats, RAG, tool calling/agents, production CI/CD, UI for labeling
- 1:15:27 – 1:34:32
Business of teaching evals: consulting to course, pricing, and why cohorts end
Hamel explains the organic shift into consulting and education, driven by repeated client bottlenecks around evals. They discuss course economics, discounts, and why they plan to stop running live cohorts to preserve quality while reinvesting into other formats (book, recorded content, tools). The episode closes with advice on course creation and where to find them online.
- •Consulting emerged from demand (e.g., Weights & Biases) and repeated eval pain
- •Course pricing filters for serious builders; discounts expand access when needed
- •Live cohorts are time-intensive; they aim to preserve “specialness” and reinvest proceeds
- •Course-creation advice: focus, relentless marketing experiments, strong partners, and high-quality guests
