Aakash GuptaThe Most Important New Skill for Product Managers in 2026: AI Evals Masterclass
CHAPTERS
- 0:00 – 2:03
AI features fail without evaluations: why the model isn’t the main problem
Ankit opens with the core claim: AI products fail when teams ship without measuring quality and behavior. The conversation frames evals as the key new skill for PMs heading into 2026, because without evals the product team can’t tell whether the system is trustworthy.
- •AI feature failures often stem from missing evaluation, not model choice
- •Shipping without evals creates a false sense of product quality (“your product is lying to you”)
- •Evals are positioned as a critical PM capability for 2026
- •Episode promises actionable frameworks and real examples beyond intro-level content
- 2:03 – 4:23
What makes this masterclass different: practical nuance + real case studies
Aakash asks what’s unique compared to other evals content. Ankit explains most material is introductory and hypothetical; this episode focuses on concrete examples, a reusable framework, and the nuances needed to apply evals across product types.
- •Most existing evals content is beginner/intermediate and lacks real examples
- •Evals aren’t one-size-fits-all; nuance matters by product context
- •Goal: give a repeatable framework for strong evals across use cases
- •Agenda preview: AI product components → eval basics → end-to-end build process → tips + case study
- 4:23 – 5:54
The 5 components of a GenAI product—and why stochastic behavior demands evals
Ankit breaks down the building blocks of GenAI products (model, context engineering, tools, orchestration, UX) and highlights how non-determinism changes product risk. Because the same input can yield different outputs, teams need guardrails and measurement to ‘tame’ the system.
- •GenAI stack: language model, context engineering (prompts/RAG), tools, orchestration, UX/humans-in-loop
- •LLMs are non-deterministic (stochastic), unlike traditional deterministic software
- •PM’s job is to ‘tame’ behavior to deliver a consistent experience
- •Evals become the mechanism to control and validate this variability
- 5:54 – 10:24
Case study: AI-first job website—what an eval looks like in practice
Using an AI job website example, Ankit shows how to evaluate generated summaries, interview questions, skills, and learning guides. He demonstrates that evals can be coded checks (e.g., length) or subjective quality checks judged by humans or an LLM-as-judge prompt.
- •Example product: crawl job portals → LLM transforms JD into summaries/questions/skills/learning guide/quiz
- •Need to validate factuality, relevance, helpfulness, and constraints (e.g., summary length)
- •Eval methods vary: code checks, human review, LLM judge, or hybrid
- •Evals support prompt iteration and model selection (quality vs cost tradeoffs)
- 10:24 – 13:43
Why prototypes fail to scale: five failure modes (and where evals help)
Ankit explains why impressive AI prototypes often break in production: drift, costs, engineering limits, missing guardrails, and collaboration failure. He then maps how evals mitigate drift and cost/model selection, strongly support guardrails, and indirectly improve collaboration through SME involvement.
- •Five failure reasons: data drift, cost, engineering limitations, missing guardrails, collaboration failure
- •Evals help detect drift via continuous monitoring/observability
- •Evals enable confident downgrades to cheaper models while preserving quality
- •Guardrails (what the model must/ must not do) are clarified and enforced via evals
- •SME involvement in eval design improves shared understanding and user empathy
- 13:43 – 19:47
Cost & model strategy: using evals to justify cheaper models and specialization
The conversation dives into token pricing and the large cost spread between top-tier and small models. Ankit and Aakash discuss leveraging evals to compare models, plus techniques like transfer learning and fine-tuning for narrow use cases to reduce cost without sacrificing performance.
- •Prototype teams often default to best/most expensive model; production economics differ
- •Evals let teams verify whether cheaper models perform ‘well enough’
- •Token pricing differences can be dramatic (order-of-magnitude spreads)
- •Alternatives include open models and specialized tuning/transfer learning
- •Key logic: prove quality equivalence via evals before switching models
- 19:47 – 22:04
Non-determinism explained with the ‘chai’ analogy: correctness isn’t enough
Ankit uses tea/chai to illustrate that even correct answers can vary in a way that affects user satisfaction. The point: hallucinations may decrease, but preferences, context, and desired experience still require PM-defined standards and evals to ensure consistent value.
- •Different ‘teas’ can all be tea, yet deliver different experiences
- •Even factual correctness doesn’t guarantee the right user experience
- •Customers have different needs and contexts; models don’t inherently optimize for them
- •PMs define what ‘good’ means through explicit expectations and evals
- 22:04 – 35:04
Building evals end-to-end: from success criteria to datasets to offline/online loops
Ankit introduces a complete workflow for building evaluations, emphasizing that most effort goes into building representative datasets and defining metrics. The system becomes a loop: define expectations, build a baseline, evaluate against curated inputs, create evals from observed failures, iterate prompts/models/tools, and monitor in production.
- •Start with success criteria + expected behavior; translate into measurable metrics
- •Build a base product (system prompt, model, tools, context, orchestration)
- •Create a high-quality dataset (logs, research, synthetic data, SME input)
- •Run dataset through baseline → review outputs with SMEs → derive eval metrics
- •Choose evaluation methods: code vs human vs LLM judge vs hybrid
- •Use offline evals pre-release and online evals for production monitoring; iterate continuously
- 35:04 – 36:52
Offline evals as the AI PRD: hill-climbing quality before shipping
Aakash reframes offline evals as the effective PRD for AI engineers: PMs define tests and target behaviors, engineers improve the system until it passes. Ankit emphasizes that without good offline evals, you don’t have a launchable product.
- •Best practice: PM-defined evals become the spec/PRD for AI engineering work
- •Engineers optimize prompts/models/orchestration to raise eval pass rates
- •Offline evals are pre-release gates (not ‘unimportant’ because they’re offline)
- •Traditional edge/corner cases become explicit eval cases
- •Aim for known risks covered before exposing real users
- 36:52 – 37:10
Online evals & observability: drift detection, sampling, latency percentiles, and feedback
Ankit explains production monitoring: run evals on sampled live traffic, track drift, and use observability tools. He also adds user feedback loops (thumbs up/down and behavioral signals) plus latency metrics like P50/P95/P99 to avoid misleading averages.
- •Online evals run on production data via sampling (e.g., 1/10, 1/100) to manage cost
- •Observability platforms (e.g., Arize, TruLens) support monitoring and alerting
- •Define thresholds (e.g., compliance or accuracy falling below targets triggers action)
- •Latency measurement should use percentiles (P50/P95/P99), not averages
- •Incorporate hard feedback (thumbs up/down) and soft feedback (retries, escalation, session behavior)
- •A/B test prompts/models, then evaluate with both eval scores and user outcomes
- 37:10 – 39:23
Case study deep dive: INDMoney Mind / Robinhood-style stock research assistant
Ankit reverse-engineers a fintech assistant that answers stock questions while staying compliant (informational, not advisory). He demonstrates how to turn regulations and product expectations into metrics, how to structure prompt principles, and how evals enforce constraints like groundedness, accuracy, and ‘no buy/sell advice’.
- •Use case: AI answers contextual stock questions inside a trading app
- •Define expected behavior: concise answers, grounded facts, no direct buy/sell recommendations
- •Convert expectations into evaluation dimensions (accuracy, compliance, relevance, safety, latency)
- •Treat product levers as variables: system prompt, model, tools (web search), context, orchestration
- •Dataset creation uses logs, research, SMEs, and synthetic generation to cover edge cases
- •Set explicit thresholds and deployment block criteria for high-stakes domains
- 39:23 – 57:56
Turning the case study into an eval PRD: layers, datasets, and evaluation types
The discussion shows how to document an AI product for stakeholders: layers (UI, orchestration, retrieval, LLM, analytics), prompt principles, context injection, and evaluation plans. Ankit details dataset structure (expected outputs, red flags, priorities) and a 3-part evaluation approach: programmatic, LLM judge, and human protocols.
- •Write PRD sections: user/business value, system layers, prompt principles, and metrics definitions
- •Keep prompts modular for easier iteration (edit specific lines vs giant prompt blocks)
- •Add domain context to reduce hallucinations and enable cheaper model use
- •Dataset rows include question type, context, expected answer traits, and red flags
- •Evaluation stack: programmatic checks (numbers/structure), LLM-as-judge (tone/relevance), human review for high-stakes/edge failures
- •Human ratings: start pass/fail, later add 1–5 with rationale for iterative improvement
- 57:56 – 1:01:14
Real-world eval examples + misconception: evals are not ‘QA rebranded’
Ankit addresses the misconception that evals are just QA with a new name, arguing PMs use evals to transform the product with SMEs and iterative improvements. He shares examples from tools like Grammarly, GitHub Copilot, Klarna, and support chatbots to show how small failures can cascade into broken products or missed business outcomes.
- •Evals differ from QA: PMs use evals to drive product transformation, not just report bugs
- •Reliability/trust example: tone errors (e.g., Grammarly) can create large downstream impact
- •GitHub Copilot anecdote: small config/YAML issues can break production without proper eval coverage
- •Klarna: evals should include business metrics and funnel outcomes, not only ‘helpfulness’
- •Support chatbots risk serving outdated policy without ongoing evals and refresh cycles
- 1:01:14 – 1:03:59
Key takeaways: evals as mandatory guardrails and a continuous practice
Ankit and Aakash recap the full arc: non-determinism makes evals essential, and evals can be code-based, LLM-judged, or human/hybrid. They reinforce the idea that evals are the operating system for shipping and maintaining AI features—never set-and-forget.
- •Evals are not optional; they are guardrails for AI-driven outcomes
- •Combine evaluation methods: code, LLM judge, and human-in-the-loop
- •Offline evals guide development; online evals keep quality stable amid drift
- •Use AI to help write prompts, but humans define what ‘good’ means
- •Continuous iteration beats one-time evaluation setup
