Aakash Gupta

AI Product Metrics Interview – Execution Case Explained

We break down the complete AI product metrics framework. The 40-minute case walkthrough, visual framework approach, and why output metrics matter more than you think. Full Writeup: https://www.news.aakashg.com/p/ai-success-metrics-interview ---- Timestamps: 0:00 - Why AI Product Execution Interviews Matter 1:18 - The Underlord Case: Live Mock Interview Begins 4:29 - Why I Pulled Up The Product Live (Don't Skip This) 6:42 - The User Segmentation Push-Back 9:16 - Building The Visual Framework in Real-Time 12:52 - Value Enumeration: The 4 Core Values 17:24 - The Positive Metrics Bank 23:00 - North Star Selection: Why Exports Won 25:47 - Breaking Down The North Star (3 Vectors) 31:00 - Trade-offs & Guardrails Deep Dive 34:12 - The "Genie Metric" Curveball 35:01 - The Output Metrics Miss (My 8.5/10 Moment) 38:08 - The Power Move: Post-Interview Follow-Up Strategy ---- Key Takeaways: 1. Visual frameworks are non-negotiable - Draw your structure live. Interviewer needs to follow you for 40 minutes. Without visual anchor, even great ideas get lost. This separates you from every other candidate. 2. Always push back on user assumptions - Bart said "beginners." I challenged it. Underlord is on homepage, so metrics need to work for ALL users. This type of thinking turns 8/10 into 10/10 answers. 3. Build metrics bank BEFORE choosing North Star - Generate 15+ positive metrics first. Time, volume, adoption, discovery, engagement, retention, output. Then evaluate. Don't pick your favorite and work backwards. 4. Control for complexity in time metrics - Underlord might INCREASE time to edit because users do more. Create table: 1 tool = 3min, 2 tools = 4min. If Underlord takes longer for SAME complexity, that's a problem. 5. The output metrics mistake cost me 2 points - I forgot upgrades, renewals, referrals. Bart had to prompt with "genie metric" question. Always include input metrics AND output metrics. Non-negotiable. 6. North Star selection needs explicit reasoning - Don't just pick. Evaluate out loud. "Time to export doesn't work because... Number of tools is hard to operationalize because... Number of exports works because..." Show your math. 7. AI guardrails are different from traditional metrics - Hallucination rate. Support requests: no increase. Build evals that verify AI actually did what it claimed. This is table-stakes. 8. Break down your North Star by segments - New users vs power editors. Free vs paid. Short-form vs long-form. By equation: sessions × completion rate × exports per session. Makes it operationalizable. 9. The eval-driven approach for discovering failures - Ship to 10%. Collect traces. Review failure modes weekly. Create synthetic evals. Add new guardrails. This is the Hamel Husain / Shreya Shankar methodology. 10. The 30-minute follow-up wins jobs - Take your framework. Find gaps. Fix them. Mock up dashboard. Email interviewer. At Descript, you'll be the ONLY person who does this. Immediate differentiation. ---- 👨‍💻 Where to find Dr. Bart Jaworski: LinkedIn: https://www.linkedin.com/in/drbartpm/ Land PM Job: https://www.landpmjob.com 👨‍💻 Where to find Aakash: Twitter: https://www.x.com/aakashg0 LinkedIn: https://www.linkedin.com/in/aagupta/ Newsletter: https://www.news.aakashg.com #productmetrics #pminterview ---- 🧠 About Product Growth: Aakash Gupta's newsletter with over 200K+ subscribers. 🔔 Subscribe and turn on notifications to get more videos like this.

AGAakash GuptahostBJDr. Bart Jaworskiguest

Jan 30, 2026·40m·Watch on YouTube ↗

📖 CHAPTERS

1. 1
   0:00 – 1:18

   Why AI product execution & metrics interviews are a distinct skill

   Aakash frames AI PM interviews as split between product sense/design vs. execution/success metrics, noting AI adds unique measurement challenges. He positions the video as a full end-to-end mock for AI product metrics, aimed at preparing candidates for top AI company roles.

2. 2
   1:18 – 4:29

   Mock interview setup: Descript’s “Underlord” success measurement prompt

   The interviewer (Dr. Bart) sets the scenario: Descript is launching Underlord and wants a success measurement plan. Aakash immediately proposes aligning on what Underlord is before jumping into metrics.

3. 3
   4:29 – 6:42

   Live product walkthrough: why looking at the product first improves metrics quality

   Aakash pulls up Descript and explores Underlord’s UI and capabilities to ground metrics in real user actions. They confirm Underlord is a natural-language co-editor with broad tool access across Descript.

4. 4
   6:42 – 9:16

   The interview framework: clarifications → users → value → metric bank → North Star → guardrails

   Aakash outlines a repeatable structure for execution/metrics cases and gets explicit buy-in from the interviewer. This becomes the backbone used to generate metrics and justify a North Star.

5. 5
   9:16 – 12:52

   User focus debate: don’t over-segment when the feature is core and broadly visible

   Bart suggests Underlord is primarily for beginners, but Aakash pushes back that a homepage/persistent agent must also serve power editors. They align that success should be measured across the full user base, not a narrow persona.

6. 6
   12:52 – 17:24

   Value enumeration: the four core user outcomes Underlord should drive

   Aakash translates feature behavior into a small set of outcomes that can be measured. They land on four major value buckets and consciously avoid getting lost in tool-level micro-metrics.

7. 7
   17:24 – 23:00

   Tool-specific metrics are ‘yellow’: avoid boiling the ocean across every edit feature

   Because Underlord touches many editing tools (captions, chapters, layouts, etc.), Aakash proposes not owning each feature’s detailed success criteria. Instead, Underlord’s PM should focus on cross-cutting success while tool PMs own their local metrics.

8. 8
   23:00 – 25:47

   Positive metrics bank: turning value into measurable signals

   Aakash builds a bank of “positive” success metrics tied to each value. They discuss efficiency vs. engagement, and refine the notion that success means more finished outputs—not more time spent in the editor.

9. 9
   25:47 – 31:00

   North Star choice: why ‘exports/publishes in 7–30 days’ beats time or tool-usage alone

   Aakash compares candidate North Stars and selects an output-oriented metric: exports/publishes within a time window. He adds guardrail thinking (support tickets) to prevent optimizing exports at the expense of user pain.

10. 10
    31:00 – 34:12

    Decomposing the North Star: three vectors for diagnosing performance

    They break down the North Star to understand where Underlord helps or harms. The breakdown covers who benefits, what content type is affected, and what exact behavior counts as the metric event.

11. 11
    34:12 – 35:01

    Trade-offs & guardrails deep dive: evals, rework, time lost, and support burden

    Aakash operationalizes key risks: hallucinations, slower editing, higher support load, and excessive rework of AI outputs. He introduces the idea of AI evals and proposes comparing time-to-edit in A/B tests controlling for number of tools used.

12. 12
    35:01 – 38:08

    The ‘genie metric’ curveball: dream signals + business outcomes

    Bart asks for an unconstrained ‘dream’ metric akin to Skype’s “you’re on mute” counter. Aakash suggests combining delight signals with business impact, and realizes he omitted explicit revenue/output metrics earlier.

13. 13
    38:08 – 40:44

    Post-interview coaching: fix misses, add input/output + leading/lagging, and follow up with a ‘power move’

    Aakash rates his performance (8.5/10) due to the missed output metrics, then shares a practical strategy: incorporate missing metrics, get AI critique, and send a thoughtful post-interview follow-up with improvements. Bart reinforces the value of visual structure and owning the answer even with ambiguous interviewer signals.