Aparna Full Pod Final AR

Aparna Dhinakaran, CPO of Arize AI ($131M raised), shows exactly how to build a PM agent in Claude Code, instrument it with observability, run evals against it, and close the self-improvement loop, all in one live session. If you want to understand what serious AI eval practice looks like in 2025, this is the episode. Full Writeup: [VERIFY - newsletter URL] Transcript: [VERIFY - transcript URL] --- Timestamps: 00:01 - What PMs are getting wrong when building agents 04:00 - Screen share begins — building the PM agent live 07:05 - What a product taste agent actually does 09:10 - When to start running evals 10:15 - Building the agent in Claude Code from scratch 16:13 - Preview of a pre-built version with tracing active 21:34 - Instrumenting the agent for observability (one command) 27:26 - Traces streaming into Arize in real time 30:38 - Asking Claude to suggest evals 34:36 - Running the priority accuracy eval 46:10 - Vibe evals vs. axial coding — when to use each 52:46 - Looping the improvement automatically 01:04:01 - What AI PMs need to do differently 01:09:05 - What enterprise PMs can realistically take on now 01:22:10 - The two things to do this weekend --- Thanks to our sponsors: 1. Superhuman - Sign up and get 1-month free of Superhuman Mail with my link: superhuman.com/akash (given by brand - Kartik) 2. Land PM Job - Land your next PM role faster - https://landpmjob.com 3. Vanta - Automate your compliance - http://vanta.com/aakash 4. Product Faculty - Get $550 off their AI PM Certification with code AAKASH550C7 - https://maven.com/product-faculty/ai-product-management-certification?promoCode=AAKASH550C7 5. Bolt - Ship AI-powered products 10x faster - https://bolt.new --- Key Takeaways: 1. Trace before you eval - A trace is the full step-by-step playback of what your agent did. Without it, you have no evidence base for evals. Every LLM call, every tool call, every intermediate output needs to be visible before you write a single eval. 2. A span is your unit of evaluation - A span is one discrete step inside a trace. Evals run at the span level, not the trace level. "Did this specific scoring step get the priority right?" is a more useful question than "was the whole run good?" 3. Instrumentation is now a one-command job - Claude Code's instrumentation skills can set up observability for your agent automatically. Arize Phoenix's skill looks at your codebase, identifies the LLM calls and tool calls, and wires them to the tracing layer. No engineering support required. 4. The vibe eval is a draft, not a verdict - An LLM can suggest what your evals should test by looking at your traces. That suggestion will not know your bug-first policy, your comp logic, or your definition of "critical." Treat it as v0 and refine against your actual judgment. 5. When evals fire, two things could be wrong - The agent produced a bad output. Or the eval is miscalibrated. Reading the flagged span yourself is the only way to know which one needs fixing. Both are normal. Both are good news. 6. Evals drift and need regular realignment - Your priorities change. Your bug policy changes. Your product changes. An eval calibrated to last quarter will start misfiring this quarter. Regular alignment to human feedback is maintenance, not a failure. 7. The self-improvement loop is already running at the best teams - Fetch all spans where evals fired. Group by failure category. Propose a specific prompt fix. Review and approve. Ship the new version. This loop runs on a schedule and requires a human at the approval step. 8. Enterprise PMs: start with one internal agent - Not a customer-facing product. An internal tool that takes four hours off your week. Once you have it, you will naturally want to trace it. That is when observability starts to matter to you personally. 9. The context graph is the enterprise unlock - Agents are only as useful as the context they have. Enterprise data lives in silos. The teams breaking through are building unified context layers that give one agent access to CRM, Gong, analytics, GitHub, and Slack. 10. Product taste is still the alpha - Code is cheap now. Shipping speed is table stakes. The PMs who pull ahead are the ones with the sharpest judgment about what to build, and the loops that make their agents better every day. --- Where to find Aparna Dhinakaran: LinkedIn: [VERIFY - Aparna LinkedIn URL] Arize AI: https://arize.com Where to find Aakash: Twitter: https://www.x.com/aakashg0 LinkedIn: https://www.linkedin.com/in/aakashgupta/ Newsletter: https://www.news.aakashg.com #AIagents #ProductManagement --- About Product Growth: The world's largest podcast focused solely on product + growth, with over 200K+ listeners. Subscribe and turn on notifications.

Aparna DhinakaranguestAakash Guptahost

May 21, 20261h 19mWatch on YouTube ↗

EPISODE INFO

Released: May 21, 2026
Duration: 1h 19m
Channel: Aakash Gupta
Watch on YouTube: ▶ Open ↗

EPISODE DESCRIPTION

Aparna Dhinakaran, CPO of Arize AI ($131M raised), shows exactly how to build a PM agent in Claude Code, instrument it with observability, run evals against it, and close the self-improvement loop, all in one live session. If you want to understand what serious AI eval practice looks like in 2025, this is the episode. Full Writeup: [VERIFY - newsletter URL] Transcript: [VERIFY - transcript URL] --- Timestamps: 00:01 - What PMs are getting wrong when building agents 04:00 - Screen share begins — building the PM agent live 07:05 - What a product taste agent actually does 09:10 - When to start running evals 10:15 - Building the agent in Claude Code from scratch 16:13 - Preview of a pre-built version with tracing active 21:34 - Instrumenting the agent for observability (one command) 27:26 - Traces streaming into Arize in real time 30:38 - Asking Claude to suggest evals 34:36 - Running the priority accuracy eval 46:10 - Vibe evals vs. axial coding — when to use each 52:46 - Looping the improvement automatically 01:04:01 - What AI PMs need to do differently 01:09:05 - What enterprise PMs can realistically take on now 01:22:10 - The two things to do this weekend --- Thanks to our sponsors:
1. Superhuman - Sign up and get 1-month free of Superhuman Mail with my link: superhuman.com/akash (given by brand - Kartik)
1. Land PM Job - Land your next PM role faster - https://landpmjob.com
1. Vanta - Automate your compliance - http://vanta.com/aakash
1. Product Faculty - Get $550 off their AI PM Certification with code AAKASH550C7 - https://maven.com/product-faculty/ai-product-management-certification?promoCode=AAKASH550C7
1. Bolt - Ship AI-powered products 10x faster - https://bolt.new --- Key Takeaways:
1. Trace before you eval - A trace is the full step-by-step playback of what your agent did. Without it, you have no evidence base for evals. Every LLM call, every tool call, every intermediate output needs to be visible before you write a single eval.
1. A span is your unit of evaluation - A span is one discrete step inside a trace. Evals run at the span level, not the trace level. "Did this specific scoring step get the priority right?" is a more useful question than "was the whole run good?"
1. Instrumentation is now a one-command job - Claude Code's instrumentation skills can set up observability for your agent automatically. Arize Phoenix's skill looks at your codebase, identifies the LLM calls and tool calls, and wires them to the tracing layer. No engineering support required.
1. The vibe eval is a draft, not a verdict - An LLM can suggest what your evals should test by looking at your traces. That suggestion will not know your bug-first policy, your comp logic, or your definition of "critical." Treat it as v0 and refine against your actual judgment.
1. When evals fire, two things could be wrong - The agent produced a bad output. Or the eval is miscalibrated. Reading the flagged span yourself is the only way to know which one needs fixing. Both are normal. Both are good news.
1. Evals drift and need regular realignment - Your priorities change. Your bug policy changes. Your product changes. An eval calibrated to last quarter will start misfiring this quarter. Regular alignment to human feedback is maintenance, not a failure.
1. The self-improvement loop is already running at the best teams - Fetch all spans where evals fired. Group by failure category. Propose a specific prompt fix. Review and approve. Ship the new version. This loop runs on a schedule and requires a human at the approval step.
1. Enterprise PMs: start with one internal agent - Not a customer-facing product. An internal tool that takes four hours off your week. Once you have it, you will naturally want to trace it. That is when observability starts to matter to you personally.
1. The context graph is the enterprise unlock - Agents are only as useful as the context they have. Enterprise data lives in silos. The teams breaking through are building unified context layers that give one agent access to CRM, Gong, analytics, GitHub, and Slack.
1. Product taste is still the alpha - Code is cheap now. Shipping speed is table stakes. The PMs who pull ahead are the ones with the sharpest judgment about what to build, and the loops that make their agents better every day. --- Where to find Aparna Dhinakaran: LinkedIn: [VERIFY - Aparna LinkedIn URL] Arize AI: https://arize.com Where to find Aakash: Twitter: https://www.x.com/aakashg0 LinkedIn: https://www.linkedin.com/in/aakashgupta/ Newsletter: https://www.news.aakashg.com #AIagents #ProductManagement --- About Product Growth: The world's largest podcast focused solely on product + growth, with over 200K+ listeners. Subscribe and turn on notifications.

SPEAKERS

Aparna Dhinakaran
guest
CPO and co-founder of Arize AI, focused on AI product management, evals, and observability tooling (e.g., Phoenix).
Aakash Gupta
host
Podcast host and interviewer who runs Aakash’s bundle (bundle.aakashg.com) and reads sponsor ads.

EPISODE SUMMARY

In this episode of Aakash Gupta, featuring Aparna Dhinakaran and Aakash Gupta, Aparna Full Pod Final AR explores build a PM agent, trace it, eval it, improve iteratively The episode live-builds a “product taste” PM agent that pulls GitHub issues/discussions/releases, scores items by priority, and outputs a structured markdown PM report of top pain points and roadmap recommendations.

RELATED EPISODES