Aakash GuptaThe AI PM Skill That Gets You Instant Job Offers | Aparna, Arize AI
CHAPTERS
- 0:00 – 2:30
Why tracing + evals put PMs in the top 1%
Aparna opens with a provocative claim: PMs who actively use observability (traces) and evals are already elite. The conversation sets up the bigger question of how PM responsibilities shift as AI products become agent-centric.
- •Observability and evals are becoming core PM skills
- •PM vs engineer boundaries blur on AI-native teams
- •Evals are valuable when they reveal what’s wrong (signal for improvement)
- •Framing the episode as a practical, end-to-end demo
- 2:30 – 3:20
What PMs get wrong about when to start evals
Aparna explains a common misconception: teams think they need evals before they have anything real. Her guidance is to build first, then use collected data to decide when and how to introduce evals.
- •Most teams start by building before formal evals
- •Evals become useful after you have data from real usage
- •End-to-end loop: build → collect traces → add evals → automate improvements
- •Goal is an actionable workflow, not theory
- 3:20 – 5:21
Product taste is the new alpha (and where it comes from)
With code getting cheaper, Aparna argues that the differentiator is product taste—knowing what to build and why. She maps taste to systematic feedback intake and proposes turning feedback digestion into an agent-driven workflow.
- •“Code is cheap,” so taste and opinion matter more
- •Taste is cultivated through high-volume, diverse feedback
- •Feedback sources: issues, discussions, Slack/Discord, Gong calls, analytics, Twitter
- •Agents can consume feedback continuously, not just humans
- 5:21 – 6:30
The build–trace–eval loop: building a “product taste” PM agent
They define the agent to build: a PM assistant that aggregates feedback and outputs a prioritized roadmap-style report. Aparna outlines the full framework: create the agent, instrument it with tracing, then add evals and iterate.
- •Agent outputs: top pain points, feature asks, themes, prioritized P0–P3
- •Start with one context source (GitHub) before expanding to others
- •Trace everything for visibility into agent behavior
- •Use evals to systematically improve the agent over time
- 6:30 – 9:29
Sponsor break / bundle + tools mention
Aakash shares sponsor and bundle promotions before returning to the hands-on workflow. This is a brief interlude before the live build begins.
- •Bundle of AI tools offer
- •Show subscription call-to-action
- •Transition back into the demo
- 9:29 – 12:49
Live build in Claude Code: prompt, scope, and scoring logic
Aparna demonstrates how to scaffold the agent in Claude Code with a starter prompt and clear plan. The agent pulls GitHub issues/discussions/releases, scores items based on priority signals, and generates a markdown report.
- •Initialize a directory/repo; provide Anthropic key + GitHub token
- •Pull discussions, issues, releases; start “piecemeal” with GitHub only
- •Scoring factors: bugs vs features, reactions/comments, recency
- •Output is a PM-style report with prioritized actions
- 12:49 – 18:26
What traces are and why PMs should care
Aparna introduces tracing as step-by-step playback of what the agent did—critical for debugging and iteration. She connects traces to the next step: creating evals that measure whether the agent’s prioritization matches product judgment.
- •Trace = end-to-end run; span = individual step within a trace
- •See tool calls (GitHub fetch) and LLM calls transparently
- •Scoring issues is the first PM task the agent must do well
- •Evals should test whether scoring matches intended prioritization rules
- 18:26 – 21:59
One-command instrumentation: adding Arize tracing via skills
Using Arize ‘skills,’ Aparna shows how Claude Code can instrument the agent automatically. The skill inspects the codebase, detects language/provider, and wires telemetry so traces stream into Arize with minimal manual work.
- •Skills are natural-language instructions packaged for coding agents
- •Instrumentation skill auto-detects Python + Anthropic usage
- •Tracing setup becomes accessible without heavy engineering effort
- •Goal: immediate visibility into agent behavior
- 21:59 – 24:03
Watching traces stream live: inspecting outputs and debugging readiness
They run the agent and verify traces appear in Arize in real time. Aparna walks through the report output (executive summary, pain points, priorities) and explains why this visibility is the foundation for improvement.
- •Traces show calls, data fetches, scoring steps, and final report generation
- •Agent produces prioritized pain points and feature requests
- •No IDE required—built and instrumented from the terminal
- •Tracing enables pinpointing why outputs are good or “slop”
- 24:03 – 31:15
From traces to evals: asking Claude to suggest evaluations
Aparna demonstrates using trace data to bootstrap eval ideas—groundedness, priority alignment, actionability. She then narrows the ask: evaluate scoring at the individual-issue level, not just the final report.
- •In-product agent (Alex) can analyze trace patterns and errors
- •Initial eval suggestions: groundedness, priority alignment, actionability
- •Need more granular evals: per-issue priority correctness
- •Iterative prompt refinement to get the eval you actually need
- 31:15 – 33:46
Ads break: AI PM course + prototyping + Customer.io
Aakash inserts an ad segment covering an AI PM certificate, prototyping tools, and Customer.io. The episode then resumes the eval workflow discussion.
- •AI PM certificate promotion
- •Bolt prototyping message
- •Customer.io personalization and MCP mention
- 33:46 – 48:07
Vibe evals vs axial coding: aligning evals to human taste
They discuss the classic workflow of finding failures in traces and categorizing them, and how Claude can accelerate that process. Aparna argues vibe evals quickly hit limits without human grounding, and explains continuous eval alignment as data changes.
- •Claude can help surface errors and propose initial evals, but humans must align them
- •Key question: “Was the eval wrong or was the agent wrong?”
- •Vibe evals are a starting point; axial coding/human labeling adds grounding
- •Evals drift over time; alignment must be ongoing
- 48:07 – 56:32
Automating the improvement loop: self-improving agents (safely)
Aparna extends the loop beyond evaluation: use eval failures to propose agent changes, then rerun and retrace—potentially on a schedule. They cover safeguards for production: PR review, human oversight on agent/eval changes, and controlled expansion of what’s automated.
- •Create cron-like loops to rerun agents and improvements automatically
- •Use eval failures to suggest prompt/tooling fixes, then ship new versions
- •Safety: humans review PRs and changes to evals and agents
- •State of the art: teams are widening the radius of automated changes
- 56:32 – 1:10:43
The AI-native PM: skills, velocity, and the context graph unlock
Aparna contrasts AI-native vs traditional companies and explains how PMs become “indistinguishable from engineers” by operating in Claude Code and building faster feedback loops. She introduces the context graph as a major enterprise unlock: giving agents cross-silo context to improve decisions and execution.
- •AI-native PMs prototype/build, analyze outcomes, and iterate rapidly
- •Key hiring signal: curiosity and tool fluency amid fast-changing capabilities
- •Some teams ship from issue → production in a single day
- •Context graphs unify siloed data to make agents smarter across orgs
- 1:10:43 – 1:19:31
Phoenix vs Arize AX: choosing open source vs enterprise scale + final takeaways
Aparna explains when to use Phoenix (self-host, sensitive data) versus the paid Arize platform (scale, high volume), and why Arize emphasizes open formats and framework-agnostic support. They close with an actionable weekend challenge: build a small agent, instrument it, and learn via traces + evals.
- •Phoenix is ideal for self-hosting and PII constraints; skills work there too
- •Arize AX fits teams with massive trace volume and scaling needs
- •Differentiators: framework-agnostic, open data formats, OpenInference telemetry
- •Two-hour challenge: build an internal agent; then use observability + evals to improve
