How I AIHow this Yelp AI PM works backward from “golden conversations” to create high-quality prototypes
CHAPTERS
- 0:00 – 3:12
Why AI PM starts with “golden conversations” (Priya’s core framing)
Claire sets up the central question: where to start when designing an AI product. Priya previews what makes AI product management different and introduces Yelp’s practice of working backward from “golden conversations” that define the ideal end-user experience.
- •AI PM has two intertwined layers: user-facing interface and behind-the-scenes behavior
- •Non-determinism: models can produce different outputs each run, raising quality challenges
- •“Golden conversations” as the anchor artifact for requirements and design
- •Using AI to role-play and draft those conversations faster
- 3:12 – 4:51
The extra complexity of AI-powered products: UI, system prompts, and quality
Priya explains why AI features are unlike traditional software: the interface still matters, but the system prompt and orchestration deeply shape outcomes. They discuss the challenge of driving consistent quality when outputs vary.
- •Classic UI/UX still matters even for LLM-based features
- •System prompts and conversation flow design become product surface area
- •Variability across runs requires new approaches to quality and testing
- •AI PM spans both design decisions and model-behavior decisions
- 4:51 – 6:35
Concrete Yelp Assistant example: adding photo upload + photo understanding
Priya grounds the workflow in a real feature: enabling users to upload photos for home-service requests. The goal isn’t just attaching an image, but having the assistant interpret it and adapt the conversation accordingly.
- •Yelp Assistant gathers project details and matches consumers to pros/quotes
- •New capability: user uploads a photo to describe the issue
- •Key question: can AI interpret the image and tailor follow-up questions/recommendations?
- •Scale challenge: many service categories and highly diverse user inputs
- 6:35 – 9:11
Designing the flow by writing the conversation first (requirements as dialogue)
Priya demonstrates starting in Claude by prompting for a complete sample conversation, with explicit formatting constraints. Claire highlights this as a novel “conversation-as-wireframe” approach that reverses the usual PRD-to-UI path.
- •Prompt Claude to generate a full end-to-end example chat
- •Add constraints: labels (Assistant/User), continuous conversation, required capabilities
- •Treat the example dialogue as a first-pass wireframe for conversational UX
- •Work backward from the desired experience to the requirements
- 9:11 – 9:57
Why this works: prototyping closest to the user’s actual experience
Claire connects Priya’s method to broader PM craft: start with artifacts closest to what users consume (docs, content, conversations), then back into specs. This section frames conversations as a high-signal product requirement for LLM agents.
- •PM principle: prototype the user-facing artifact, not a generic PRD first
- •Conversation content is the “product” users feel in an assistant
- •Golden conversations clarify what “good” looks like before implementation
- •Back-solving requirements becomes easier once the end state is vivid
- 9:57 – 12:05
Demo: single-image test and reading model reasoning for debugging
Priya uploads an example photo (cracked porch) and reviews Claude’s output for image recognition and question quality. They discuss why inspecting the model’s thought process/reasoning can help refine prompts and troubleshoot misunderstandings.
- •Use a real image to validate photo understanding and initial follow-ups
- •Assess recognition accuracy (does it describe the photo correctly?)
- •Use reasoning/thought process to see how the model interpreted instructions
- •Reasoning can reveal prompt gaps and model “personality” differences
- 12:05 – 13:26
Scaling coverage: generate many conversations across many images/use cases
After one successful test, Priya broadens to a suite of images (appliance error codes, wasp nest, renovation) to surface patterns and failure modes. She adds output structure (titles/numbering) to make results usable for review.
- •Create multiple examples to reveal trends and edge cases
- •Test across diverse categories to ensure cross-domain robustness
- •Add output formatting requirements (titles, numbering) for easier analysis
- •Move from anecdotal success to systematic coverage
- 13:26 – 15:07
Early eval thinking: define qualitative criteria, then build a rubric
Priya explains what she looks for in the generated conversations: image recognition quality, conversational flow, concision, and clarity. Claire ties this to the idea that “evals are the new PRD,” with examples as the first step toward formal evaluation.
- •Start with qualitative read-through: flow, brevity, comprehensibility
- •Primary gate: correct image interpretation and relevance
- •Identify missing behavior (e.g., better diagnosis from error codes)
- •Evolve toward a rubric/criteria for consistent evaluation
- 15:07 – 15:59
Iterate fast: rewrite conversations with targeted feedback (opinionated guidance)
Priya shows how to refine the entire conversation set by giving Claude direct critique (be more opinionated, avoid budget questions). The model rewrites the examples, enabling rapid iteration on “voice,” policy, and guidance style.
- •Give concrete feedback on tone/behavior (more recommendations, less budget talk)
- •Rewrite multiple conversations in one pass for consistency
- •Check whether changes actually reflect the guidance
- •Example: stronger safety guidance for pest control vs DIY
- 15:59 – 21:37
Demo: turn golden conversations into an interactive Claude Artifact prototype
Priya uses Claude Artifacts to generate a working chat UI that calls the Claude model, plus a system prompt derived from the example conversations. She adds Yelp Assistant screenshots to guide the prototype’s look-and-feel and then tests the artifact end-to-end.
- •Artifacts can produce an interactive prototype without manual API wiring
- •Generate system instructions from the golden conversations automatically
- •Use reference screenshots to approximate the real product UI
- •Test realism: message length, latency feel (typing dots), mobile bubble constraints
- 21:37 – 22:38
From prototype to team collaboration: bridging PM, design, and engineering
Priya emphasizes the artifact is a starting point, not the final design process. The workflow helps her clarify thinking, understand system prompts better, and collaborate with designers/engineers who co-author and critique conversations.
- •Prototype clarifies intent and improves cross-functional communication
- •System prompt literacy becomes part of PM collaboration
- •Conversations are co-written and iterated by product, design, engineering
- •Prototype helps answer: what’s feasible, what feels good, what to refine
- 22:38 – 25:33
UI exploration in Magic Patterns: adding entry points and guided flows
They switch to Magic Patterns for visual interface iteration: e.g., a “Start with a photo” prompt suggestion. Priya explains why these tools change ideation speed while not replacing core PM work of understanding users and prioritizing problems.
- •Use Magic Patterns to explore UI flows and entry-point affordances
- •Natural language → components and layout updates for rapid iteration
- •PM work unchanged at the core (problem understanding, prioritization)
- •Big gain: faster exploration and clearer communication of ideas
- 25:33 – 34:00
Inspiration mode: generate multiple UI variants fast (and handle prototype errors)
Priya demonstrates Inspiration mode to create multiple differentiated design directions (guided category selection, real-time detection concept, multi-photo flow). The demo hits React compatibility errors, prompting discussion of debugging and the reality of AI prototyping tools.
- •Prompt for multiple differentiated UI concepts in one go
- •Compare variants quickly by clicking through generated options
- •Expect failures; use “fix/debug” commands and iterative repair loops
- •Use for early ideation even before a full PRD exists
- 34:00 – 38:56
Applying the workflow beyond work: newsletters, parenting helper, and game timer
In a lightning-round tour, Priya shares personal projects that use the same AI building blocks: Claude Projects to summarize Slack into a newsletter draft, an Artifact-based parenting helper, and a Lovable-built Catan timer with a leaderboard.
- •Claude Projects: reusable instructions + uploads for repeatable summarization workflows
- •Newsletter automation: Slack logs → formatted digest with minimal prompting
- •Personal Artifact: “Parent Pal” for interactive guidance via clarifying questions
- •Lovable prototype: Catan timer + feature requests like leaderboards/handicaps
- 38:56 – 40:48
When AI stops cooperating: reset context, fork/remix, and start fresh
Priya’s troubleshooting advice: remember AI isn’t human and often fails due to context-window and long-thread issues. She recommends forking/remixing prototypes or summarizing and restarting chats to restore reliability.
- •Failure mode: losing context over long, multi-turn threads
- •Fork/remix to reset the context window while keeping relevant code/state
- •Ask the model to summarize context, then start a fresh chat
- •Treat AI limitations explicitly instead of arguing with it conversationally
- 40:48 – 41:44
Wrap-up: where to find Priya and final sign-off
Priya shares where people can follow her work, and Claire closes the episode. They reinforce the theme: work backward from the end-user experience and use AI tooling to iterate quickly on both conversation quality and interface design.
- •Find Priya on LinkedIn and her Substack: AlmostMagic.Substack
- •Core takeaway: golden conversations → prototype → UI iterations
- •AI tools accelerate exploration, but teams still collaborate to ship
- •Episode close: like/subscribe/reviews and where to find the show
