How I AIHow this Yelp AI PM works backward from “golden conversations” to create high-quality prototypes
CHAPTERS
Meet Priya (Yelp AI PM) and the “golden conversations” idea
Claire introduces Priya Mathew Badger, a PM at Yelp, and tees up her distinctive workflow for building AI products. The core concept: start from “golden conversations” (ideal user-assistant exchanges) and work backward into requirements, prompts, and prototypes.
- •Priya’s role: PM for Yelp Assistant
- •Framing AI PM as both managing AI products and using AI in PM work
- •“Golden conversations” as the north star artifact for defining experience
- •Working backward from what users should feel/see, not a traditional PRD first
Why AI product management is different: UI + system behavior + quality variability
Priya breaks down what changes when products are powered by AI: the user interface still matters, but so do system prompts and backend behavior. A major challenge is that AI outputs vary run-to-run, so quality and consistency become central product problems.
- •Two layers: user-facing interaction and behind-the-scenes prompting/orchestration
- •System prompts shape conversation flow and constraints
- •Non-determinism: “different results each time” requires new quality practices
- •Need to define what “good” looks like before implementation
Case study setup: Adding photo upload intelligence to Yelp Assistant
Priya explains Yelp Assistant’s service-request flow (describe a need → gather details → match to pros → get quotes). The new capability: users upload photos, and the assistant should interpret the image and adapt the conversation accordingly.
- •Assistant supports many home/auto service categories (handyman, plumber, electrician, etc.)
- •Photo upload adds valuable context for pros and quoting
- •Goal: AI analyzes the photo and tailors follow-up questions/recommendations
- •Key risk: wide variability in what users submit across categories
Starting point: Draft the ideal conversation flow in Claude
Priya demonstrates prompting Claude to generate a full sample conversation, including image analysis and suggested replies. She emphasizes specifying output format (roles, continuous dialogue) to reduce iteration and produce usable artifacts.
- •Prompt Claude to write an end-to-end user/assistant conversation
- •Include constraints: photo upload, analysis, suggested replies, info needed for quotes
- •Output formatting directives (User/Assistant labels, continuous thread)
- •This conversation becomes a “first-pass wireframe” for a conversational product
Why “example conversation first” works (and how teams use it)
Claire highlights the novelty: instead of beginning with UI mockups or PRDs, Priya begins with a realistic transcript of usage. Priya explains this mirrors Yelp’s internal LLM playbook—define the intended experience, then derive requirements and implementation details.
- •Conversation-as-prototype is closer to what users actually experience
- •Aligns to product principle: start with the end artifact, then reverse-engineer requirements
- •Yelp’s playbook: “start with golden conversations”
- •Role-playing with AI helps PMs rapidly explore flows and edge cases
Testing across scenarios: multiple images to surface patterns and evaluation criteria
Priya scales testing by feeding Claude many images (appliance error codes, pests, renovation photos) to generate multiple conversation examples. She uses these to identify patterns, define what “good” means, and begin shaping an evaluation rubric.
- •Generate many scenario conversations to reveal trends and gaps
- •Assess image recognition quality as a first criterion
- •Qualitative review: flow, concision, clarity, usefulness
- •Early step toward “evals are the new PRD” mindset
Refining the “golden conversations” with feedback loops
Priya shows how to iteratively improve the conversations by giving targeted feedback (be more opinionated, avoid budget questions). Claude rewrites all scenarios, accelerating convergence on the desired tone, guidance level, and structure.
- •Provide editorial guidance (recommendations, tone, avoid unhelpful questions)
- •Batch rewrite multiple conversations based on feedback
- •Spot-check whether revisions reflect intent
- •Use refined conversations to clarify product behavior before UI/engineering work
Demo: Turn conversations into an interactive prototype with Claude Artifacts
Priya uses Claude Artifacts to generate a working chat app that calls Claude’s model, including a draft system prompt derived from the example conversations. She also supplies Yelp Assistant screenshots to steer the prototype UI toward realistic styling.
- •Artifacts can create an interactive chat prototype powered by Claude’s LLM
- •No separate API-key integration needed (faster prototyping loop)
- •Artifact includes system instructions inferred from the examples
- •Upload reference screenshots to approximate real product UI
Why interactive prototyping matters: bubble length, latency, and “feel”
Priya tests the Artifact and explains why it’s more revealing than reading a transcript in a doc. Message length, waiting indicators, and pacing can change perceived quality—so simulating the real interface can expose issues early.
- •Prototype shows response length in real chat bubbles (often feels longer than in text)
- •Latency and typing indicators affect user experience
- •Helps teams evaluate “does this feel right?” not just “is it correct?”
- •Shareable artifact improves cross-functional alignment (PM/design/engineering)
From conversation design to UI exploration with Magic Patterns
Priya pivots to the user interface layer: entry points, prompts, and flows that communicate assistant capabilities. She demos Magic Patterns to visually explore UI changes, like adding a “Start with a photo” prompt suggestion.
- •UI details matter: discoverability, user prompts, flow into assistant experience
- •Magic Patterns supports natural-language-driven UI prototyping
- •Iterate on components quickly (e.g., add a guided photo entry point)
- •PMs can externalize ideas visually without heavy design overhead
Exploring variants fast: Magic Patterns “Inspiration mode” + iteration realities
Using Inspiration mode, Priya generates multiple differentiated design options for a more guided “start with photo” flow. They also hit a prototyping error (React version issue), illustrating the practical reality: AI tools accelerate ideation but still require debugging and judgment.
- •Inspiration mode generates multiple UI concepts side-by-side
- •Prompting tips: ask for differentiated options and explanations
- •Fork/remix variants to pursue multiple directions in parallel
- •AI prototyping still breaks sometimes—debug tools and manual triage are part of the workflow
Applying the workflow beyond work: newsletters, parenting helper, and game timers
In a lightning-round tour, Priya shares personal uses of AI prototyping: auto-summarizing Slack threads into a newsletter via a Claude project, building a “Parent Pal” artifact, and creating a Settlers of Catan timer/leaderboard app using Lovable.
- •Claude Projects: reusable instruction templates for repeatable tasks (newsletter from Slack logs)
- •Artifacts for personal micro-apps (Parent Pal behavior coaching)
- •Lovable for more feature-heavy apps (leaderboards, tracking, handicaps)
- •Advice: build personal tools if you want AI product experience without a work project
When AI won’t cooperate: reset context and “treat it like a system, not a human”
Priya recommends stepping back from conversational habits and diagnosing AI limitations, especially context-window issues. Practical fixes include forking/remixing prototypes to reset context and summarizing long chats into a fresh session.
- •AI loses context over long multi-turn conversations
- •Fork/remix to reset the context window in prototyping tools
- •Ask the model to summarize state and restart in a new chat
- •Debugging mindset: reason about limitations rather than arguing with the model
Wrap-up: where to follow Priya and final takeaways
They close by reiterating the non-linear, fast iteration enabled by AI: start with end-user experience, iterate cheaply, and bring teams along with interactive prototypes. Priya shares where to find her work and tips.
- •AI enables starting at the “end experience” and iterating backward
- •Interactive prototypes improve communication and alignment
- •Priya on LinkedIn + Substack: AlmostMagic.Substack
- •Show outro and calls to like/subscribe/review
