How this Yelp AI PM works backward from “golden conversations” to create high-quality prototypes
Claire Vo (host), Priya Badger (guest)
In this episode of How I AI, featuring Claire Vo and Priya Badger, How this Yelp AI PM works backward from “golden conversations” to create high-quality prototypes explores yelp AI PM prototypes by reverse-engineering “golden conversations” into interfaces The episode showcases a practical AI product management workflow that begins with writing (and generating) exemplar “golden conversations” to define the intended user experience before formal requirements.
Yelp AI PM prototypes by reverse-engineering “golden conversations” into interfaces
The episode showcases a practical AI product management workflow that begins with writing (and generating) exemplar “golden conversations” to define the intended user experience before formal requirements.
Priya uses Claude to generate and refine multi-scenario conversations—especially for a new Yelp Assistant feature that analyzes user-uploaded photos for home service requests—then extracts quality criteria and system-prompt direction from those examples.
Next, she turns the conversations into an interactive prototype using Claude Artifacts, enabling realistic testing of response length, latency feel, and overall UX in a chat UI without setting up API keys.
Finally, she explores front-end UI variations in Magic Patterns (including Inspiration mode) to iterate on entry points like “Start with a photo,” emphasizing rapid solution-space exploration and collaboration with design/engineering.
Key Takeaways
Start AI product design from the intended conversation, not the UI.
Priya’s “golden conversations” approach defines what success looks like in the user’s words first, then works backward into prompts, flows, and interface requirements—mirroring the end-user experience early.
Get the full analysis with uListen AI
Treat variability as a core AI product constraint and design for quality explicitly.
Because LLM outputs differ run-to-run, she focuses on methods to drive consistent quality: multiple examples, qualitative review, and eventually a rubric—aligned with the idea that “evals are the new PRD.”
Get the full analysis with uListen AI
Use multi-scenario, multimodal examples to uncover gaps and patterns fast.
Testing across diverse images (cracked porch, appliance error codes, wasp nest, bathroom renovation) reveals whether image recognition is robust and whether the conversation flow generalizes across categories.
Get the full analysis with uListen AI
Iterate conversations by giving targeted critique, then re-generating in bulk.
She provides concrete feedback (be more opinionated with recommendations, avoid asking about budget) and asks the model to rewrite all examples—accelerating convergence on a consistent voice and policy.
Get the full analysis with uListen AI
Interactive prototypes surface UX issues that text transcripts hide.
In an artifact/chat UI, message length, scrolling, and perceived waiting time (“three dots”) can make acceptable copy feel too long or slow—critical signals before engineering investment.
Get the full analysis with uListen AI
Let example conversations inform system instructions and prompt strategy.
Claude Artifacts generates both code and system instructions derived from the golden conversations, helping PMs understand how conversation intent translates into behind-the-scenes prompt constraints.
Get the full analysis with uListen AI
Use AI UI tools to compress ideation cycles and improve cross-functional alignment.
Magic Patterns (and Inspiration mode) enables rapid exploration of UI entry points and guided flows; even when imperfect, clickable variants make it easier to discuss tradeoffs with designers and engineers than static wireframes.
Get the full analysis with uListen AI
Notable Quotes
“So we start with golden conversations. What’s the experience that you’re trying to drive?”
— Priya Badger
“Write an example conversation… and you’re working backwards from that example conversation.”
— Claire Vo
“A lot of people talk about, like, evals are the new PRD. And this is, like, the very early step of getting… to the eval process.”
— Priya Badger
“Sometimes a response that looks fine when you have it in a doc feels really long when you see it in… the little chat bubble.”
— Priya Badger
“I think it’s helpful to actually think about the ways that AI is different than a human.”
— Priya Badger
Questions Answered in This Episode
What does Yelp’s internal “golden conversations” playbook look like—who writes them, how are they reviewed, and what makes one ‘golden’ enough to ship against?
The episode showcases a practical AI product management workflow that begins with writing (and generating) exemplar “golden conversations” to define the intended user experience before formal requirements.
Get the full analysis with uListen AI
When you say you assess conversations qualitatively first, what specific rubric categories do you typically formalize next (e.g., concision, correctness, safety, task completion)?
Priya uses Claude to generate and refine multi-scenario conversations—especially for a new Yelp Assistant feature that analyzes user-uploaded photos for home service requests—then extracts quality criteria and system-prompt direction from those examples.
Get the full analysis with uListen AI
How do you prevent golden conversations from overfitting the system prompt to a narrow set of ‘happy path’ scenarios, especially across many Yelp service categories?
Next, she turns the conversations into an interactive prototype using Claude Artifacts, enabling realistic testing of response length, latency feel, and overall UX in a chat UI without setting up API keys.
Get the full analysis with uListen AI
In the photo-upload feature, where do you draw the line between ‘AI should diagnose the issue’ (e.g., error code meaning) vs ‘AI should only triage and collect info’ to avoid hallucinated advice?
Finally, she explores front-end UI variations in Magic Patterns (including Inspiration mode) to iterate on entry points like “Start with a photo,” emphasizing rapid solution-space exploration and collaboration with design/engineering.
Get the full analysis with uListen AI
What are the most common failure modes you see when testing with many images (misclassification, wrong next question, unsafe recommendation), and how do you prioritize fixes?
Get the full analysis with uListen AI
Transcript Preview
Where do you start when you're thinking about designing and framing out a AI product for what you're working on at work?
What's different about managing products that are powered by AI is there is the interface of how a user interacts with any product or product feature, and that still really matters, and there's also a lot going on behind the scenes. There's a lot also about how do you drive good quality products, because these technologies produce different results each time you use them. So we start with golden conversations. What's the experience that you're trying to drive? And so this is just a way for me to think about how to write that, role-playing a little bit with AI.
What you're saying is actually write an example conversation that can represent what a real user might do, and you're working backwards from that example conversation, which I have actually not seen anybody do before. [upbeat music] Welcome back to How I AI. I'm Claire Vo, product leader and AI obsessive, here on a mission to help you build better with these new tools. Today, we have an AI PM showing us how to AI PM. Priya Mathew Badger is a PM at Yelp and is showing us a completely new way to think about product requirements, prototyping, and how to build effective conversational agents using conversational agents. Let's get to it. This episode is brought to you by GoFundMe Giving Funds, the zero-fee DAF. I wanna tell you about a new product GoFundMe has launched called Giving Funds, a smarter, easier way to give, especially during tax season, which is basically here. GoFundMe Giving Funds is the DAF, or donor-advised fund, from the world's number-one giving platform, trusted by 200 million people. It's basically your own mini foundation without the lawyers or admin costs. You contribute money or appreciated assets, get the tax deduction right away, potentially reduce capital gains, and then decide later where to donate from 1.4 million nonprofits. There are zero admin or asset fees, and while the money sits there, you can invest and grow it tax-free, so you have more to give later, all from one simple hub with one clean tax receipt. Lock in your deduction now and decide where to give later. Perfect for tax season. Join the GoFundMe community of 200 million and start saving money on your tax bill, all while helping the causes you care about the most. Start your giving fund today in just minutes at gofundme.com/howiai. We'll even cover the DAF pay fees if you transfer your existing DAF over. That's gofundme.com/howiai to start your giving fund. Priya, welcome to How I AI. I am so excited to have you here, because whenever anybody asks me, and they ask me a lot, "How do I do AI product management?" I have to say, "Wait, are you talking about product managing with AI? 'Cause I have some ideas about that. Or are you talking about product managing AI products?" And what's really great about the conversation we're about to have is you actually do both. So what, in your mind, is really different about product managing products using AI?
Install uListen to search the full transcript and get AI-powered insights
Get Full TranscriptGet more from every podcast
AI summaries, searchable transcripts, and fact-checking. Free forever.
Add to Chrome