By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Aakash GuptaHow Hims & Hers Uses Evals to Let AI Talk to Patients (CPO Interview)
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
Great conversations run long. Their best ideas don't have to.
uListen is free on iPhone, iPad, Android, Chrome, and Claude.
See howEpisode Details
EPISODE INFO
- Released
- November 17, 2026
- Duration
- 1h 3m
- Channel
- Aakash Gupta
- Watch on YouTube
- ▶ Open ↗
EPISODE DESCRIPTION
Dheerja Kaur, the first CPO of Hims & Hers, shows how her team put an AI agent at the front door of healthcare, with doctors in the same chat and an LLM judge checking every answer. Then she covers how she hires AI PMs. Full Writeup: https://www.news.aakashg.com/p/ai-agent-in-regulated-industries Transcript: https://www.aakashg.com/how-to-build-an-ai-agent-in-a-regulated-industry-with-the-cpo-of-hims-hers/ Timestamps 1:26 - Why regulated industries are taking to AI first 5:07 - Why Hims & Hers is going AI-native 9:06 - Ads 11:12 - Inside the Hers app: AI and doctors in one chat 14:45 - Protocols as code and a live LLM judge 20:54 - Results: 80% resolved by AI, 3x engagement 21:42 - Specs in Git and Wild West mode 28:12 - The CPO job now, and PM vs product builder 32:17 - What she hires for and how she interviews 43:23 - What makes an AI PM different 52:05 - How anyone can become an AI PM 58:30 - The mistake AI PMs make in regulated industries 🏆 Thanks to our sponsor, Jira Product Discovery! Plan with purpose, ship with confidence - https://tinyurl.com/yc6dnr5k I'm starting the next cohort of my Land a PM Job program on November 23rd. We have a jam-packed schedule to help you with your search through March. Join the 5th cohort: https://www.landpmjob.com/ Key Takeaways
- Healthcare is AI's top use case because it's broken - Dheerja calls health "the number one use case for AI because it is so broken." The reason it's regulated is the reason people are turning to AI.
- Put the agent and the doctors in one thread - The new Hers app is "a group chat with your agent and your doctors." Nobody copies answers out of ChatGPT to bring to a doctor, and when a doctor steps in, the agent takes the handoff back. A couple of months in, still an experiment, it resolves 80% of conversations without a human and has tripled engagement.
- Check clinical protocols into Git like code - Medical leads write the protocols, and PMs and engineers sit "side by side" with them to turn those into code. Her CTO, Mo, likes to say "we treat clinical protocols as code." Doctors don't write the evals, though. The PM and evals teams build those from the protocols.
- Score every answer before it reaches a patient - The team runs "an LLM judge that's live in production that's scoring an answer before it even gets sent to a customer." Offline evals carry v1. After launch, a daily standup reviews real conversations and whatever the judge flags.
- Write a rubric for warmth, not just accuracy - The first internal version felt "really cold and very clinical." So one PM wrote a personalization rubric with nine dimensions, from whether the agent knows your treatment plan and goals to the warmth of its voice. That warmth is different for Hims than for Hers.
- Keep your conversational guides loose - The first guides got "so prescriptive and deterministic" that the docs ran super long. Then the team reread them. "The whole power of an LLM is that it's flexing to the customer itself," so they cut the guides back to guidelines and kept testing.
- Run new AI tools in Wild West mode - Mandating one tool means "change management every single month." So Hims & Hers gave teams a starter Team OS repo and a PM OS for PRD templates, then let them make it their own and swap notes in forums. My guess is 90% of teams aren't doing this.
- Interview for judgment, not tactics - "The tactics are getting automated away." Dheerja hires for product judgment first, and she'll take high slope "over experience any day." Her favorite interview is an hour on "the hardest product and problem that you've ever shipped," down to every micro piece of it. It "just picks people apart."
- Build your way into an AI PM role - The easy path is a company where AI is "the thesis of how that company wins." Without that, build something on the side that goes layers deep, with fine-tuning or RAG. One of her pharmacy PMs did exactly that, then joined the AI team. A few weeks in, he said "it's not that much of a black box."
- Treat legal and clinical as builders, not sign-offs - Elsewhere, legal is a line in a PRD. In regulated industries, "they are builders with you," and half the battle is understanding the constraints. Legal and clinical partners sit in the Hims & Hers standup every day. When your agent is the front door, that's the job.
👨💻 Where to find Dheerja Kaur: LinkedIn: https://www.linkedin.com/in/dheerja/ Hims & Hers: https://www.himsandhershealth.com/ 👨💻 Where to find Aakash: Twitter: https://x.com/aakashgupta LinkedIn: https://www.linkedin.com/in/aagupta/ Newsletter: https://www.news.aakashg.com 🧠 About Product Growth: The world's largest podcast focused solely on product + growth, with over 200K+ listeners. You can also catch it on Apple Podcasts: https://podcasts.apple.com/us/podcast/the-growth-podcast/id1763555775 Spotify: https://open.spotify.com/show/7vVEMqCSKb7I7xPk8xZtg5 🔔 Subscribe and turn on notifications to get more videos like this.
SPEAKERS
Aakash Gupta
hostHost of Product Growth with Aakash Gupta.
Dheerja Kaur
guestChief Product Officer at Hims & Hers.
EPISODE SUMMARY
In this episode of Aakash Gupta, featuring Aakash Gupta and Dheerja Kaur, How Hims & Hers Uses Evals to Let AI Talk to Patients (CPO Interview) explores How Hims & Hers builds patient-facing AI with evals Dheerja Kaur explains how Hims & Hers builds an AI companion for patients using eval-driven development and a live LLM judge that scores responses before they reach customers.
RELATED EPISODES