By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Aakash GuptaHow Hims & Hers Uses Evals to Let AI Talk to Patients (CPO Interview)
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
Great conversations run long. Their best ideas don't have to.
uListen is free on iPhone, iPad, Android, Chrome, and Claude.
See howAt a glance
WHAT IT’S REALLY ABOUT
How Hims & Hers builds patient-facing AI with evals
- Dheerja Kaur explains how Hims & Hers builds an AI companion for patients using eval-driven development and a live LLM judge that scores responses before they reach customers.
- She describes a new Hers weight-loss app experience built as a two-tab product where the primary interface is a single group chat that includes both the AI companion and doctors in one thread.
- She outlines Hims & Hers’ broader “AI-native” vision as a full-stack healthcare loop spanning diagnostics and health data, personalized medications, and an AI layer integrated with over 1,000 providers.
- She shares early outcomes from an experimental rollout, including 3× customer engagement and the AI resolving 80% of conversations without a human handoff.
- She discusses how AI is changing product execution (specs in Git, rapid iteration, “Wild West mode”) and what fundamentals she hires for in PMs, especially for AI product roles in regulated settings.
IDEAS WORTH REMEMBERING
5 ideasHims & Hers runs eval-driven development with a live LLM judge in production.
She describes building with “eval-driven development” (test → release → iterate), and adds an always-on safeguard: a live LLM judge that scores an answer before it’s sent to a customer.
The Hers app experience is designed as a single group chat with an AI companion and doctors.
She frames the product as combining proactive AI support and holistic “dot connecting” with real providers who handle diagnosis and prescribing, keeping both in a single conversation thread for the user.
Their AI-native shift is built on a three-part full-stack healthcare model.
She says the company needs diagnostics, treatments, and a doctor experience together: a unified health-data picture (including labs), verticalized/personalized medications, and an AI layer that works with human providers.
Clinical protocols are codified and versioned like software, then translated into evals.
Clinical specialists document protocols, and the team “treats clinical protocols as code” by checking them in and updating them like software; PMs and an evals group turn those protocols into evals.
Beyond accuracy, the team explicitly evaluates personalization, warmth, and brand voice.
She gives an example where early versions felt cold because they only reflected protocols; a PM created a “personalization rubric” spanning context-awareness, clinical synthesis, and voice/tone (which can vary by brand and journey).
WORDS WORTH SAVING
5 quotesWe also have an LLM judge that's live in production that's scoring an answer before it even gets sent to a customer.
— Dheerja Kaur
It's like, how do you build delightful, engaging consumer experience in a category that people are inherently have, like, a negative relationship with sometimes?
— Dheerja Kaur
We're already seeing 3X the customer engagement, which is, like, kind of insane.
— Dheerja Kaur
We're seeing the AI companion resolve 80% of conversations without needing a human in the loop, which is just massive.
— Dheerja Kaur
The anxiety around being AI-pilled enough is real.
— Dheerja Kaur
Ask ChatGPT about this episode
Get its key ideas, plus a summary of each chapter linked to that point in the video, right in your chat.
Add uListen AI to ChatGPT.
Install uListen AI in ChatGPT (opens in a new tab)Needs a ChatGPT account.Paste this prompt into a new chat.
Use uListen AI to give me this episode's key ideas and a chapter-by-chapter summary, with timestamps: https://www.youtube.com/watch?v=j1jRqOLHl_w
SPEAKERS
Aakash Gupta
hostHost of Product Growth with Aakash Gupta.
Dheerja Kaur
guestChief Product Officer at Hims & Hers.
High quality AI-generated summary created from speaker-labeled transcript.