Skip to content

By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

Aakash GuptaAakash Gupta

How Hims & Hers Uses Evals to Let AI Talk to Patients (CPO Interview)

Dheerja Kaur, the first CPO of Hims & Hers, shows how her team put an AI agent at the front door of healthcare, with doctors in the same chat and an LLM judge checking every answer. Then she covers how she hires AI PMs.

Aakash GuptahostDheerja Kaurguest
Nov 17, 20261h 3mWatch on YouTube

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

Great conversations run long. Their best ideas don't have to.

uListen is free on iPhone, iPad, Android, Chrome, and Claude.

See how

At a glance

WHAT IT’S REALLY ABOUT

How Hims & Hers builds patient-facing AI with evals

  1. Dheerja Kaur explains how Hims & Hers builds an AI companion for patients using eval-driven development and a live LLM judge that scores responses before they reach customers.
  2. She describes a new Hers weight-loss app experience built as a two-tab product where the primary interface is a single group chat that includes both the AI companion and doctors in one thread.
  3. She outlines Hims & Hers’ broader “AI-native” vision as a full-stack healthcare loop spanning diagnostics and health data, personalized medications, and an AI layer integrated with over 1,000 providers.
  4. She shares early outcomes from an experimental rollout, including 3× customer engagement and the AI resolving 80% of conversations without a human handoff.
  5. She discusses how AI is changing product execution (specs in Git, rapid iteration, “Wild West mode”) and what fundamentals she hires for in PMs, especially for AI product roles in regulated settings.

IDEAS WORTH REMEMBERING

5 ideas

Hims & Hers runs eval-driven development with a live LLM judge in production.

She describes building with “eval-driven development” (test → release → iterate), and adds an always-on safeguard: a live LLM judge that scores an answer before it’s sent to a customer.

The Hers app experience is designed as a single group chat with an AI companion and doctors.

She frames the product as combining proactive AI support and holistic “dot connecting” with real providers who handle diagnosis and prescribing, keeping both in a single conversation thread for the user.

Their AI-native shift is built on a three-part full-stack healthcare model.

She says the company needs diagnostics, treatments, and a doctor experience together: a unified health-data picture (including labs), verticalized/personalized medications, and an AI layer that works with human providers.

Clinical protocols are codified and versioned like software, then translated into evals.

Clinical specialists document protocols, and the team “treats clinical protocols as code” by checking them in and updating them like software; PMs and an evals group turn those protocols into evals.

Beyond accuracy, the team explicitly evaluates personalization, warmth, and brand voice.

She gives an example where early versions felt cold because they only reflected protocols; a PM created a “personalization rubric” spanning context-awareness, clinical synthesis, and voice/tone (which can vary by brand and journey).

WORDS WORTH SAVING

5 quotes

We also have an LLM judge that's live in production that's scoring an answer before it even gets sent to a customer.

— Dheerja Kaur

It's like, how do you build delightful, engaging consumer experience in a category that people are inherently have, like, a negative relationship with sometimes?

— Dheerja Kaur

We're already seeing 3X the customer engagement, which is, like, kind of insane.

— Dheerja Kaur

We're seeing the AI companion resolve 80% of conversations without needing a human in the loop, which is just massive.

— Dheerja Kaur

The anxiety around being AI-pilled enough is real.

— Dheerja Kaur

Read the available transcript
Eval-driven developmentLive LLM judge in productionProtocols as codeGroup chat UX with AI and doctorsPersonalization rubric (voice, warmth, context)Proactive outreach using synced patient dataSpecs in Git and fast execution workflows

Ask ChatGPT about this episode

Get its key ideas, plus a summary of each chapter linked to that point in the video, right in your chat.

  1. Add uListen AI to ChatGPT.

  2. Paste this prompt into a new chat.

    Use uListen AI to give me this episode's key ideas and a chapter-by-chapter summary, with timestamps: https://www.youtube.com/watch?v=j1jRqOLHl_w

SPEAKERS

  • Aakash Gupta

    host

    Host of Product Growth with Aakash Gupta.

  • Dheerja Kaur

    guest

    Chief Product Officer at Hims & Hers.

High quality AI-generated summary created from speaker-labeled transcript.

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.