By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.
Aakash GuptaHow Hims & Hers Uses Evals to Let AI Talk to Patients (CPO Interview)
Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.
Great conversations run long. Their best ideas don't have to.
uListen is free on iPhone, iPad, Android, Chrome, and Claude.
See howCHAPTERS
Jump to publisher timestampsChapter summaries
Evals as a core PM skill: eval-driven development and a live LLM judge
Aakash opens by zooming in on evals as a must-have product skill for AI products. Dheerja explains that Hims & Hers uses eval-driven development and even runs a live “LLM judge” in production to score answers before they reach customers.
- •Evals are framed as a new essential PM capability for AI products
- •The team follows an eval-driven development loop (test → release → iterate)
- •A live LLM judge scores responses in production before sending to customers
- •Dheerja signals that quality control is more than just evals in practice
Why healthcare (and other “vegetable” categories) are adopting AI quickly
Dheerja describes her career in finance, media, and healthcare as building “vegetables”—categories people don’t naturally enjoy engaging with. She argues AI helps create more delightful, accessible experiences in systems that are traditionally archaic and hard to navigate.
- •Regulated categories can be hard consumer problems because users often avoid them
- •Finance and healthcare share traits: archaic systems and accessibility barriers
- •AI can scale experiences that feel like “financial advisors” or “concierge doctors”
- •Healthcare is described as highly siloed, making holistic support difficult without AI
Why AI shows up in regulated industries before “fun” apps
Aakash asks why AI appears to be emerging first in regulated spaces. Dheerja differentiates finance (no clear “killer AI product” yet, in her view) from healthcare, where the everyday pain and fragmentation make AI immediately valuable.
- •Finance and healthcare may be on different AI adoption timelines
- •Healthcare’s daily pain makes people turn to AI for symptom and journey support
- •The healthcare system is described as siloed across specialties and locations
- •Regulation and incentives (insurance, efficiency, siloing) shape what’s possible
Why Hims & Hers is going AI-native: the three building blocks
Dheerja explains the company’s shift toward an AI-native, longitudinal health experience. She outlines three components: diagnostics, treatments/medications, and a care experience that blends AI with human clinicians.
- •A “magic moment” convinced Dheerja the company could tackle longitudinal health
- •Diagnostics: unify health data from labs, wearables, medical history, and more
- •Treatments: verticalized medications enabling personalized dosing over time
- •Care layer: AI plus humans-in-the-loop for diagnosis and prescribing
Verticalized lab testing and personalized treatment loops
Dheerja goes deeper on how the company closes the data loop through at-home blood testing and internal processing. She also describes how treatment personalization can adjust medication dosage as biomarkers change over time.
- •At-home blood testing is shipped to members and processed through owned facilities
- •Owning the “full data loop” enables an ongoing picture of health
- •Treatments are personalized (e.g., dosage changes as levels improve)
- •The stack supports combining multiple treatment needs into one plan
Inside the new app: one chat thread with AI companion + doctors, plus synced data
Dheerja describes a newly launched greenfield app focused first on Hers weight loss. The experience centers on a single group-chat thread that includes the AI agent and doctors, paired with health data like a Bluetooth scale that syncs into the conversation.
- •A new two-tab app launched, starting with Hers weight loss as the first use case
- •Home screen is a group chat: agent and doctors in a single conversation thread
- •No copying between AI tools and doctors; clinicians join when needed
- •A free Bluetooth scale syncs weight data; the companion can detect plateaus and check in
Provider workflow: AI-augmented EMR summaries and clean handoffs back to the agent
Aakash asks what providers see, and Dheerja explains the doctor-facing charting tool and how AI supports it. The AI summarizes medical history, treatment plan, and conversation context so doctors can respond quickly, after which the agent can resume follow-up.
- •Doctors use an EMR-like patient charting tool inside the platform
- •AI provides streamlined summaries: history, plan, side effects, conversation state
- •Doctors respond with correct context; the agent then continues the flow
- •The system is designed to reduce context switching for both patient and provider
Clinical protocols as code: embedding clinicians into product development
Dheerja explains the cross-functional process of building in a regulated healthcare context. Clinical experts codify protocols, which are treated like code and checked in, while teams run daily clinical standups to triage what’s happening during dogfooding and early rollout.
- •The company has years of clinical data and outcomes at scale
- •Medical leads codify detailed clinical protocols by specialty
- •Protocols are “checked in” and updated like code, with clinical teams trained to contribute
- •Daily clinical standups help triage issues observed during dogfooding and early production
Turning protocols into evals: rubrics for accuracy, personalization, and tone
Dheerja details how clinical protocols become evals through collaboration between PMs and an evals function within the applied AI team. She also shares an example: a PM-created personalization rubric spanning multiple dimensions like context-awareness and voice/tone (which can vary by brand and journey).
- •Doctors codify protocols; PMs and the evals function translate them into evals
- •Evals include clinical accuracy plus additional dimensions beyond correctness
- •A personalization rubric defines what “personal” feels like across multiple dimensions
- •Voice and tone can vary between Hims vs Hers and between treatment journeys
Operating the system in the wild: live evaluation, daily review, and early results
Dheerja describes how evaluation shifts once the product is live, including monitoring triggered checks and reviewing real conversations daily. She shares early indicators: higher engagement and a large share of conversations resolved without a human clinician.
- •Offline evals matter early, then focus shifts heavily to live behavior
- •The team does daily standups reviewing conversations and production triggers
- •Internal dogfooding continues alongside rollout
- •Early metrics shared: 3X engagement and 80% of conversations resolved by the AI companion
How the team builds fast: specs in Git, lean execution, and “Wild West mode”
Aakash asks how work has changed, and Dheerja describes a lean, code-centric workflow. She explains that specs are committed into Git, execution moves quickly, while planning remains more traditional; overall, teams are encouraged to experiment rather than follow a mandated toolchain.
- •Specs (PRDs-style documents) are committed into Git; some work is auto-generated into PRs
- •Execution aims to move from spec to code very quickly, with checks and balances
- •Quarterly/yearly planning still exists, but day-to-day execution is changing most
- •The company encourages experimentation: provide scaffolding (Team OS/PM OS) but avoid rigid mandates
What the CPO job looks like now: enabling teams and codifying judgment
Dheerja reflects on what has changed at the leadership level. She notes tactical efficiency gains from AI, experiments with a personal agent to codify her judgment, and reframes her leverage as enabling teams to adopt effective workflows.
- •AI helps with efficiency (meeting prep, doc creation), but leadership fundamentals remain
- •She experimented with a personal agent to codify product judgment
- •She emphasizes enabling teams as the biggest leverage point for AI impact
- •She describes the era as especially fun, like returning to an engineering mindset
PM vs product builder: fundamentals over titles, and what hiring should test
Aakash raises the “product builder” framing, and Dheerja argues renaming doesn’t change the core job. She explains that fundamentals matter more as tactics get automated, then outlines the main traits she hires for and how she pressure-tests candidates using deep dives into past work and live thinking.
- •Titles matter less than whether the role still covers core product fundamentals
- •She argues best PMs were always builders at heart, even pre-AI
- •Fundamentals she emphasizes: judgment/taste, high slope/horsepower, and ownership/high agency
- •Hiring approach includes deep walkthroughs of the hardest shipped problem and live product cases; she prefers avoiding take-home homework
What makes an AI PM different: new “documents” like personalization rubrics and conversational guides
Dheerja explains why “AI PM” is distinct when you’re building AI products rather than simply using AI tools. She describes new artifacts—personalization rubrics and conversational guides—and the challenge of guiding an LLM without becoming overly deterministic or losing the benefits of flexibility.
- •Difference: building an AI product vs being a PM who uses AI tools
- •AI PMs must understand the tech deeply enough to partner with engineers in execution details
- •Personalization rubrics codify warmth, context, and brand voice beyond clinical correctness
- •Conversational guides evolved from overly prescriptive decision trees to higher-level guidelines after noticing they fought the LLM’s strengths
How to become an AI PM—and the common mistake in regulated industries
Dheerja describes how people can transition into AI PM work by being in an environment where AI is core and by getting hands-on reps, including side projects or internal adjacency. She closes with the biggest pitfall in regulated industries: treating legal/clinical/compliance as mere sign-off instead of builder partners, and underestimating the need for ongoing monitoring once an agent is live.
- •Transitioning is easiest where AI is central to the company’s product strategy
- •Hands-on building matters more than content consumption for learning to build AI products
- •An internal example: a PM moved from pharmacy/fulfillment systems to the AI team after building and enabling others
- •Key regulated-industry mistake: not embracing legal/clinical/compliance as true co-builders and not operationalizing ongoing monitoring for production risk
Publisher timestamps
- 1:26Why regulated industries are taking to AI firstRead this chapter from 1:27
- 5:07Why Hims & Hers is going AI-native
- 9:06AdsRead this chapter
- 11:12Inside the Hers app: AI and doctors in one chat
- 14:45Protocols as code and a live LLM judge
- 20:54Results: 80% resolved by AI, 3x engagementRead this chapter
- 21:42Specs in Git and Wild West modeRead this chapter
- 28:12The CPO job now, and PM vs product builder
- 32:17What she hires for and how she interviews
- 43:23What makes an AI PM differentRead this chapter
- 52:05How anyone can become an AI PMRead this chapter
- 58:30The mistake AI PMs make in regulated industries