By playing, you agree to uListen's Terms and Privacy Policy and YouTube's Terms of Service.

Lenny's Podcast

Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini)

Tulsee Doshi is head of product for Google’s frontier AI models, including Gemini and other multimodal models, such as Nano Banana, Genie, Veo, and Lyria.

Tulsee DoshiguestLenny Rachitskyhost

Nov 8, 202658mWatch on YouTube ↗

Summaries, chapters, transcripts, and FAQs on this page are made by uListen, not YouTube.

CHAPTERS

Jump to publisher timestamps

Chapter summaries

Models are “grown,” not predictable: the PM’s job is to find the magic
Tulsee Doshi describes why model development doesn’t behave like traditional product development: even with clear goals, outcomes can be surprising. She compares training to “Snape’s potions class,” where you can choose ingredients but should expect unexpected emergent behavior. She frames a key PM responsibility as identifying where a model uniquely shines and turning those surprises into user value.
- •Model goals and eval targets matter, but model behavior can still be unpredictable
- •Model development is monitored and made as scientific as possible, yet surprises remain
- •A PM must study what comes out of training and spot where the model is uniquely strong
- •Model “vibes” include behavior, persona, and how it feels to communicate with it
Google’s full‑stack AI strategy: frontier models plus breadth across cost and use cases
Lenny asks about perceptions that Google lags in the AI race, and Tulsee explains Google’s strategy and ambitions. She emphasizes a full-stack approach spanning hardware, model building, and products at massive scale. She also says Google wants to “win the frontier conversation,” while serving a wide range of user needs along the cost-quality curve.
- •Google’s strategy is described as full stack: hardware to models to end-user products
- •Tulsee says Google does want to win the frontier conversation and be seen at the frontier
- •Goal includes models across the cost-to-quality curve for different user needs
- •Examples of users include search users, engineers, and cyber defenders
AI safety in practice: monitoring, mitigations, and using AI to strengthen cyberdefenses
Drawing on her background leading responsible AI product work, Tulsee explains how safety is embedded from training through deployment. She highlights monitoring as a key mitigation, including detecting misalignment and intervening appropriately. She also argues the safety conversation should include how AI can harden systems, describing partnerships with cyber defenders and a vulnerability-patching example.
- •Safety is described as embedded across training and deployment, not bolted on later
- •Monitoring is highlighted as a mitigation to detect misalignment and support intervention
- •Monitoring is also framed as a way to learn where models fail and improve over time
- •AI is positioned as a tool to help identify and patch vulnerabilities with cyber partners
Shipping fast vs. great UX: experimenting with form factors while matching the user
Lenny raises the tension between pushing frontier capability out quickly and maintaining usable product experiences. Tulsee argues teams must experiment with new UX paradigms and form factors, while recognizing that different users require different interfaces. She contrasts mass-market experiences like search with power-user tools that can handle more complexity.
- •New UX paradigms and form factors are necessary as model capabilities expand
- •Examples of different form factors include NotebookLM and products for media generation/editing
- •UX should vary based on user type: broad consumer vs. power users
- •Power-user environments can support complex workflows and commands more readily
The model itself has a UX: behavior specs, rubrics, and reward modeling
Tulsee argues that UX isn’t only the app UI—using the model has UX properties like transparency, communication style, and what questions it asks back. She describes PM involvement in shaping model behavior direction using user feedback and a “spec” for what good looks like. That behavioral intent is translated into rubrics and evaluation data that feeds training approaches such as reward modeling.
- •Model UX includes transparency, communication quality, and interaction behavior
- •PMs help define what “good” behavior looks like for the model using feedback
- •Behavior goals are translated into specs and rubrics for evaluation
- •Rubrics produce data that can be used to shape model behavior through training processes
Why PMs are “back”: taste, signal vs. noise, and turning speed into alignment
Lenny observes renewed appreciation for PMs as building gets easier and teams risk shipping lots of low-value work. Tulsee agrees, emphasizing that taste and insight become differentiators when anyone can build quickly. She also stresses the PM role in driving consensus and converting organizational chaos into clarity—especially as speed increases.
- •When building is easy, the differentiator becomes building something great
- •Taste and insight are framed as critical to separate signal from noise
- •In large orgs, PMs help align people and drive consensus amid rapid change
- •Speed without structure can introduce chaos; PMs add clarity and order
Hiring research PMs: model “vibes,” hands-on eval building, and adaptability
Tulsee explains what she increasingly looks for in PM hires for frontier-model work. She values PMs who actively use models, form opinions, and can stress-test capabilities based on real use cases. She also looks for technical scrappiness—diving into data and building evals—and for adaptability as roadmaps must be revisited frequently in a fast-changing space.
- •Strong candidates actively play with models and have informed opinions about them
- •Stress-testing models on meaningful use cases is emphasized
- •Scrappiness includes diving into data and being able to build an eval
- •Adaptability matters because roadmaps can change quickly as the field evolves
How to build your first eval: start tiny, find signal, and scale up
Tulsee shares practical guidance for people new to evaluations. She recommends starting small—just a handful of prompts—rather than aiming for a huge benchmark immediately. By rating results yourself first, you can clarify what “good” means for the task and then scale the evaluation as you learn.
- •Start with a very small set of prompts rather than a massive benchmark
- •Even a 20-prompt mini-eval can reveal meaningful differences between models
- •Begin by rating outputs yourself to learn what you consider “good”
- •Collecting good examples first helps clarify targets before scaling evaluation
Getting better at judging “vibes”: test what you care about and track progress
Lenny asks how to develop intuition for what a new model is better at. Tulsee says there’s no single right approach and encourages people to stress test models on tasks they personally care about and find exciting. She notes people probe different dimensions—visual aesthetics, writing, synthesis, financial analysis—and recommends choosing tasks where models show promise but still aren’t perfect, making differences easier to see.
- •There’s no single correct method; choose tests tied to what excites you
- •People probe models via different tasks (visual, writing, synthesis, analysis)
- •Pick tasks where models show “signs of life” but are not yet perfect
- •Comparing how models handle the same task reveals performance differences and strengths
Improving skin tone representation: updating outdated evals with a better scale
Tulsee recounts responsible AI work improving skin tone representation in products and models. She explains limitations of the Fitzpatrick scale: it has only six tones, was created for a different purpose, and skews toward lighter tones. She describes collaborating with Professor Monk to adopt a 10-skin-tone scale and redoing evaluations, which helped drive improvements in areas like camera and image generation.
- •The Fitzpatrick scale is described as too limited and misaligned with global representation needs
- •It was originally built for healthcare purposes and skewed toward lighter skin tones
- •Collaboration with Professor Monk led to adopting a 10-skin-tone scale for evaluations
- •Evaluations were updated and used to improve product and model behavior around skin tones
Evaluating the full experience: judge quality, serving setup, and end-to-end effects
Tulsee expands evaluation beyond just the base model, arguing evals must reflect what end users actually experience. She points out that judge quality affects measurement, and production mitigations and filters can change observed performance. Using audio as an example, she highlights that serving and delivery can impact voice naturalness and inflections, making end-to-end evaluation critical.
- •Evals should be holistic and reflect the full end-user experience
- •Judge quality can materially affect evaluation accuracy
- •Production mitigations/filters and serving setups can change model performance
- •Audio evaluation must account for delivery factors like naturalness, inflections, and tenor
From crowdsourced flash teams to agent swarms: organizing capability at the moment
Lenny brings up agent swarms and asks about Tulsee’s earlier research. Tulsee describes her Stanford work on “flash teams,” which focused on assembling crowdsourced workers with complementary skills to complete tasks efficiently. She connects those principles to today’s agent systems, where similar ideas can be applied to orchestrating sub-agents into effective short-lived teams.
- •Flash teams research focused on assembling crowdsourced workers with the right skills for tasks
- •The approach aimed to make crowdsourcing more effective and efficient
- •Tulsee maps the same principles to organizing systems of sub-agents today
- •The appeal is quickly mobilizing a capable team for a task and then spinning it down
The 60-20-20 rule: build for today while reaching toward near-future capability
Tulsee introduces her framework for building products atop rapidly improving models. Instead of only shipping what works now, she suggests a portfolio where most of the product works today, some works well enough with near-term improvements, and some targets the aspirational experience enabled by future capabilities. The goal is to ship value now while positioning the product to become “magical” as models improve.
- •Model capabilities change continuously, so product plans must anticipate improvement
- •60%: features that work today and make the prototype functional
- •20%: features that aren’t perfect but are close enough to be viable soon
- •20%: aspirational experiences with visible “glimmers of light” that could unlock magic
Keeping products adaptable as models improve: retest often and understand the “harness”
Tulsee and Lenny discuss how rigid systems built around older model limitations can hold teams back as models improve. Tulsee emphasizes retesting and avoiding force-fitting new models into old structures that may hamper them. She then defines an AI “harness” as the wrapper around the model—system instructions, tools, context, and skills—creating the environment in which the model operates.
- •Rigid systems can become outdated as models get better, requiring redesign or simplification
- •Consistent retesting helps avoid hampering a new model with old constraints
- •A harness is described as the wrapper around the model that shapes its environment
- •Harness components include system instructions, tools, context/information, and skills
Upskilling PMs, avoiding burnout and AI slop, and closing with lightning-round personal notes
Tulsee explains how her team shares skills internally via repositories, peer-led workshops, and dedicated time to practice tools, including pulling PMs out of day-to-day work for focused sessions. She reflects on burnout as driven by chaos and unfulfilling work, and hopes AI can remove grunt work to increase time for creative thinking. She also discusses avoiding “AI slop” by using AI for data-heavy updates while keeping human insight central, then closes with a lightning round covering books, shows, tools, mottos, dance, and how listeners can help (by using products and sharing feedback).
- •Upskilling approach includes sharing skills, running workshops, and creating time/space to learn tools
- •Burnout is linked to churn/chaos and doing work you don’t enjoy; meaningful work helps
- •To reduce AI slop, use AI for data gathering and updates, while humans provide insight
- •Lightning round includes reading and viewing picks, a favorite AI tool, life mottos, dance background, and a request for user feedback on Google products

Publisher timestamps

Get more out of YouTube videos.

Read along on your phone, search it beside the video, or ask your favorite AI app what was said.