Lenny's PodcastKevin Weil: Why evals are the new core skill in AI products
Through fine-tuning runs and writing evals against the fuzzy outputs; OpenAI builds at the edge of capabilities, betting on better models every two months.
Episode Details
EPISODE INFO
- Released
- April 10, 2025
- Duration
- 1h 31m
- Channel
- Lenny's Podcast
- Watch on YouTube
- ▶ Open ↗
EPISODE DESCRIPTION
Kevin Weil is the chief product officer at OpenAI, where he oversees the development of ChatGPT, enterprise products, and the OpenAI API. Prior to OpenAI, Kevin was head of product at Twitter, Instagram, and Planet, and was instrumental in the development of the Libra (later Novi) cryptocurrency project at Facebook. In this episode, you’ll learn:
- How OpenAI structures its product teams and maintains agility while developing cutting-edge AI
- The power of model ensembles—using multiple specialized models together like a company of humans with different skills
- Why writing effective evals (AI evaluation tests) is becoming a critical skill for product managers
- The surprisingly enduring value of chat as an interface for AI, despite predictions of its obsolescence
- How “vibe coding” is changing how companies operate
- What OpenAI looks for when hiring product managers (hint: high agency and comfort with ambiguity)
- “Model maximalism” and why today’s AI is the worst you’ll ever use again
- Practical prompting techniques that improve AI interactions, including example-based prompting
Find the transcript at: https://www.lennysnewsletter.com/p/kevin-weil-open-ai Brought to you by:
- Eppo—Run reliable, impactful experiments: https://www.geteppo.com/
- Persona—A global leader in digital identity verification: https://withpersona.com/lenny
- OneSchema—Import CSV data 10x faster: OneSchema — Import CSV data 10x faster
Where to find Kevin Weil:
Where to find Lenny:
- Newsletter: https://www.lennysnewsletter.com
- X: https://twitter.com/lennysan
- LinkedIn: https://www.linkedin.com/in/lennyrachitsky/
In this episode, we cover: (00:00) Kevin’s background (05:16) OpenAI’s new image model (08:13) The role of chief product officer at OpenAI (11:42) His recruitment story and joining OpenAI (15:59) Working at OpenAI (18:44) The importance of evals in AI (24:40) Opportunities in the space (26:34) Shipping quickly and consistently (29:47) Product reviews and iterative deployment (32:53) Winning consumer awareness (36:03) Designing thoughtful experiences (40:56) Chat as an interface for AI (45:21) Collaboration between researchers and product teams (48:05) Hiring product managers at OpenAI (53:06) How OpenAI uses AI: vibe coding, AI prototyping, and more (01:04:34) Raising kids in an increasingly intelligent AI world (01:08:07) Why Kevin feels optimistic about our AI future (01:14:20) The AI model you're using today is the worst AI model you'll ever use (01:17:58) Reflections on the Libra project (01:21:51) Lightning round and final thoughts Referenced:
- OpenAI: https://openai.com/
- The AI-Generated Studio Ghibli Trend, Explained: https://www.forbes.com/sites/danidiplacido/2025/03/27/the-ai-generated-studio-ghibli-trend-explained/
- Introducing 4o Image Generation: https://openai.com/index/introducing-4o-image-generation/
- Waymo: https://waymo.com/
- X: https://x.com
- Facebook: https://www.facebook.com/
- Instagram: https://www.instagram.com/
- Planet: https://www.planet.com/
- Sam Altman on X: https://x.com/sama
- A conversation with OpenAI’s CPO Kevin Weil, Anthropic’s CPO Mike Krieger, and Sarah Guo: https://www.youtube.com/watch?v=IxkvVZua28k
- OpenAI evals: https://github.com/openai/evals
- Deep Research: https://openai.com/index/introducing-deep-research/
- Ev Williams on X: https://x.com/ev
- OpenAI API: https://platform.openai.com/docs/overview
- Dwight Eisenhower quote: https://www.brainyquote.com/quotes/dwight_d_eisenhower_164720
- Inside Bolt: From near-death to ~$40m ARR in 5 months—one of the fastest-growing products in history | Eric Simons (founder & CEO of StackBlitz): https://www.lennysnewsletter.com/p/inside-bolt-eric-simons
- StackBlitz: https://stackblitz.com/
- Claude 3.5 Sonnet: https://www.anthropic.com/news/claude-3-5-sonnet
- Anthropic: https://www.anthropic.com/
- Four-minute mile: https://en.wikipedia.org/wiki/Four-minute_mile
- Chad: https://chatgpt.com/g/g-3F100ZiIe-chad-open-a-i
- Dario Amodei on LinkedIn: https://www.linkedin.com/in/dario-amodei-3934934/
- Figma: https://www.figma.com/
- Julia Villagra on LinkedIn: https://www.linkedin.com/in/juliavillagra/
- Andrej Karpathy on X: https://x.com/karpathy
...References continued at: https://www.lennysnewsletter.com/p/kevin-weil-open-ai Recommended books:
- Co-Intelligence: Living and Working with AI: https://www.amazon.com/Co-Intelligence-Living-Working-Ethan-Mollick/dp/059371671X
- The Accidental Superpower: Ten Years On: https://www.amazon.com/Accidental-Superpower-Ten-Years/dp/1538767341
- Cable Cowboy: https://www.amazon.com/Cable-Cowboy-Malone-Modern-Business/dp/047170637X
Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com. Lenny may be an investor in the companies discussed.
SPEAKERS
Kevin Weil
guestLenny Rachitsky
hostNarrator
other
EPISODE SUMMARY
In this episode of Lenny's Podcast, featuring Kevin Weil and Lenny Rachitsky, Kevin Weil: Why evals are the new core skill in AI products explores openAI’s CPO on building products atop rapidly evolving AI foundations Kevin Weil, Chief Product Officer at OpenAI, explains how building on AI is fundamentally different from past tech shifts because the underlying capabilities improve dramatically every few months. This forces product teams to plan loosely, ship quickly, and design around fuzzy, probabilistic model behavior instead of deterministic software. He highlights the rising importance of evals, fine-tuning, and ensembles of models, and argues that every serious product team will eventually embed ML researchers as core members. Weil also reflects on missed opportunities like Facebook’s Libra, the transformative potential of AI tutoring, and the skills he’s encouraging his kids (and future builders) to develop in an AI-first world.
RELATED EPISODES
