Edwin Chen: Why optimizing for benchmarks creates AI sloth

ℹ️ EPISODE INFO

Released

December 7, 2025

Duration

1h 10m

Channel

Lenny's Podcast

Watch on YouTube

▶ Open ↗

📝 EPISODE DESCRIPTION

Edwin Chen is the founder and CEO of Surge AI, the company that teaches AI what’s good vs. what’s bad, powering frontier labs with elite data, environments, and evaluations. Surge surpassed $1 billion in revenue with under 100 employees last year, completely bootstrapped—the fastest company in history to reach this milestone. Before founding Surge, Edwin was a research scientist at Google, Facebook, and Twitter and studied mathematics, computer science, and linguistics at MIT. *We discuss:*

1. How Surge reached over $1 billion in revenue with fewer than 100 people by obsessing over quality
2. The story behind how Claude Code got so good at coding and writing
3. The problems with AI benchmarks and why they’re pushing AI in the wrong direction
4. How RL environments are the next frontier in AI training
5. Why Edwin believes we’re still a decade away from AGI
6. Why taste and human judgment shape which AI models become industry leaders
7. His contrarian approach to company building that rejects Silicon Valley’s “pivot and blitzscale” playbook
8. How AI models will become increasingly differentiated based on the values of the companies building them

*Brought to you by:* Vanta—Automate compliance. Simplify security: https://vanta.com/lenny WorkOS—Modern identity platform for B2B SaaS, free up to 1 million MAUs: https://workos.com/lenny Coda—The all-in-one collaborative workspace: https://coda.io/lenny *Transcript:* https://www.lennysnewsletter.com/p/surge-ai-edwin-chen *My biggest takeaways (for paid newsletter subscribers):* https://www.lennysnewsletter.com/i/180055059/my-biggest-takeaways-from-this-conversation *Where to find Edwin Chen:*

• X: https://x.com/echen
• LinkedIn: https://www.linkedin.com/in/edwinzchen
• Surge’s blog: https://surgehq.ai/blog

*Where to find Lenny:*

• Newsletter: https://www.lennysnewsletter.com
• X: https://twitter.com/lennysan
• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/

*In this episode, we cover:* (00:00) Introduction to Edwin Chen (04:48) AI’s role in business efficiency (07:08) Building a contrarian company (08:55) An explanation of what Surge AI does (09:36) The importance of high-quality data (13:31) How Claude Code has stayed ahead (17:37) Edwin’s skepticism toward benchmarks (21:54) AGI timelines and industry trends (28:33) The Silicon Valley machine (33:07) Reinforcement learning and future AI training (39:37) Understanding model trajectories (41:11) How models have advanced and will continue to advance (42:55) Adapting to industry needs (44:39) Surge’s research approach (48:07) Predictions for the next few years in AI (50:43) What’s underhyped and overhyped in AI (52:55) The story of founding Surge AI (01:02:18) Lightning round and final thoughts *Referenced:*

• Surge: https://surgehq.ai
• Surge’s product page: https://surgehq.ai/products
• Claude Code: https://www.claude.com/product/claude-code
• Gemini 3: https://aistudio.google.com/models/gemini-3
• Sora: https://openai.com/sora
• Terrence Rohan on LinkedIn: https://www.linkedin.com/in/terrencerohan
• Richard Sutton—Father of RL thinks LLMs are a dead end: https://www.dwarkesh.com/p/richard-sutton
• The Bitter Lesson: http://www.incompleteideas.net/IncIdeas/BitterLesson.html
• Reinforcement learning: https://en.wikipedia.org/wiki/Reinforcement_learning
• Grok: https://grok.com
• Warren Buffett on X: https://x.com/WarrenBuffett
• OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more | Kevin Weil (CPO at OpenAI, ex-Instagram, Twitter): https://www.lennysnewsletter.com/p/kevin-weil-open-ai
• Anthropic’s CPO on what comes next | Mike Krieger (co-founder of Instagram): https://www.lennysnewsletter.com/p/anthropics-cpo-heres-what-comes-next
• Brian Armstrong on LinkedIn: https://www.linkedin.com/in/barmstrong
• Interstellar on Prime Video: https://www.amazon.com/Interstellar-Matthew-McConaughey/dp/B00TU9UFTS
• Arrival on Prime Video: https://www.amazon.com/Arrival-Amy-Adams/dp/B01M2C4NP8
• Travelers on Netflix: https://www.netflix.com/title/80105699
• Waymo: https://waymo.com
• Soda versus pop: https://flowingdata.com/2012/07/09/soda-versus-pop-on-twitter

*Recommended books:*

• Stories of Your Life and Others: https://www.amazon.com/Stories-Your-Life-Others-Chiang/dp/1101972122
• The Myth of Sisyphus: https://www.amazon.com/Myth-Sisyphus-Vintage-International/dp/0525564454
• Le Ton Beau de Marot: In Praise of the Music of Language: https://www.amazon.com/dp/0465086454
• Gödel, Escher, Bach: An Eternal Golden Braid: https://www.amazon.com/G%C3%B6del-Escher-Bach-Eternal-Golden/dp/0465026567

_Production and marketing by https://penname.co/._ _For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com._ Lenny may be an investor in the companies discussed.

🎤 SPEAKERS

• Lenny Rachitsky

  host

• Edwin Chen

  guest

• Narrator

  other

🧾 EPISODE SUMMARY

In this episode of Lenny's Podcast, featuring Lenny Rachitsky and Edwin Chen, Edwin Chen: Why optimizing for benchmarks creates AI sloth explores bootstrapped AI Data Giant Surge Reimagines Responsible Path To AGI Founder Edwin Chen explains how Surge AI became a $1B-revenue, sub‑100‑person, fully bootstrapped company by obsessing over ultra‑high‑quality training data for frontier models like ChatGPT, Claude, and Gemini.

🔗 RELATED EPISODES

How to build a company that withstands any era | Eric Ries, Lean Startup author

Head of Claude Code: What happens after coding is solved | Boris Cherny

Building product at Stripe: craft, metrics, and customer obsession | Jeff Weinstein (Product lead)

Building a world-class data org | Jessica Lachs (VP of Analytics and Data Science at DoorDash)

What most people miss about marketing | Rory Sutherland (Vice Chairman of Ogilvy UK, author)

5 essential questions to craft a winning strategy | Roger Martin (author, advisor, speaker)