Francois Chollet — Why the biggest AI models can't solve simple puzzles

ℹ️ EPISODE INFO

Released

June 11, 2024

Duration

1h 34m

Channel

Dwarkesh Podcast

Watch on YouTube

▶ Open ↗

📝 EPISODE DESCRIPTION

Here is my conversation with Francois Chollet and Mike Knoop on the $1 million ARC-AGI Prize they're launching today. I did a bunch of socratic grilling throughout, but Francois’s arguments about why LLMs won’t lead to AGI are very interesting and worth thinking through. It was really fun discussing/debating the cruxes. Enjoy! Check out ARC-AGI Prize here: https://arcprize.org 𝐄𝐏𝐈𝐒𝐎𝐃𝐄 𝐋𝐈𝐍𝐊𝐒

• Transcript: https://www.dwarkeshpatel.com/p/francois-chollet
• Apple Podcasts: https://podcasts.apple.com/us/podcast/francois-chollet-mike-knoop-llms-wont-lead-to-agi-%241/id1516093381?i=1000658672649
• Spotify: https://open.spotify.com/episode/7bmeJQOvXGy4LYl6YoiYYP?si=obUSUEwjSA6tkB8EBcb18w
• Follow me on Twitter: https://x.com/dwarkesh_sp

𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 00:00:00 – The ARC benchmark 00:11:53 – Why LLMs struggle with ARC 00:19:43 – Skill vs intelligence 00:28:38 – Do we need “AGI” to automate most jobs? 00:49:11 – Future of AI progress: deep learning + program synthesis 01:01:23 – How Mike Knoop got nerd-sniped by ARC 01:09:20 – Million $ ARC Prize 01:11:16 – Resisting benchmark saturation 01:18:51 – ARC scores on frontier vs open source models 01:27:02 – Possible solutions to ARC Prize

🎤 SPEAKERS

• Francois Chollet

  guest

• Dwarkesh Patel

  host

• Mike Knoop

  guest

🧾 EPISODE SUMMARY

In this episode of Dwarkesh Podcast, featuring Francois Chollet and Dwarkesh Patel, Francois Chollet — Why the biggest AI models can't solve simple puzzles explores aRC prize challenges LLM dominance, demands true machine intelligence progress Francois Chollet explains the ARC (Abstraction and Reasoning Corpus) benchmark and a new $1M ARC Prize as a way to measure and drive progress toward genuine machine intelligence, not just larger language models.

🔗 RELATED EPISODES

David Reich – Bronze Age shock, the Neanderthal puzzle, & the sudden spread of farming

Jensen Huang – TPU competition, why we should sell chips to China, & Nvidia’s supply chain moat

Dario Amodei — “We are near the end of the exponential”

Andrej Karpathy — “We’re summoning ghosts, not building animals”

Why Leonardo was a saboteur, Gutenberg went broke, and Florence was weird – Ada Palmer

Richard Sutton – Father of RL thinks LLMs are a dead end