Francois Chollet — Why the biggest AI models can't solve simple puzzles

ℹ️ EPISODE INFO

Released

June 11, 2024

Duration

1h 34m

Channel

Dwarkesh Podcast

Watch on YouTube

▶ Open ↗

📝 EPISODE DESCRIPTION

Here is my conversation with Francois Chollet and Mike Knoop on the $1 million ARC-AGI Prize they're launching today. I did a bunch of socratic grilling throughout, but Francois’s arguments about why LLMs won’t lead to AGI are very interesting and worth thinking through. It was really fun discussing/debating the cruxes. Enjoy! Check out ARC-AGI Prize here: https://arcprize.org 𝐄𝐏𝐈𝐒𝐎𝐃𝐄 𝐋𝐈𝐍𝐊𝐒

• Transcript: https://www.dwarkeshpatel.com/p/francois-chollet
• Apple Podcasts: https://podcasts.apple.com/us/podcast/francois-chollet-mike-knoop-llms-wont-lead-to-agi-%241/id1516093381?i=1000658672649
• Spotify: https://open.spotify.com/episode/7bmeJQOvXGy4LYl6YoiYYP?si=obUSUEwjSA6tkB8EBcb18w
• Follow me on Twitter: https://x.com/dwarkesh_sp

𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 00:00:00 – The ARC benchmark 00:11:53 – Why LLMs struggle with ARC 00:19:43 – Skill vs intelligence 00:28:38 – Do we need “AGI” to automate most jobs? 00:49:11 – Future of AI progress: deep learning + program synthesis 01:01:23 – How Mike Knoop got nerd-sniped by ARC 01:09:20 – Million $ ARC Prize 01:11:16 – Resisting benchmark saturation 01:18:51 – ARC scores on frontier vs open source models 01:27:02 – Possible solutions to ARC Prize

🎤 SPEAKERS

• Francois Chollet

  guest

• Dwarkesh Patel

  host

• Mike Knoop

  guest

🧾 EPISODE SUMMARY

In this episode of Dwarkesh Podcast, featuring Francois Chollet and Dwarkesh Patel, Francois Chollet — Why the biggest AI models can't solve simple puzzles explores aRC prize challenges LLM dominance, demands true machine intelligence progress Francois Chollet explains the ARC (Abstraction and Reasoning Corpus) benchmark and a new $1M ARC Prize as a way to measure and drive progress toward genuine machine intelligence, not just larger language models.

🔗 RELATED EPISODES

Machiavelli is the most misunderstood thinker of all time – Ada Palmer

The better AI gets, the smaller its share of the economy might get – Alex Imas and Phil Trammell

What rebuilding AlphaGo teaches us about self-play, RL, and future of LLMs - Eric Jang

How GPT, Claude, and Gemini are actually trained and served – Reiner Pope

Jensen Huang – Will Nvidia’s moat persist?

Terence Tao – How the world’s top mathematician uses AI