Francois Chollet — Why the biggest AI models can't solve simple puzzles

ℹ️ EPISODE INFO

Released

June 11, 2024

Duration

1h 34m

Channel

Dwarkesh Podcast

Watch on YouTube

▶ Open ↗

📝 EPISODE DESCRIPTION

Here is my conversation with Francois Chollet and Mike Knoop on the $1 million ARC-AGI Prize they're launching today. I did a bunch of socratic grilling throughout, but Francois’s arguments about why LLMs won’t lead to AGI are very interesting and worth thinking through. It was really fun discussing/debating the cruxes. Enjoy! Check out ARC-AGI Prize here: https://arcprize.org 𝐄𝐏𝐈𝐒𝐎𝐃𝐄 𝐋𝐈𝐍𝐊𝐒

• Transcript: https://www.dwarkeshpatel.com/p/francois-chollet
• Apple Podcasts: https://podcasts.apple.com/us/podcast/francois-chollet-mike-knoop-llms-wont-lead-to-agi-%241/id1516093381?i=1000658672649
• Spotify: https://open.spotify.com/episode/7bmeJQOvXGy4LYl6YoiYYP?si=obUSUEwjSA6tkB8EBcb18w
• Follow me on Twitter: https://x.com/dwarkesh_sp

𝐓𝐈𝐌𝐄𝐒𝐓𝐀𝐌𝐏𝐒 00:00:00 – The ARC benchmark 00:11:53 – Why LLMs struggle with ARC 00:19:43 – Skill vs intelligence 00:28:38 – Do we need “AGI” to automate most jobs? 00:49:11 – Future of AI progress: deep learning + program synthesis 01:01:23 – How Mike Knoop got nerd-sniped by ARC 01:09:20 – Million $ ARC Prize 01:11:16 – Resisting benchmark saturation 01:18:51 – ARC scores on frontier vs open source models 01:27:02 – Possible solutions to ARC Prize

🎤 SPEAKERS

• Francois Chollet

  guest

• Dwarkesh Patel

  host

• Mike Knoop

  guest

🧾 EPISODE SUMMARY

In this episode of Dwarkesh Podcast, featuring Francois Chollet and Dwarkesh Patel, Francois Chollet — Why the biggest AI models can't solve simple puzzles explores ARC prize challenges LLM dominance, demands true machine intelligence progress Francois Chollet explains the ARC (Abstraction and Reasoning Corpus) benchmark and a new $1M ARC Prize as a way to measure and drive progress toward genuine machine intelligence, not just larger language models.

🔗 RELATED EPISODES

Joe Carlsmith — Preventing an AI takeover

Patrick McKenzie — Money laundering, big tech censorship, SBF & Japan

Tony Blair — Why political leaders keep failing at major change

Leopold Aschenbrenner — 2027 AGI, China/US super-intelligence race, & the return of history

John Schulman (OpenAI Cofounder) — Reasoning, RLHF, & plan for 2027 AGI

Mark Zuckerberg — Llama 3, $10B models, Caesar Augustus, & 1 GW datacenters