François Chollet: Measures of Intelligence | Lex Fridman Podcast #120

François Chollet: Measures of Intelligence | Lex Fridman Podcast #120

Lex Fridman PodcastAug 31, 20202h 34m

Lex Fridman (host), François Chollet (guest), Narrator

Definition and measurement of general intelligence in humans and machinesHuman cognitive priors and core knowledge (objects, agents, space, number)Limits of current deep learning and large language models (e.g., GPT‑3)ARC Challenge as a benchmark for abstraction and reasoningPsychometrics, IQ, and the structure of cognitive abilities (g-factor, CHC theory)Types of generalization: robustness, flexibility, and extreme generalizationRole of culture, language, and external tools in augmenting human cognition

In this episode of Lex Fridman Podcast, featuring Lex Fridman and François Chollet, François Chollet: Measures of Intelligence | Lex Fridman Podcast #120 explores françois Chollet Redefines Intelligence, Critiques Deep Learning’s True Limits Lex Fridman and François Chollet discuss what intelligence really is, arguing it should be defined as the *efficiency of acquiring new skills in novel situations*, not the accumulation of skills themselves.

François Chollet Redefines Intelligence, Critiques Deep Learning’s True Limits

Lex Fridman and François Chollet discuss what intelligence really is, arguing it should be defined as the *efficiency of acquiring new skills in novel situations*, not the accumulation of skills themselves.

They contrast human cognitive abilities and priors with current machine learning systems, criticizing trends like scale-only language models (e.g., GPT‑3) and end‑to‑end deep learning for lacking robust, out-of-distribution generalization.

Chollet presents his ARC (Abstraction and Reasoning Corpus) benchmark as a psychometrics-inspired test for machine intelligence, built on explicit human core knowledge priors and designed to measure genuine abstraction and generalization rather than memorization.

They explore broader themes including developmental psychology, language as an operating system for the mind, limits of compression-as-cognition, the structure of human intelligence (g-factor), and the cultural, ripple-like meaning of human life.

Key Takeaways

Intelligence is about learning efficiency, not raw skill.

Chollet defines intelligence as the efficiency with which a system acquires new skills in tasks it was not prepared for. ...

Get the full analysis with uListen AI

You must distinguish the intelligent process from its artifacts.

A static chess program or a hand-engineered driving system encodes the *results* of human intelligence, not intelligence itself. ...

Get the full analysis with uListen AI

Human cognition rests on powerful innate priors missing in machines.

Humans come equipped with core knowledge systems—objectness/physics, agentness/goals, space/topology, and basic number sense—which underlie rapid learning and abstraction. ...

Get the full analysis with uListen AI

Scaling deep learning hits hard limits without genuine abstraction.

Models like GPT‑3 are impressive at generating plausible text but mainly perform sophisticated pattern matching over massive data. ...

Get the full analysis with uListen AI

A good intelligence test must control for priors and experience.

To fairly compare humans and machines, a test must make explicit which priors are allowed and tightly control exposure to training data. ...

Get the full analysis with uListen AI

Generalization has levels: robustness, flexibility, and extreme generalization.

Current ML excels at robustness (handling new samples from a known distribution). ...

Get the full analysis with uListen AI

Cognition is not just compression or prediction over past data.

While compression is a useful cognitive tool, real intelligence must hedge against future uncertainty and novelty. ...

Get the full analysis with uListen AI

Notable Quotes

Intelligence is the efficiency with which you acquire new skills at tasks that you did not previously know about.

François Chollet

We should not confuse a road-building company with one specific road.

François Chollet

Language is a kind of operating system for the mind.

François Chollet

You are not a very good source of unfakeable novelty.

François Chollet

Our actions today create ripples, and these ripples basically sum up the meaning of life.

François Chollet

Questions Answered in This Episode

If intelligence is defined as efficient skill acquisition, how should we redesign today’s ML systems to prioritize learning speed and adaptability over benchmark scores?

Lex Fridman and François Chollet discuss what intelligence really is, arguing it should be defined as the *efficiency of acquiring new skills in novel situations*, not the accumulation of skills themselves.

Get the full analysis with uListen AI

What would it practically look like to build AI architectures that explicitly incorporate human-like core knowledge priors such as objectness or agentness?

They contrast human cognitive abilities and priors with current machine learning systems, criticizing trends like scale-only language models (e. ...

Get the full analysis with uListen AI

How far can large language models progress without new mechanisms for factual grounding, consistency, and explicit reasoning over their latent spaces?

Chollet presents his ARC (Abstraction and Reasoning Corpus) benchmark as a psychometrics-inspired test for machine intelligence, built on explicit human core knowledge priors and designed to measure genuine abstraction and generalization rather than memorization.

Get the full analysis with uListen AI

In what concrete ways does the ARC Challenge change research directions compared to traditional benchmarks like ImageNet or standard NLP leaderboards?

They explore broader themes including developmental psychology, language as an operating system for the mind, limits of compression-as-cognition, the structure of human intelligence (g-factor), and the cultural, ripple-like meaning of human life.

Get the full analysis with uListen AI

Given that cognition is not just compression, what alternative theoretical frameworks (beyond prediction and compression) might better capture intelligence in dynamic, uncertain environments?

Get the full analysis with uListen AI

Transcript Preview

Lex Fridman

The following is a conversation with Francois Chollet, his second time on the podcast. He's both a world-class engineer and a philosopher in the realm of deep learning and artificial intelligence. This time, we talk a lot about his paper titled On the Measure of Intelligence that discusses how we might define and measure general intelligence in our computing machinery. Quick summary of the sponsors: Babbel, MasterClass, and Cash App. Click the sponsor links in the description to get a discount and to support this podcast. As a side note, let me say that the serious, rigorous, scientific study of artificial general intelligence is a rare thing. The mainstream machine learning community works on very narrow AI with very narrow benchmarks. This is very good for incremental and sometimes big incremental progress. On the other hand, the outside-the-mainstream, renegade, you could say, AGI community works on approaches that verge on the philosophical and even the literary without big public benchmarks. Walking the line between the two worlds is a rare breed, but it doesn't have to be. I ran the AGI series at MIT as an attempt to inspire more people to walk this line. DeepMind and OpenAI for a time, and still on occasion, walk this line. Francois Chollet does as well. I hope to also. It's a beautiful dream to work towards and to make real one day. If you enjoy this thing, subscribe on YouTube, review it with five stars on Apple Podcasts, follow on Spotify, support on Patreon, or connect with me on Twitter @LexFridman. As usual, I'll do a few minutes of ads now and no ads in the middle. I try to make these interesting, but I give you timestamps so you can skip. But still, please do check out the sponsors by clicking the links in the description. It's the best way to support this podcast. This show is sponsored by Babbel, an app and website that gets you speaking in a new language within weeks. Go to babbel.com and use code LEX to get three months free. They offer 14 languages, including Spanish, French, Italian, German, and yes, Russian. Daily lessons are 10 to 15 minutes, super easy, effective, designed by over 100 language experts. Let me read a few lines from the Russian poem, (Russian) , by Alexander Blok, that you'll start to understand if you sign up to Babbel. Ночь. Улица. Фонарь. Аптека. Бессмысленный и тусклый свет. Живи еще хоть четверть века. Все будет так. Исхода нет. Now, I say that you'll start to understand this poem because Russian starts with a language and ends with vodka. Now, the latter part is definitely not endorsed or provided by Babbel and will probably lose me the sponsorship, although it hasn't yet. But once you graduate with Babbel, you can enroll in my advanced course of late-night Russian conversation over vodka. No app for that yet. So get started by visiting babbel.com and use code LEX to get three months free. This show is also sponsored by MasterClass. Sign up at masterclass.com/lex to get a discount and to support this podcast. When I first heard about MasterClass, I thought it was too good to be true. I still think it's too good to be true. For $180 a year, you get an all-access pass to watch courses from, to list some of my favorites, Chris Hadfield on space exploration; hope to have him on this podcast one day. Neil deGrasse Tyson on scientific thinking and communication. Neil too. Will Wright, creator of SimCity and Sims on game design. Carlos Santana on guitar. Garry Kasparov on chess. Daniel Negreanu on poker and many more. Chris Hadfield explaining how rockets work and the experience of being launched to space alone is worth the money. By the way, you can watch it on basically any device. Once again, sign up at masterclass.com/lex to get a discount and to support this podcast. This show, finally, is presented by Cash App, the number one finance app in the App Store. When you get it, use code LEXPODCAST. Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1. Since Cash App allows you to send and receive money digitally, let me mention a surprising fact related to physical money. Of all the currency in the world, roughly 8% of it is actually physical money. The other 92% of the money only exists digitally, and that's only going to increase. So again, if you get Cash App from the App Store or Google Play and use code LEXPODCAST, you get 10 bucks. And Cash App will also donate $10 to FIRST, an organization that is helping to advance robotics and STEM education for young people around the world. And now, here's my conversation with Francois Chollet. What philosophers, thinkers, or ideas had a big impact on you growing up? And today?

Install uListen to search the full transcript and get AI-powered insights

Get Full Transcript

Get more from every podcast

AI summaries, searchable transcripts, and fact-checking. Free forever.

Add to Chrome