Francois Chollet — Why the biggest AI models can't solve simple puzzles

Francois Chollet 0:00 LLMs are very good at memorizing study programs. If you scale up the size of your database, you are not increasing the intelligence of the system one bit.

Dwarkesh Patel 0:09 I feel like you're using words like memorization, which we would never use for human children if, if they can just solve any ar- uh, arbitrary algebraic problem. You wouldn't say, like, they've memorized algebra. They'd say they've learned algebra.

Mike Knoop 0:19 So we got a million dollar prize pool, and there's a $500,000 prize for the first team that can get to the 85% benchmark. If ARC survives three months from here, we'll, we'll up the prize.

Francois Chollet 0:28 OpenAI basically set back progress towards AGI by probably, like, five to 10 years. They caused this complete closing down of Frontier Research Publishing. And now LLMs have, uh, sucked the oxygen out of the room. Like, everyone is just doing LLMs.

Dwarkesh Patel 0:44 Okay. Today, I have the pleasure to speak with Francois Chollet, who is a AI researcher at Google and creator of Keras. And he's launching a prize in collaboration with Mike Knuth, the co-founder of Zapier, who we'll also be talking to in a second. A million dollar prize to solve the ARC benchmark that he created. So first question, what is the ARC benchmark and why do you even need this prize? Why won't the biggest LLM we have in a year be able to just saturate it?

Francois Chollet 1:11 Sure. So ARC is intended as a kind of IQ test for machine intelligence. And what makes it different from, uh, most LM benchmarks out there is that it's designed to be resistant to memorization. So if you look at the way LLMs work, they are basically this, uh, big interpretive memory. And the way you scale up their capabilities is by trying to cram as much, uh, knowledge and patterns as possible into them. And, uh, by contrast, uh, ARC does not require a lot of knowledge at all. It's designed to only require what's known as, uh, core knowledge, which is, uh, basic knowledge about things like, um, elementary physics, object nests, counting, that sort of thing. Um, the sort of knowledge that any four-year-old or five-year-old, uh, possesses, right? Um, but what's interesting is that each puzzle in ARC is novel, is something that you've probably not encountered before, even if you've memorized the entire internet. And that's what, that's what makes it... (clears throat) Sorry. That's what, uh, makes, makes ARC challenging for LLMs. And so far, LLMs have not, uh, been doing very well on it. In fact, the approaches that are working well, uh, are more towards, uh, discrete program search, program synthesis.

Dwarkesh Patel 2:25 Mm. So f- first of all, I'll, I'll make a comment that I'm glad that as a skeptic of LLM, you have put out, yourself, a benchmark that... Uh, is it accurate to say that suppose that the biggest model we have in a year is able to get 80% on this, then your view would be we are on track, uh, to AGI with LLMs? How would you think about that?

Francois Chollet 2:45 Right. Um, I'm pretty skeptical that we're going to see LLM do 80% in a year.

Dwarkesh Patel 2:50 Yeah.

Francois Chollet 2:51 Uh, that said, if we do see it, you would also have to look at how this was achieved. If you just, uh, train the model on millions or billions of puzzles similar to ARC so that you're relying on, uh, the ability to have some overlap between the tasks that you train on and the tasks that you're going to see at this time, then you're still using memorization, right? And maybe, maybe it can work. You know, hopefully, um, ARC is gonna be good enough that it's gonna be, uh, resistant to this sort of attempt and brute forcing.

Dwarkesh Patel 3:21 Mm.

Francois Chollet 3:22 Um, but, you know, you never know. May- maybe, maybe it could happen. I'm not saying it's not going to happen. ARC is not a perfect benchmark. Maybe, maybe it has flaws, flaws. Maybe it could be hacked in that way.

Dwarkesh Patel 3:32 Mm. I'm... So I guess I'm curious about what would GPT-5 ha- ha- have to do that you're very confident that, you know, it's on the path to AGI?

Francois Chollet 3:42 What, what would make me change my mind about LLMs is basically if I start seeing, uh, a critical mass of cases where you show the model with something it has not seen before, a task that's actually novel from the perspective of its training data-

Dwarkesh Patel 3:57 Mm.

Francois Chollet 3:57 ... something that's not in the training data, if it can actually adapt on the fly. Um, and this is true for LLMs, but really, uh, this would catch my attention with any... for AI, any AI technique out there. If I can see the ability to adapt to novelty on the fly to pick up new skills efficiently, then, uh, uh, I would be extremely interested. I wou- I would, I would think this is, uh, uh, on the path to AGI.

Dwarkesh Patel 4:21 So the advantage they have is that they do get to see everything. Maybe I'll take issue with how much they are relying on that. But le- suppose that they are relying. Obviously, they're relying on that more than humans do. To the extent that they do have so much in distribution, to the extent that we have trouble distinguishing whether a sin- uh, an example is in distribution or not, well, if they have everything in distribution, then they can do everything that we can do. Maybe it's not in distribution for us. Why, uh, why is it so crucial that it has to be out of distribution for them? Uh, you know, eh, why can't we just leverage the fact that they do get to see everything?

Francois Chollet 4:54 Right. What, you're, you're asking basically what's the difference between actual intelligence, which is the ability to adapt to things you've not been prepared for, and pure memorization, like reciting, uh, what you've seen before? And it's not just some semantic difference. Uh, the big difference is that you can never, uh, pre-train on everything that you might see at test time, right? Uh, because the world changes all the time. So it's not just the fact that the space of possible tasks is infinite. And if, even if you're trained on millions of them, uh, you've, you've only seen 0% of the total space. It's also the fact that the world is changing every day, right? Um, this is why, uh, we, uh, uh, uh, the human species has developed intelligence in the first place.If there was such a thing as a distribution for the world, for the universe, for our lives, then we would not need intelligence at all. In fact, uh, many, uh, creatures, many insects, for instance, do not have intelligence. Instead, what they have is, uh, they have in their, in their connectome, uh, in their genes, uh, hardcoded programs, behavioral programs that map some stimuli to appropriate response. And they can actually navigate their lives, their environments in a way that's very evolutionarily fits, that way, without needing to learn anything. And well, if our environment was, uh, static enough, predictable enough, uh, what would have happened is that evolution would have found the perfect behavioral program, a hardcoded, static behavioral program, would have written it, uh, into our genes. We would have a hardcoded brain connectome, and that's what we would be running on. But no, that's not what happened. Instead, we have general intelligence. So we are born with extremely little knowledge about the world, but we are born with the ability to learn very efficiently and to adapt in the face of things that we, we, we've never seen before. And that's what makes us unique, and that's what, that's what is really, really challenging to recreate in machines.

Dwarkesh Patel 6:54 I, I want to rabbit hole on that a little bit. But before I do that, maybe, uh, I'm gonna overlay some examples of what an ARC-like challenge look like for, uh, for the YouTube audience. But maybe for people listening on audio, can you just describe what, what, what would an... a sample ARC challenge look like?

Francois Chollet 7:07 Sure. So one ARC puzzle, it looks kind of like an IQ test puzzle. You've got a number of demonstration input-output pairs. So, uh, uh, one pair is, uh, made of two grids. So one grid shows you an input and the second grid shows you, uh, what you should produce as a response to that input. And you get, uh, a couple, uh, pairs like this to demonstrate the nature of the task, to demonstrate what you're supposed to do with your inputs. And then you get, uh, a new test input, and your job is to produce the corresponding, uh, test outputs. You look at the demonstration pairs, and from that, you figure out what you're supposed to do and you show that you've understood it on this new test pair. And, um, importantly, in order to... the sort of like knowledge basis that you need in order to approach these chan- these challenges is you just need core knowledge. And core knowledge is, uh, it's basically the knowledge of what makes an object, uh, basic counting, basic geometry, topology, symmetries, uh, that sort of thing. So extremely basic knowledge. LLMs for sure possess such knowledge. Any child possesses, uh, such knowledge. Um, and what's really interesting is that each puzzle is new. So it's not something that you're gonna find, uh, uh, uh, elsewhere on the internet, for instance. Uh, and that means that whether it's as a human or as machine, every puzzle, uh, you have to approach it from scratch. You have to actually reason your way through it. You cannot just fetch the response from your memory.

Dwarkesh Patel 8:43 So the core knowledge, one contention here is we are only now getting multimodal models who, because of the data they're trained on, are trained to do spatial reasoning. Whereas obviously not only humans, but for billions of years of evolution, we've had... our ancestors have had to learn how to understand abstract, physical and spatial properties and recognize the patterns there. And so one view would be in the next year, as we gain models that are multimodal native, that isn't just a sort of second class that is an add-on, but the multimodal capability is a priority, that it will understand these kinds of patterns because that's something we'd see natively. Whereas right now what ARC sees is some JSON string of 100100, and it's supposed to recognize a pattern there. And even if you showed a human such a g- like a just a, a sequence of these kinds of numbers, it would have a challenge making, uh, making sense of what, what kind of question you're asking it. So why would it be the case that as soon as we get multimodal models, which we're on the path to unlock right now, they're gonna be so much better at ARC-type spatial reasoning?

Francois Chollet 9:52 That, that's an empirical question, so I guess we're gonna see the answer within a few months. But, uh, my answer to that is, you know, our grids, they're just, uh, discrete 2D grids of symbols. They're pretty, pretty small. Like, it's not like, uh, if you flatten an i- an image as a sequence of pixels, for instance, uh, then you get something that's actually very, very difficult to parse. But that's not true for ARC because the grids are very small. Uh, you only have 10 possible symbols. So there's these 2D grids that are actually very easy to flatten, uh, as sequences and transformers. LLMs, they're very good at parsing such sequences. In fact, uh, you can show that LLMs do fine, uh, with processing ARC-like data, uh, by simply fine-tuning, uh, LL- uh, LLM on, uh, some subsets, uh, of the tasks and then trying to test it on small variations of these tasks. And you see that yeah, the, the LLM can encode just fine solution programs for tasks that it has seen before. So it does not really have a problem parsing the input or figuring out, uh, uh, um, the program. The reason why, uh, LLMs, uh, uh, don't do well on ARC is really just, uh, the unfamiliarity aspect. The fact that each new task is different from eth- e- every other task. You cannot... basically, you cannot memorize the solution programs in advance. You have to synthesize a new solution program on the fly for each new task. And that's really what LLMs are struggling with.

Dwarkesh Patel 11:20 Mm-hmm. So before I do more a devil's advocate, I just wanna step back and explain why I'm especially interested in having this conversation. And obviously the million dollar ARC Prize, I'm, uh, uh, excited to actually play around with it myself and hopefully, um, the Vesuvius Challenge, which was Nat Friedman's prize for solving, uh, decoding scrolls, the winner of that, decoding the scrolls from... that were buried in the volcanoes in, uh, the Herculaneum library, that was solved by a 22-year-old who was listening to the podcast, Luke Farritor. So hopefully somebody listening will find this challenge intriguing and

Dwarkesh Patel 11:53 find a solution. So I'm... and the reason I... I've had on recently a lot of people who are, uh-... bullish on LLMs and I've had discussions with them before interviewing you about, how do we explain the fact that LLMs don't seem to be natively performing that well on ARC? And I found their explanations somewhat contrived and I'll try out some, uh, some of the reasons on you. But it is actually an intriguing fact that they actually... these are- some of these problems are relatively straightforward for humans to understand and they do struggle with them if you just input them natively.

Francois Chollet 12:25 All of them are very easy for humans. Like, any- any smart human should be able to do 90%, 95% on ARC.

Dwarkesh Patel 12:32 A smart human.

Francois Chollet 12:33 A smart human. But even a five-year-old, so with very, very little knowledge, they could- they could definitely do over 50%.

Dwarkesh Patel 12:39 Hmm. So le- let's talk about that because you, y- I agree that smart humans will do very well on this test, but the im- average human will probably do, you know, mediocre.

Francois Chollet 12:53 Not- not really average. So we actually tried with average humans, uh, they score about 85.

Dwarkesh Patel 12:58 That was with Amazon Mechanical Turk workers, right?

Francois Chollet 13:00 Yeah, that's right.

Dwarkesh Patel 13:00 I'm- I- I honestly don't know the demographic profile of Amazon Mechanical Turk workers, but imagine just interacting with, uh, the- the platform that Amazon has set up to do remote work. That's not the median human across the planet, I'm guessing.

Francois Chollet 13:11 Uh-

Dwarkesh Patel 13:11 Well, I mean, the broader point here being that... So we see this spectrum in humans where humans obviously have AGI. But even within humans, you see a spectrum where some people are relatively dumber and they'll do- perform work on IQ-like tests. For example, uh, Raven's Progressive Matrices. If you look at how the average person performs on that and you look at the k- kind of questions that ha- is a sort of hit or miss, half of the people will get it right, half of the people will get it wrong. Some of them are, like, pretty trivial. Uh, for us, we might think, like, "This is a little..." This is kind of trivial. And so humans have AGI, but f- for, from relatively small tweaks, you can go from somebody who misses these kinds of basic IQ test questions to somebody who gets them all right. Which suggests that, actually, if these models are doing natively... Um, we'll talk about some of the previous performances that people have tried with these models. But somebody with a- t- uh, a Jack Cole with a 240 million parameter model got 35%. Doesn't that suggest that they're on the s- on the spectrum that clearly exists within humans and they're gonna go saturated at pretty soon?

Francois Chollet 14:08 Yeah, so that's, uh, that's a bunch of interesting points. Yeah. So, uh, there is indeed a, um, a branch of LLM approaches suspended by Jack Cole that are doing quite well, that are in fact, uh, uh, state of the art. Uh, but you have to look at, uh, what's going on there. So there are two things. The first thing is that, uh, to get these numbers, you need to pre-train your LLM on millions of generated ARC tasks. And of course, if you- if you compare that to a five-year-old child looking at ARC for the first time, uh, the child has never done an IQ test before, has never seen something like- like an ARC- ARC test before. The only overlap between what they know and what they have to do in the test is core knowledge, is knowing about, like, counting and objects and symmetries and things like that. And still, uh, they're gonna do really well and they're gonna do much better than the LLM trained on millions of similar tasks. And the second thing that's, uh, that's, um, something to note about, uh, the Jack Cole approach is, um, one thing that's really critical to making the model work at all is test-time fine-tuning. And that's something that's really missing, by the way, from LLM approaches, um, right now is that, you know, most of the time when you're- when you're using an LLM, it's just doing static inference. The model is frozen and you're just, uh, prompting it and then you're get- you're getting an answer. So the model is not actually learning anything on the fly. Its- its state is not adapting, uh, to do, to the task at hand. And what Jack Cole is actually doing is that for every test problem is on the fly, is fine-tuning a version of the LLM, uh, uh, for that task. And that's really what's unlocking performance. If you don't do that, you get like 1%, 2%. So basically, something completely- completely negligible. And if you do test-time fine-tuning and you add a bunch of tricks on top, then you end up with interesting performance numbers. So I think what he's doing is trying to address one of the key limitations of LLMs today, uh, which is the lack of active inference, is actually adding active inference to LLMs and that's working ex- extremely well actually. So that's fascinating to me.

Dwarkesh Patel 16:14 That- there's so many interesting rabbit holes there. Should I take them in sequence or deal with them all at once? Let me, let me just start. So a c- uh, the point you made about the fact that you need to unlock the adapter compute/test-time compute, a lot of the scale maximalists... I think this will be an interesting rabbit hole to explore with you, because a lot of the scaling maximalists have your broader perspective in the sense that they think that in addition to scaling, you need these kinds of things like unlocking adaptive compute or doing some sort of RL to get the system to working. And their perspective is that this is a relatively straightforward thing that will be added atop the capab- the representations that a scaled-up model has greater access to.

Francois Chollet 16:57 No, it's- it's not, it's not just a technical detail. It's not a straightforward thing. It is everything. It is the important part. And the- the- the scale, uh, maximalist argument, even it boils down to, um... You know, th- these people, they- they refer to scaling laws, which is this- this empirical relationship that you can draw between how much compute you spend on training a model and the performance you're getting on benchmarks, right? And the- the key question here, of course, is, well, how do you measure performance? What it is that you're actually, uh, uh, improving by adding more compute and more data. And well, it's- it's benchmark performance, right? And th- the- the thing is, the way you measure performance is not a technical detail. Uh, it- it's- it's not, it's not an afterthought because it's gonna, uh, narrow down the sort of questions that you're asking. And so, uh, accordingly, it's gonna narrow down the set of answers that- that you're- that you're looking for. If you look at, uh, the benchmarks we're using for LLMs, they are all memorization-based benchmarks. Like, sometimes they're literally just knowledge-based, like- like a school test. And even if you look at the ones that are, uh, uh-... you know, uh, explicitly about reasoning. You realize if you look closely that it's, uh, in order to solve them, it's enough to memorize, uh, a finite set of, uh, uh, reasoning patterns, uh, and then you just reapply them. They're, they're like static programs. LLMs are very good at memorizing static programs, small statics programs. And, and they've got this sort of like bank of, uh, solution programs. And when you give them a new puzzle, uh, they can just fetch, uh, the appropriate program, uh, apply it, and it's looking like it's reasoning, but really it's not doing any sort of on-the-fly program synthesis. All it's doing is program fetching. So you can actually solve all these benchmarks with memorization. And so what, what you're scaling up here, like if you look at the models, they are, uh, uh, big biometric curves, uh, fitted to a data distribution, which I call an incandescent. So there are basically these big interpretive, uh, databases, interpolative memories. And of course, if you scale up the size of your database and you cram into it, uh, more knowledge, more patents and so on, uh, you are gonna be increasing its, its performance as measured by memorization benchmark. That's, that's kind of obvious. But as you're doing it, you are not increasing the intelligence of the system one bit. You are increasing the skill of the system. You are, you are increasing its usefulness, its, uh, scope of applicability, but not its intelligence, because skill is not intelligence. And that's the fundamental confusion, um, that, that, that people, uh, uh, run into is that they're confusing skill

Francois Chollet 19:43 and intelligence.

Dwarkesh Patel 19:44 Th- th- yeah, there, there's a lot of fascinating things to talk about here. So skill, intelligence, interpolation, um, I mean, okay, so the, the thing about they're fitting some manifold into, uh, that maps the input data, there's a reductionist way to talk about what happens in the human brain that says that it's just axons firing at each other, but we, but we, we don't care about the reductionist explanation of what's happening. We care about what the, the, the sort of m- meta at the, uh, at the macroscopic level, what happens when these things, uh, combine. As far as the interpolation goes, so okay, let, let's look at one of the benchmarks here. There's th- there's one benchmark that does great school math, and these are problems that m- m- like a smart high schooler would be able to solve. Um, it's called GSM8K. And these models get 95% on these, like basically-

Francois Chollet 20:32 Sure.

Dwarkesh Patel 20:32 ... they always nail it.

Francois Chollet 20:33 That's memorization benchmark.

Dwarkesh Patel 20:34 Okay. Let, let's talk about what that means. So here's one question about from that benchmark. So 30 students are in a class, one-fifth of them are 12-year-olds, one-third are 13-year-old, one-tenth are 11-year-olds. How many of them are not 11, 12, or 13 years old? So I agree, like this is not rocket science, right? You can write down on paper how you go through this problem, and a high school kid, at least a smart high school kid, should be able to solve it.

Francois Chollet 20:56 (smacks lips)

Dwarkesh Patel 20:56 Now, when you say memorization, it still has to reason through how to think about fractions and what is the context of the whole problem, and then combining the different calculations it's doing.

Francois Chollet 21:07 It depends how you, how you wanna define reasoning, but there, there are two definitions you can use. So one is, I have available, uh, a set of program templates. It's, it's like the structure of the puzzle, uh, which, which can also generate its solution. And I'm just gonna identify the right template, which is in my memory. Um, I'm gonna input the new values into the template, run the program, get the solution. And you could say, this is reasoning. And I say, "Yeah, sure. Okay." Uh, but another definition we can use is re- reasoning is the ability to when you're faced with a, with a puzzle, given that you don't have already a program in memory to solve it, you must synthesize on the fly a new program based on, uh, bits and pieces of existing programs that you have, you have to do on-the-fly program synthesis. And this is actually dramatically harder than just fetching the right memorized program and reapplying it.

Dwarkesh Patel 22:01 So I think maybe we are overestimating the extent to which humans are so sample efficient, they also don't need training in this way where they have to drill in these kinds of pathways of reasoning through c- certain kind of problems. So let's take math, for example.

Francois Chollet 22:18 Yeah.

Dwarkesh Patel 22:19 It's not like you can just show a baby the axioms of set theory, and now they know math, right? So they, they, when they're growing up, you had to do years of teaching them pre-algebra, then you gotta do a year of teaching them, uh, doing drills and going through the same kind of problem in algebra, then geometry, pre-calculus, calculus.

Francois Chollet 22:34 Uh, absolutely. So training-

Dwarkesh Patel 22:36 And, but, yeah. But isn't that like the same kind of thing where you, you, you can't, you can't just see one example and now you have the program or whatever? You actually had to drill it. These models also had to drill with a bunch of free training data.

Francois Chollet 22:45 Sure. I mean, in order to do on-the-fly program synthesis, you actually need, uh, building blocks to work from. So knowledge and memory are actually tremendously important in the process. I'm not say- I'm not saying it's memory versus reasoning. In order to do effective reasoning, you need memory.

Dwarkesh Patel 23:04 But it sounds like it's compatible with your story that through seeing a lot of different kinds of examples, these things can learn to reason within the context of those examples. And, uh, we can also see within bigger and bigger models. So that was an example of a high school level math problem. Um, uh, let's say a model that's like smaller than GPT-3 couldn't do that at all. As these models get bigger, they can, they seem to be able to pick up bigger and bigger-

Francois Chollet 23:26 It's, it's not, it's not really a size issue, it's more like a training data issue in this case.

Dwarkesh Patel 23:30 Well, uh, uh, y- yeah, bigger models can pick up these kinds of circuits, which smaller models apparently, uh, uh, don't do a good job of doing this, even if you were to train them on this kind of data. Doesn't that just suggest that as you have bigger and bigger models, they can pick up bigger and bigger pathways or, uh, uh, more general ways of reasoning?

Francois Chollet 23:44 Absolutely.

Dwarkesh Patel 23:45 But then isn't that intelligence?

Francois Chollet 23:46 No. No, it's not. If, if you scale up your database and you keep adding to it more knowledge, uh, more program templates, then sure, it becomes more and more skillful, you can apply to more and more tasks. But general intelligence is not task-specific skill scaled up to many skills. It- because there is an infinite, uh, space of possible skills. General intelligence is the ability to approach any problem, any skill, and very quickly master it using valuable data, because this is what makes you able to face anything you might have encountered. This is what makes, uh, uh...- this, this is the definition of generality. Like, generality is not specificity scaled up, it is, uh, the ability to apply your mind to anything at all, to arbitrary things. And this requires, fundamentally this requires the ability to adapt, to learn on-the-fly, efficiently.

Dwarkesh Patel 24:37 So I, my claim i- is that by doing this pre-training on bigger and bigger models, you are gaining the capacity to then generalize very efficiently. Let me give you an example-

Francois Chollet 24:47 And I think in practice-

Dwarkesh Patel 24:48 Let me, let me give you an example. So, your own company, Google, in their p-paper on Gemini 1.5, they had this very interesting example where the, um, they would give, in context, they would give the model the grammar book and the dictionary of a language that has less than 200 living speakers. So it's not in the pre-training data. And you just give them the, the dictionary, and it basically is able to speak this language and translate to it, including the complex and organic ways in which languages- languages are structured. So, a human, if you showed me a dictionary from, like, English to Spanish, I'm not gonna be able to pick up the how to structure sentences and how to say things in Spanish. Eh, the fact that because of the representations that it has gained through this pre-training, it is able to now extremely efficiently learn a new language. Doesn't that show that, uh, this kind of pre-training actually does increase your abil- ability to learn new tasks?

Francois Chollet 25:41 If you're right, if you were right, LLMs would do really well on ARC puzzles because ARC puzzles are not complex. Each one of them requires very little knowledge, each one of them is, is very low on complexity. You don't need to think that hard about it. Uh, they're actually extremely obvious for humans, like, even children can do them. But LLMs cannot. Uh, even if, uh, even LLMs that have, you know, 100,000 times more knowledge than you do, they still cannot. And the only thing, uh, that makes ARC special is that it was designed with this intent to resist memorization. This is the only thing, and this is the huge blocker, uh, for LLM performance. Right. And so, uh, you know, I think if, if you look at LLMs closely, it's pretty obvious that they're not really, like, synthesizing new programs on-the-fly t- to solve the tasks that, that they're faced with. They're very much replying things that they've, uh, they've stored in memory. For instance, um, one thing that's very striking is LLMs can solve a Caesar cipher. You know, like a-

Dwarkesh Patel 26:43 Mm-hmm.

Francois Chollet 26:43 ... a Caesar cipher, like transposing, uh, letters to, to, to code, uh, a message. Um, and while that's a fairly complex algorithm, right? Uh, but it comes up quite a bit on the internet. So they've b- basically memorized it, and what's really interesting is that they can do it, uh, for a transposition length of, like, three or five because they are very, very common numbers in examples provided on the internet. But if you, if you try to do it with an arbitrary number, like nine, it's gonna fail because it does not encode the generalized form of the algorithm, but only specific cases. It has memorized specific cases with the algorithm, right? And if, if it could actually synthesize on-the-fly the solver algorithm, uh, then the value of N would not matter at all because it does not increase the problem complexity.

Dwarkesh Patel 27:30 I, I think this is true of humans as well, where, uh, ev- what was the study that-

Francois Chollet 27:34 Humans use memorization and pattern matching all the time, of course, but humans are not limited to memorization and pattern matching. They have this very unique ability to adapt to new situations on-the-fly. This is exactly what, uh, enables you to navigate, uh, uh, every-

Dwarkesh Patel 27:49 So-

Francois Chollet 27:49 ... new day in your life.

Dwarkesh Patel 27:50 I, I'm forgetting the details, but there was a, some study that chess grandmasters will perform very well within the context of the moves that-

Francois Chollet 27:56 Excellent example because chess, at the highest level, is all about memorization. Chess is a memorization game.

Dwarkesh Patel 28:01 Okay, sure. We can leave that aside. What, what is your explanation for the original question of, um, why can, uh, why in context the GPT-1... or sorry, Gemini 1.5 was able to learn a language, including the complex gr- eh, uh, grammar structure? Doesn't that show that they can pick up new knowledge?

Francois Chollet 28:17 I would assume that it has simply, uh, mined from its extremely extensive, unimaginably vast, uh, training data. It has mined, uh, the, the required template and then it's just reusing it. We know that LLMs have, uh, very poor ability to synthesize new program templates like this on-the-fly, or even adapt existing ones. They're very much limited to fetching.

Dwarkesh Patel 28:38 Suppose there's a programmer at Google. They g- go into the office in the morning. At what point are they doing something that 100% cannot be due to fetching some, the template that could, could... e- even if they, suppose they were an LLM, they could not do it if they had fetched some template from the program. Like, at w- at what point do they have to use this so-called extreme generalization capability?

Francois Chollet 28:55 Forget about Google software developers. Every human, every day of their lives is full of novel things that they've not been prepared for. You cannot navigate your life based on memorization alone. It's impossible.

Dwarkesh Patel 29:08 I, I'm, I'm sort of d- denying the premise that they're all, uh, uh, you are also agreed they're not doing, like, quote-unquote memorization. They, you're, eh, it seems like you're saying they're l- less capable of generalization, but I'm just curious of, like, the kind of generalization they do. You, if you, if you, you g- get into the office and you try to do this kind of generalization, you're gonna fail at your job. What is the first point... You're a programmer, what is the first point when you try to do that generalization and you would, you would, you would lose your job because you can't do the extreme generalization?

Francois Chollet 29:34 I, I don't have any specific examples, but l- literally, like, take this situation, for instance. You've never been here in this room. Uh, maybe you've been in, in, in this, in this city a few times, I don't know. But, eh, there's a fair amount of novelty. You've never been, been, you know, uh, uh, interviewing me. There's a fair amount of novelty in every hour of every day in your life. And it's, in fact, uh, um, b- by and large, more novelty than, uh, any, any LLM could handle. Like, if you just put a LLM in, in a robot, it could not be doing all the things that you've been doing today. Right. Um, or take, take, I don't know, like self-driving cars, for instance. You take, uh, a self-driving car operating in the, in the Bay Area. Do you think you could just drop it in New York City or drop it in London where people drive on the left? Uh, no, it's, it's gonna fail. So not, not only can you drop, not, like, make it generalize to, uh, uh, a change of rules, um-... uh, uh, of driving rules, but you cannot even make it generalize to a new city. It needs to be trained on each specific environment.

Dwarkesh Patel 30:38 Yeah. I mean, I, I agree that self-driving cars aren't AGI. (laughs) Um, but it -

Francois Chollet 30:41 But it's the same type of model. They are transformers as well.

Dwarkesh Patel 30:44 I mean, we... w- w- ... I don't know. It's... it's also-

Francois Chollet 30:46 It's the same architecture.

Dwarkesh Patel 30:46 ... has brains with neurons in them, but they're less intelligent because they're smaller.

Francois Chollet 30:49 Based on the same architecture.

Dwarkesh Patel 30:50 Uh, we, we can get into that. But so I still don't understand wha- uh, uh, like, a concrete thing of... we also need training, that's why education exists, that's why we have to spend the first 18 years of our life doing drills.

Francois Chollet 31:03 We have a memory, but we are not a memory. We are not limited to just a memory.

Dwarkesh Patel 31:07 But I, I'm denying the premise that that's, uh, necessarily the only thing these models are doing. And I'm still not sure what is the... what is the task that a, a remote worker would be doing... uh, have to... like, suppose you just started out a remote work with an LLM and they're, they're a programmer. What is the first point at which you realize this is not a human, this is an LLM?

Francois Chollet 31:23 What about I just send them, uh, an ARC puzzle and see how they do? (laughs)

Dwarkesh Patel 31:27 No, no. Like part of their job, you know.

Francois Chollet 31:29 But you, you have to deal with novelty, uh, all the time.

Dwarkesh Patel 31:32 Okay. So if you... is there a world in which all the programmers are replaced and then we're still saying, "Ah, but they're only doing memorization-laden programming tasks," but they're still producing a trillion dollars of, uh, worth of, uh, you know, output in the form of code.

Francois Chollet 31:46 So, so software development is actually a pretty good example of a job where you're dealing with novelty all the time. Uh, or if you're not, well, I'm not sure what you're doing. So I, I personally use, uh, generative AI very little in my software development job. And before, before LLMs were a thing, I was also using Stack Overflow very little. You know, some people maybe are just copy-pasting stuff from Stack Overflow, or nowadays, copy-pasting stuff from, from an LLM. Um, personally, I try to focus on problem-solving. The syntax is just a technical detail.

Dwarkesh Patel 32:16 Mm-hmm.

Francois Chollet 32:16 What's really important is the problem-solving. Like the essence of programming is, uh, engineering, uh, mental models like mental representations of the problem you're trying to solve.

Dwarkesh Patel 32:29 But, uh, uh, you can a-... uh, you know, we, we have many... people can interact with these systems themselves and you can go to ChatGPT and say, "Here's a specification of the kind of program I want," they'll build it for you.

Francois Chollet 32:39 Yeah, as long as there are many examples of this program on like GitHub and Stack Overflow and so on, sure, they will fetch the program for you from their memory.

Dwarkesh Patel 32:47 But you can change arbitrary details.

Francois Chollet 32:49 No, it doesn't work.

Dwarkesh Patel 32:49 You can say, "I need it to work on this different kind of server." It, uh... "I, I need it to-"

Francois Chollet 32:53 If, if, if that were true, there would be no software engineers today.

Dwarkesh Patel 32:56 I, I, I agree we're not at a full AGI yet in the sense that these models have, let's say, less than a trillion parameters. A human brain has somewhere on the order of 10 to 30 trillion synapses. It... I mean if you're just doing some naive math, you're like at least 10X under-parameterized. So I agree we're not there yet, but it... I'm, I'm sort of confused on why we're not on the spectrum where, yes, I agree that there's m- many kinds of generalization they can do, but it seems like they're on this kind of smooth spectrum that we see even within humans where some humans would have a hard time doing an ARC-type test. We see that based on their performance on Progressive Raven's Matrices type IQ tests.

Francois Chollet 33:31 I'm, I'm not a fan of IQ tests because, uh, for the most part you can train, uh, on IQ tests and get better at them. So they're very much memorization-based. And this is actually what... the main pitfall that ARC tries not to fall for.

Dwarkesh Patel 33:45 I'm, I'm still not compu-... so if, um, if all remote jobs are automated in the next five years let's say, um, th- at least they don't require you to be like sort of a service... yo- it's not like a salesperson where you n-... you want the human to be talking, but like programming or whatever. In that world, um, w- would you say that that's not possible because a lot of what a programmer needs to do definitely requires things that would not be in any free training corpus?

Francois Chollet 34:08 Sure. I mean, in five years there will be more software engineers than there are today, not, not fewer.

Dwarkesh Patel 34:12 Right. But, but I, I just want to understand. So, um, still not sure. I, I mean I know how to co-... I, I studied computer science. Like if I had become a code monkey out of college, like what would I be doing? (laughs)

Francois Chollet 34:21 (laughs)

Dwarkesh Patel 34:22 I go to my job. What, what is the first thing... m- my boss tells me something to do. When does he realize I'm an LLM? Uh, if I was an LLM.

Francois Chollet 34:30 Probably on the first day, you know? (laughs) Again, uh, i- i- if it were true that, uh, uh, LLMs could generalize to novel problems like this and, and actually, uh, develop software to solve problems they've never seen before, you would not need software engineers anymore. In practice, if I look at how people are using LLMs in their software engineering job today, they're using it, uh, as a, as a Stack Overflow replacement. So they're using it as, uh, a way to copy-paste code snippets to perform very common actions. And this is... what, what they actually need is a database of code snippets. They don't actually need, uh, any of the abilities that actually make them software engineers.

Dwarkesh Patel 35:10 I mean, w- when we talk about th- um, interpolating between Stack Overflow databases, if you look at the kinds of math problems or coding problems, maybe to say that they're, um... maybe let's step back on interpolation and that... uh, let me ask the question this way. Why can't creativity... why isn't creativity just interpolation in a higher dimension where if a bigger model can learn a more complex manifold, if we're gonna use the ML language, and if you look at... read a biography of a scientist, right? It doesn't feel... uh, like they're not zero-shoting new scientific theories. They're playing with existing ideas. They're trying to juxtapose them in their head. They try out some, uh, uh, slight... like slightly ev- uh, uh, in the tree of evolut- um, intellectual descendants, they try out a different evolutionary path. We... y- you, you sort of run the experiment there in terms of publishing the paper, whatever. It seems like a similar kind of thing humans are doing. There's like... at a higher level of generalization. And what you see across bigger and bigger models is they can... they seem to be approaching higher and higher levels of generalization. Where GPT-2 couldn't do a great school level math problem that requires more generalization that it has cap- capability for, even that skill, then GPT-4... uh, 3 and 4 can.

Francois Chollet 36:15 So not quite. So, uh, GPT-4 has a higher degree of skill and higher range of skills. It has the same-

Dwarkesh Patel 36:21 Uh, uh, I don't want to get into semantics here, but I, I-

Francois Chollet 36:23 ... same degree of generalization.

Dwarkesh Patel 36:24 I, I don't want to get into semantics here, but, uh, the question of why, why can't, uh, creativity be, um, just, uh, interpolation on a higher, uh, higher dimension?

Francois Chollet 36:32 I think interpolation can be creative. Absolutely. And, you know, uh, to your point, I do think that, uh, on some level humans also do a lot of memorization, a lot of reciting, a lot of pattern matching and interpolation as well. So it, it's, it's very much a spectrum. Um-... between, between pattern matching and true reasoning, it's a spectrum and humans are never, uh, uh, really at one hand, end of the spectrum. They're never really do- doing pure pattern matching or pure reasoning. They're usually doing some mixture of both. Even if you're, uh, doing something that seems very reasoning heavy, like proving a mathematical theorem, uh, as you're doing it, sure, you're doing quite a bit of, you know discrete search in your mind, quite a bit of, uh, uh, actual reasoning, but you're also, uh, very much guided by intuition, guided by pattern matching, guided by the shape of proofs that you've seen before, by, by your knowledge of mathematics. So it's, it's never really, you know, all of our thoughts, everything we do is a mixture of this sort of like interpretive memorization-based thinking, this sort of like type one thinking and, uh, type two thinking.

Dwarkesh Patel 37:39 Um, why are bigger models more sample efficient?

Francois Chollet 37:43 Because they have more reasonable, uh, uh, building blocks, uh, that, uh, they can lean on, uh, to pick up, uh, uh, new patterns in their training data.

Dwarkesh Patel 37:52 And does that pattern keep continuing as you keep getting bigger and bigger?

Francois Chollet 37:56 To the extent that the new patterns you're, you're giving the model to learn, uh, are good match for what it has learned before. If you present something that's actually novel, that is not in this data distribution, like an ARC puzzle, for instance, it will fail.

Dwarkesh Patel 38:09 Uh, let me make this claim. The program synthesis, I think, is a very, uh, very useful intuition pump. Why can't it be the case that what's happening in the transformer is the early layers are doing the figuring out how to represent the inputting tokens, and what the middle layers do is this kind of program search, program synthesis, where they combine the inputs to the, uh, to the, you know, the, all the circuits in the model, where they, they go from the low-level representation to a higher level of representation near the middle of the model, they use these programs and they do, they, they combine these concepts, then what comes out the other end is the reasoning based on that high level, uh, uh, intelligence?

Francois Chollet 38:45 Possibly. Why not? Um, but, you know, if, uh, if these models were actually capable of synthesizing, synthesizing novel pro- programs, however simple, they should be able to do ARC. Because for any ARC task, if you write down the solution program in Python, it's not a complex program, it's extremely simple, and, and, and humans can figure it out. So why, why can LLMs not do it?

Dwarkesh Patel 39:10 Okay. That, I think that's a, a fair, uh, fair point. And, uh, I mean, if I turn the question around to you, so suppose that it's the case that in a year, a multimodal model can solve ARC, l- let's say get 80%-

Francois Chollet 39:23 Sure.

Dwarkesh Patel 39:23 ... what, whatever the a- average human would get, then AGI?

Francois Chollet 39:27 Qu- quite possibly, yes.

Dwarkesh Patel 39:28 Hmm.

Francois Chollet 39:28 I think if you, if you start... So honestly, what I would like to see is, uh, an LLM type model solving ARC at like 80%, but after having only been trained on core knowledge-related stuff.

Dwarkesh Patel 39:42 But human kids, I don't think we're necessarily just traded non... It's not just that we have in our genes object permanence.

Francois Chollet 39:48 So what's... Okay, let, let me rephrase that.

Dwarkesh Patel 39:49 Yeah.

Francois Chollet 39:50 Uh, only trained on information that is not, um, explicitly trying to anticipate what's gonna be in the ARC test set.

Dwarkesh Patel 39:59 But isn't the wh- isn't the whole point of ARC that you can't sort of... it's a new chart of... type of intelligence test every single time?

Francois Chollet 40:04 Yes, that is the point. So if ARC were per- perfect, flawless benchmark, it would be impossible to anticipate what's in the test set. And, you know, ARC was released, uh, more than four years ago, and so far, it's been resistant to memorization. So I think it has, uh, to some extent, uh, uh, passed the test of time. Um, but I don't think it's perfect. I think if you try to make by hand, uh, hundreds of thousands of ARC tasks, and then you try to multiply them, uh, uh, by, uh, programmatically generating evaluations, and then you end up with maybe hundreds of millions of tasks, uh, just by brute forcing the task space, there will be enough overlap between what you train on and what's in the test set that you can actually score very highly. So, you know, with enough scale, you can always cheat.

Dwarkesh Patel 40:49 If you can do this for every single thing that supposedly requires intelligence, then what good is intelligence? Apparently, you can just brute force intelligence.

Francois Chollet 40:55 If, if the world, if your life were a static distribution, uh, then sure, you could just brute force the space of possible behaviors. You could... like, you know, the, the, the way I would think about intelligence, there, there are, there are several metaphors I like to use, but one of them is you can think of intelligence as a pathfinding algorithm in future situation space. Like, I don't know if you're familiar with game development, like RTS game development-

Dwarkesh Patel 41:20 Mm-hmm.

Francois Chollet 41:20 ... but you have a map, right? And, and you have, uh... it's like a 2D, 2D, 2D map, and, um, you have partial information about it. Like, there is, uh, some fog of war on your map. There are areas that you haven't explored yet, you know nothing about them, and then there are areas that you've explored, but, um, you only know how they were like in the past, you don't know how they are like today. And, um, thi... and, and now, instead of thinking about a 2D map, think about the space of possible future situations that you might encounter and how they're connected to each other. Intelligence is a pathfinding algorithm. So once you set a goal, it will tell you, uh, how to get there optimally. Um, but of course, it's, uh, it's constrained by the information you have. Uh, it, it cannot pathfind in an area that you know nothing about. It cannot also anticipate, uh, uh, changes. And, um, the, the, the, the thing is, if you had complete information about the map, uh, then you could solve the pathfinding problem by simply memorizing every possible path, every mapping from, uh, point A to point B. Uh, you could s- you could-

Dwarkesh Patel 42:29 But...

Francois Chollet 42:30 ... solve the problem with pure memory. But where, the reason you cannot do that in real life is because you don't actually know what's gonna happen in the future.

Dwarkesh Patel 42:38 I, I-

Francois Chollet 42:38 Life is ever-changing.

Dwarkesh Patel 42:40 I feel like you're using words like memorization, which we would never use for human children. If, if you're like, your kid learns to do algebra and then, like, now learns to do calculus, you wouldn't say they memorized calculus. If they can just solve any ar- ar- a- a arbitrary algebraic problem, you wouldn't say, like, they've memorized algebra. They'd say they've learned algebra.

Francois Chollet 42:55 Humans are never really doing pure memorization or pure reasoning.

Dwarkesh Patel 42:58 But that's only because you're semantically labeling when the human does a skill, it's a memorization. When the exact same skill is done by the LLM, as you can measure by these benchmarks and you can just, like, plug in any sort of math problem.

Francois Chollet 43:06 Sometimes humans are doing the exact same as the LLM is doing, which is just, for instance, I don't know, if you learn to add numbers, you're memorizing, uh, an algorithm, you're memorizing a program and then you, you can reapply it. You are not synthesizing on the fly, uh, the addition program.

Dwarkesh Patel 43:21 So, obviously at some point, some human had to fi- figure out how to do addition.

Francois Chollet 43:24 Yes.

Dwarkesh Patel 43:24 But, like, the way a kid learns it is not that they, uh, sort of figure out from the axioms of set theory how to do addition.

Francois Chollet 43:30 I think what you learn in school is mostly memorization.

Dwarkesh Patel 43:33 Right.

Francois Chollet 43:33 Yeah.

Dwarkesh Patel 43:33 So my claim is that, listen, uh, y- these models are vastly under-parameterized, relative to how many flops or how many parameters you have in the human brain. And so, yeah, they're, they're not gonna be, like, coming up with new theorems, like, we, we... the smartest humans can. But most humans can't do that either. Um, mo- what most humans do, it sounds like it's similar to what you're calling memorization, which is memorizing skills or memorizing, um, you know, uh, techniques that you've learned. And so it sounds like it's compatible in your... or let, so tell me if this is wrong. Is it compatible in your world if, like, all the remote workers are gone, but they're doing skills which we can potentially make synthetic data of? So we record everybody's screen and... every single remote worker's screen. We sort of understand the skills they're performing there. And now we've trained a model that can do all this. All the remote workers are unemployed. We're g- generating trillions of dollars of economic activity from AI, uh, remote workers. In that world, is, are we still in the memorization regime?

Francois Chollet 44:29 So sure, uh, with memorization, you can automate almost anything.

Dwarkesh Patel 44:33 Mm-hmm.

Francois Chollet 44:33 As long as it's, uh, it's a static distribution, as long as you don't have to deal with change.

Dwarkesh Patel 44:37 Are, are most jobs part of such a sta- static distribution?

Francois Chollet 44:41 Potentially there are a lot, there are lots of things that you can automate.

Dwarkesh Patel 44:44 Yeah.

Francois Chollet 44:44 And LLMs are an excellent tool for automation.

Dwarkesh Patel 44:46 Yeah.

Francois Chollet 44:46 And I think that's re- and, but you have to understand that automation is not the same as intelligence. I'm not saying that LLMs are useless. I've been a huge proponent of deep learning for many years.

Dwarkesh Patel 44:56 Mm-hmm.

Francois Chollet 44:56 Uh, and you know, for many years I've been saying two things. I've been saying that if you keep scaling up deep learning, it will keep paying off. And at the same time, I've been saying, if you keep scaling up deep learning, this will not lead to AGI.

Dwarkesh Patel 45:08 Yeah.

Francois Chollet 45:08 Uh, so we can automate more and more things. And yes, this is economically valuable and yes, potentially there are many jobs you could automate away like this and that would be economically valuable. Uh, but you're not, still not gonna have intelligence. So you can ask, you know, okay, so what does it matter if we can generate all this economic value? Maybe we don't need intelligence after all. Well, you need intelligence the moment you have to deal with change, with novelty, with uncertainty. As long as you're in a space that can be exactly described in advance, you can just, um, you can just automate a pure memorization, right? In fact, you can always solve any problem, you can always display, uh, arbitrary levels of skills, uh, uh, on, on any task, um, without leveraging any intelligence whatsoever, as long as it is possible to describe the problem and its solution very, very precisely.

Dwarkesh Patel 46:01 But wh- when they do deal with novelty, then you just call it interpolation, right? And so-

Francois Chollet 46:05 No, no, no.

Dwarkesh Patel 46:06 I, I

Francois Chollet 46:06 No, interpolation is not enough to deal with all kinds of novelty. If it were, um, then LLMs would be, would be AGI.

Dwarkesh Patel 46:13 Well, I agree they're not AGI. I'm just trying to figure out how do we figure out we're on the path to AGI. And I think, uh, sort of crux here, um, is maybe that it seems to me that these things are on a spectrum and we're clearly covering the earliest part of the spectrum with LLMs.

Francois Chollet 46:27 I think so.

Dwarkesh Patel 46:27 And... Oh, okay, interesting. Uh, but here's another, uh, sort of thing that I think is evidence for this. Grokking, right? So clearly even within deep learning, there's a difference between the memorization regime and the generalization regime where at first they'll just memorize the data set of, you know, if you're doing modular addition, how to add digits. And then at some point, if you keep training on that, they'll learn the skill. So the fact that there is that distinction suggests that there... the, the, the generalized circuit that deep learning can learn, uh, there is a regime it enters where it generalizes if you have an over-parameterized model, which we don't have in comparison to all the tasks we want these models to do right now.

Francois Chollet 47:04 Gro- grokking is very, very old phenomenon. We've been, we've been, uh, observing it for decades. It's- it's basically an instance of the, uh, minimum description lengths principle where sure you can... given a problem, you can just memorize, uh, an input, a pointwise input to output mapping, which is completely overfit. So it does not generalize at all, but it solves the problem on the train data. Uh, and, uh, from there you can actually keep pruning it, keep making, uh, your mapping simpler and simpler and more compressed. And at some point it will start generalizing.

Dwarkesh Patel 47:40 Mm-hmm.

Francois Chollet 47:40 And so that's something called the minimum description lengths principle. It's this idea that the program that will generalize best is the shortest, right? Um, and, uh, it doesn't mean that you're doing anything other than memorization, but you're doing memorization plus regularization, right?

Dwarkesh Patel 47:59 AKA generalization? (laughs)

Francois Chollet 48:00 Yeah.

Dwarkesh Patel 48:00 I mean, but then-

Francois Chollet 48:01 And, and that is... absolutely, that, that leads to, to generalization, yes.

Dwarkesh Patel 48:03 Right. And then so you do that within one skill, but then the pattern you see here of meta-learning is that it's more efficient to store a program that can perform many skills-

Francois Chollet 48:11 Yeah.

Dwarkesh Patel 48:11 ... rather than one skill, which is what we might call fluid intelligence. And so as you get bigger and bring models, you would expect it to go up this hierarchy of generalization where it generalizes to a skill, then it generalizes across multiple skills.

Francois Chollet 48:22 That's correct. That's correct. And you know, LLMs, they're not infinitely large. Uh, they have only, uh, uh, a fixed number of parameters and so they have to compress their knowledge as much as possible. And in practice, so LLMs are mostly storing, uh, reusable bits of programs like vector programs. And, uh, because they have this need for compression, it means that every time they're learning a new program, they're gonna try to express it in terms of existing bits and pieces of programs that they've already learned before, right?

Dwarkesh Patel 48:52 And isn't, isn't this the generalization?

Francois Chollet 48:54 Absolutely.

Dwarkesh Patel 48:55 Oh, I, I, wait, so-

Francois Chollet 48:57 This is why, you know-

Dwarkesh Patel 48:58 Wait, wait-

Francois Chollet 48:58 ... clearly LLMs have some degree of generalization-

Dwarkesh Patel 49:01 Yeah.

Francois Chollet 49:01 ... and this is precisely why, it's because they have to compress.

Dwarkesh Patel 49:03 And why, why is that intrinsically limited? Why can't you just go, it w- be, a- at some point it has to learn a higher level of generalization, a higher level, and then the, the highest

Dwarkesh Patel 49:11 level is the fluid intelligence?

Francois Chollet 49:11 It's intri- intrinsically limited because the substrate of your, of your model is a big biometric curve and all you can do with this is local generalization. Uh, if you want to go beyond this towards broader or even extreme generalization, you have to move to a different type of model and, uh, my paradigm of choice is discrete program search, program synthesis. So, and if you want to understand that, you can sort of like compare, compare it, contrast it with deep learning. So in deep learning, your model is a, a, a parametric, a differentiable parametric curve. In program synthesis, your model is a discrete graph of operators. So you've got, like, a set of logical operators, like, a domain-specific language, uh, you're picking, uh, instances of it, you're structuring that into a graph that's a program. And that's actually very similar to, like, a program you might, you might write in, in, in Python or C++ and so on. And, uh, in deep learning, your learning engine, because we are doing much learning here, like, we are trying to automatically learn these models, uh, in deep learning, your learning engine is, uh, gradient descent, right? Uh, and gradient descent is very compute-efficient because you have this very strong informative feedback signal about, uh, where the solution is, so you can get to the solution very quickly. But it is very data inefficient, uh, meaning that in order to make it work, you need a dense sampling of the operating space. You need a dense sampling of the data distribution and then you're, you're limited to only generalizing within that data distribution. And the reason why you have this limitation is because your model is a curve. And meanwhile, if you look at discrete program search, um, the learning engine is combinatorial research. You're just trying a bunch of programs until you find one that actually meets, uh, your spec. This process is extremely data efficient. You can learn a generalizable program from just one example, two examples, which is why it works so well on ARC, by the way. But, uh, the big limitation is that it's extremely, uh, compute inefficient because you're running into, uh, combinatorial explosion, of course. And so you can, you can sort of see, uh, here how deep learning and discrete program search, uh, they have very complime- uh, complementary strength and limitations as well. Like, every limitation of deep learning, uh, has a strength, a corresponding strength in, in, uh, in program synthesis and, and inversely. And I think the path forward is gonna be to merge the two.

Dwarkesh Patel 51:40 Mm.

Francois Chollet 51:40 To basically start doing... So, uh, another way you can think about it is, uh, so these, these parametric curves, uh, uh, trained with gradient descent, they're a great fit for everything that's, uh, system one type thinking, like pattern cognition, intuition, uh, memorization and so on. And discrete program search, uh, uh, is, is a great fit for a type two thinking, system two thinking. Uh, for instance, planning, reasoning, uh, uh, quickly figuring out a generalizable model that matches just one or two, uh, examples, like for an ARC puzzle, for instance. And, um, I think humans are never doing pure system one or pure system two, they're always mixing and matching both, and right now we have all the tools for system one, we have almost nothing for system two. The way forward is to create a, a hybrid system, and I think the, the form it's gonna take is it's gonna be, uh, mostly system two, so the, the outer structure is gonna be a discrete program search system, uh, but you're gonna fix the fundamental limitation of discrete program search, which is combinatorial explosion, you're gonna fix it with deep learning. You're gonna leverage deep learning to guide, uh, to provide intuition, uh, in program space to guide, uh, the, the program search. And I think that's very similar to, um, what you see, for instance, uh, when, when, when you're playing chess or when you're trying to prove, uh, a theorem is that, um, it's mostly, uh, uh, a, a reasoning thing, but you start out with some intuition about the shape of the solution. And that's very much something you can get, uh, via deep learning models, yeah. Deep learning models, they're very much like intuition machines, uh, they're pattern matching, uh, machines. So you, you, you start from this, uh, shape of the solution and then you're, you're gonna do, uh, uh, actual explicit discrete program search, uh, but you're not gonna do it, uh, via brute force. You're not, you're not gonna try things kind of like randomly. Um, um, you're actually gonna ask another deep learning model for suggestions, like, "Here's the best likely next step."

Dwarkesh Patel 53:49 Yeah.

Francois Chollet 53:49 "Here's where in the graph you should be going." And you can also use a, yet another deep learning model for feedback about, "Well, here's what I have so far. Is it looking good? Should I just backtrack and try something new?" So, um, I think discrete program search is gonna be the key, but you want to make it dramatically better, orders of magnitude more efficient, by leveraging deep learning. And by the way, another thing that you can use deep learning for is of course things like, um, common sense knowledge and, and knowledge in general. Um, and I think you, you're gonna end up with this sort of system where you have this, uh, on the fly, uh, synthesis engine that can adapt to new situations. But the way it adapts is that it's gonna, uh, fetch, um, from a, a bank of patterns, uh, modules that could be themselves, uh, curves that could be, uh, uh, differentiable modules, and some others that could be algorithmic in nature. It's gonna assemble them, uh, via this, uh, process that's, uh, intuition guided and it's gonna give you, for every new situation you might be faced with, it's gonna give you with, uh, uh, a generalizable model that was synthesized using very, very little data.

Dwarkesh Patel 54:58 Mm.

Francois Chollet 54:59 ... and that, that something like this would solve ARC.

Dwarkesh Patel 55:02 That's actually a really interesting, um, uh, a prompt because I think, uh, a w- an interesting crux here is when I talk to my friends who are extremely optimistic about LLMs and expect AGI within the next couple of years, they also, in some sense, agree that scaling is not all you need, but that the rest of the progress is undergirded and enabled by scaling. And, but still you need to add the system to the test and compute atop these models. And their perspective is that it's relatively straightforward to do that because you have this library representations that you built up from free training, but it's almost talking like, uh, you know, it just, like, skimming through textbooks. You need some more deliberate way in which it engages with the material it learns. In-context learning is- is extremely sample efficient, but to actually distill that into the weights, you need the model to, like, talk through the things it sees and then add it back to the weights. As far as the System 2.0 goes, they talk about adding some kind of RL setup so that it is encouraged to proceed on the reasoning traces that end up being correct. And they think this is, uh, relatively straightforward stuff that will be added within the next couple of years.

Francois Chollet 56:16 That's an empirical question.

Dwarkesh Patel 56:17 Yeah.

Francois Chollet 56:17 So I think we'll see.

Dwarkesh Patel 56:18 Uh, your intuition, I assume, is not that. I'm curious why.

Francois Chollet 56:21 My intuition is, in fact, this whole, like, System 2.0 architecture is the hard part, is the very hard and non-obvious part. Scaling up the interpretative memory is the easy part. All you need is, is, uh, like, it's literally just a, a big curve. All you need is more data. It's a representation of a dataset, an interpretative representation of a dataset. That's the easy part. The hard part is the architecture of intelligence. Memory and intelligence are separate components. We have the memory, we don't have the intelligence yet. And I agree with you that, well, having the memory is actually very useful, and, um, if you just had the intelligence but it was not hooked up to an extensive memory, it would not be that useful because it would not have enough material, uh, uh, to work from.

Dwarkesh Patel 57:04 Yeah. Uh, the- the alternative hypothesis here that former guest Trenton Brooking, uh, uh, a- advanced is that intelligence is just, um, hierarchically-associated memory where higher level patterns... When Sherlock Holmes goes into a crime scene, and he's extremely sample efficient, he can just, like, look at a few clues and figure out who was the murderer. And the way he's able to do that is he has learned higher r- uh, higher level sort of associations. It's memory in some fundamental sense, but... So, I mean, here- here's, uh, one way to ask the question. In the brain, uh, supposedly we do program synthesis, but it is just synapses connected to one a- each other, and so physically it's gotta be that you just query the right circuit, right?

Francois Chollet 57:45 You are, yeah, yeah, yeah.

Dwarkesh Patel 57:46 So...

Francois Chollet 57:46 You know, it's, it's a matter of degree.

Dwarkesh Patel 57:48 But if you can learn it, if, if, uh, you know, training in the environment that the hu- human ancestors were trained in means you learn that, those circuits, trai- uh, training on the same kinds of outputs that humans produce, which to replicate require these kinds of circuits, wouldn't that train the same kind of, um, whatever humans have?

Francois Chollet 58:03 You know, it's, it's a matter of degree. Um, if you have a system that has a memory and is only capable of, of doing local generalization from that, it's not gonna be very adaptable. Um, to be really general, you, you need the memory plus the ability to search, uh, to quite some depth, uh, to achieve, you know, broader even extreme generalization. Um, you know, uh, like, one, one of my favorite, uh, uh, psychologists, so Jean Piaget was the, the, the founder of, uh, developmental psychology. He had, uh, a very good quote about intelligence. He said, "Intelligence is what you use when you don't know what to do." And it's like, as, as a human, uh, living your life, I- in most situations you already know what to do because you've been in this situation before. You already have the answer, right? Um, and you're, you're only gonna need to use intelligence when you're faced with, uh, with novelty, with, with something you didn't expect, with something that you weren't prepared for, either by, uh, your own experience, your own life experience, or by your evolutionary history. Like, this day that you're living right now is different, uh, in some important ways from, uh, every day you've lived before, but it's also different from any day ever lived by any of your ancestors, and still you're capable of, of being functional, right?

Dwarkesh Patel 59:22 Right, I mean, uh...

Francois Chollet 59:22 How is it possible?

Dwarkesh Patel 59:23 I, I'm, I'm not denying that generalization is extremely imp- uh, and is the basis for i- i- intelligence. I, that, that's not the qu- crux. The crux is, like, how much is, of that is happening in the models. But, um, okay, l- let me ask a separate question. Uh, we mo- we might, uh, keep, uh, going in this circle here. There are differences in intelligence between humans. Uh, maybe the intelligence tests, because of, uh, reasons you mentioned are not measuring it well, but clearly there's differences in intelligence between different humans.

Francois Chollet 59:47 Sure.

Dwarkesh Patel 59:47 What is your explanation for what's going on there? 'Cause I think that's sort of compatible with this, my story that there's a spectrum of generality and that these models are climbing up, uh, to a human level, and even some humans haven't even climbed up to the Einstein level or the, uh, uh, the, uh, the Francois level, but... (laughs)

Francois Chollet 1:00:02 So that's a great question. You know, um, there is extensive evidence that intelligence, uh, difference in intelligence are mostly genetic in nature, right? Meaning that if you take someone who is not very intelligent, there is no amount of training, of, like, training data-

Dwarkesh Patel 1:00:17 Yeah.

Francois Chollet 1:00:17 ... you can expose that person to that would, uh, uh, make them become Einstein. And this kinda points to the fact that you really need, uh, a better architecture, you need a better algorithm, and more training data is not in fact what you need.

Dwarkesh Patel 1:00:34 I think I agree with that. I think what maybe a w- way I might phrase it is that the people who are smarter have, in ML language, better initializations. It just... Th- their, th- the neural wiring, if you just look at, it's u- it's more efficient. They have maybe greater density of, um, firing. And so some, some part of this story is scaling. There is some correlation between brain size and intelligence. And we also see within the context of, quote-unquote, "Scaling," that people talk about within the context of LLMs, architectural improvements where a model like Gemini 1.5 Flash is... Performs as well as GPT-4 did when GPT-4 was released a year ago, but is 57 times cheaper on output. So...The p- part of the scaling story is that the architectural improvements are, we're in, like, extremely low-hanging fruit, uh, territory when it comes to those.

Dwarkesh Patel 1:01:23 Okay, we're back now with the co-founder of Zapier, Mike Knuth. We had to restart a few times there. (laughs) And you're funding this prize and you're running this prize with Francois. And so, tell me about how this came together. How, how, what, what prompted you guys to launch this prize?

Mike Knoop 1:01:41 Yeah. Um, I guess I've been sort of, like, AI curious for 13 years. Uh, I've been... I co-founded Zapier, been running it for the last 13 years. And I think I first got introduced to your work in, during COVID. Um, I kind of went down the rabbit hole. (laughs) I had a lot of free time, um, and it was right after you'd, um, published your On Measures of Intelligence paper where you sort of introduced the concept of AGI, this, like, efficiency of skill acquisition is, like, the right definition in, in the ARC puzzles. But I don't think the first Kaggle contest was done yet. I think it was still running. And so I kind of... It was interesting, um, but I just parked the idea. Uh, and, uh, I had bigger fish to fry at Zapier. We were in the middle of this big turnaround of trying to get to our second product. Um, and then, uh, it was January 2022 when the Chain of Thought paper came out that really, like, awoken me to sort of the progress. I, I gave a whole presentation to the Zapier on, like, the GPT-3 paper even. So I, I sort of felt like I had priced in everything that LLMs could do, and that paper was really shocking to me in terms of, oh, there's these latent capabilities that LLMs have that, um, I didn't expect that they had. And so I actually gave up my, uh, exec team role at Zapier. I was running half the company at that point. I went back to be an individual contributor and just do, to go do AI research, uh, alongside Bryan, my co-founder. Um, and ultimately that led me to back towards ARC. Uh, I was looking into it again, and I had sort of expected to see, uh, this h- you know, saturation effect that, you know, MMLU has, that GMSK-8K has. And when I looked at the scores and the progress th- since the last four years, um, I was really, again, shocked to see actually we've made very little objective progress towards it. And, um, it felt very... It felt like a really, really important e- eval. And as I sort of spent the last year asking people, quizzing people about it in sort of my networking community, um, very people, few people n- even knew it existed. And, uh, that, that felt like, okay, if it's right that this is a really, really, like, globally, singularly unique, um, AGI eval, and it's different from every other eval that exists that are more, that more nar- you know, m- narrowly measures AI skill, um, like, more people should know about this thing. Uh, I had my own ideas on how to beat the ARC as well. So I'm, like, like I was working on nights and weekends on that, and I flew up to meet Francois earlier this year, um, to sort of quiz him, show him my ideas, and ultimately

Francois Chollet 1:03:52 Yeah.

Mike Knoop 1:03:52 ... I was like, "Well, you know, why don't you think more people know about ARC?" Um, I think you should actually answer that. I think it's a really interesting question. Like, why, why don't you think more people know about ARC?

Francois Chollet 1:04:01 Sure. You know, I think benchmarks that gain, uh, traction in the research community are benchmarks that are already fairly tractable. Because the dynamic that you see is that some research group is gonna make some initial breakthrough, and then this is gonna catch the attention of everyone else. And so you're going to get follow-up papers with, with people trying to, uh, beat the first team and so on. And for ARC, this has not really happened because ARC is actually very hard for existing AI techniques. Kind of ARC requires you to try new ideas. And that's very much the point, by the way. Like, the point is not that, uh, yeah, you should just be able to apply existing technology and, and solve ARC. The point is that existing technology, uh, has reached a plateau. And if you want to go beyond that, if you want to start being able to tackle problems that you haven't memorized, that you haven't seen before, uh, you need to try new ideas. And, um, ARC is not just meant to be, um, uh, this sort of like, uh, measure of how close we are to AGI. It's also meant to be a s- a source of inspiration. Like, I want, I want researchers to look at these puzzles and be like, "Hey, it's really strange that these puzzles are so simple and, uh, most humans can, can just do them, uh, very quickly. Why is it so hard for existing AI systems? Why, why is it so hard for LLMs?" And so on. And this is true for LLMs, but ARC was actually released, uh, before LLMs were really a thing. And, um, the only thing that made it special at the time was that it was designed to be resistant to memorization. And the fact that it has survived LLMs and gen AI in general so well, uh, kind of shows that, yes, it is actually resistant to memorization.

Mike Knoop 1:05:40 Yeah. Th- this is what nerds night me, um, because I went and took a bunch of the puzzles myself. I showed it to all my friends and family, too, and they're all like, "Oh, yeah, this is, like, super easy. Um, are you sure AI can't solve this?" Like, that, that's the reaction, and, and the same one for me as well. And the more you dig in, you're like, okay, yep, there's not just empirical evidence over the last four years that it's unbeaten, but there's theoretical, like, concepts behind why. Um, and I, I completely agree at this point that, like, new ideas basically are needed to beat ARC, and there's a lot of current trends in the world that are actually, I think, working against that, um, happening, basically. I think we're actually less likely to generate new ideas right now. Um, you know, I think one of the kind of trends is the closing up frontier research, right? The GPT-4 paper from OpenAI had no technical details shared. Uh, the Gemini paper had no technical details shared on, like, the longer context, uh, part of that work. And, and yet that open innovation, that open progress and sharing is what got us to transformers in the first place. That's what got us to L- LLMs in the first place. Um, so it's, it's kind of, it, uh, disappointing a little bit actually that, like, so much frontier work has gone closed. It's really making a bet that, like, these individual labs are gonna be the b- have the breakthrough and not the ecosystem, um, is gonna have the breakthrough. And I think sort of the internet open source has shown that that's, like, the most powerful innovation ecosystem that's ever existed, probably in the entire world.

Francois Chollet 1:06:52 I think that's, that's actually really sad that, uh, frontier research is no longer being published. Uh, if you look, if you look back, you know, four years ago, um, well, everything was just openly shared. Like, all the state-of-the-art results were, were published. And this is no longer the case. And it very much, you know, OpenAI single-handedly, uh, changed the game. And I think, um, OpenAI basically set back, uh, progress towards AGI by quite a few years, probably like five to 10 years, for two reasons. And one is that, while they co- they caused this, uh, uh, complete closing down, uh, of research, frontier research publishing, uh, but also, uh, they triggered this, uh, initial, uh-... burst of, uh, hype, uh, ar- around LLMs. And now LLMs have, uh, sucked the oxygen out of the room. Like, everything, everyone is just doing LLMs. Um, and I see LLMs as more of an off-ramp on the, on the path to AGI actually. Um, and all these new resources, uh, they're actually going to LLMs instead of everything else that could be, they could be going to. And, you know, if you look, uh, further into the past to, like, 2015, 2016, there were like a thousand times fewer people doing AI back then and yet I feel like the rate of progress was higher because, uh, people were exploring more directions. Um, the world felt, felt more open-ended, like, you could just go and try, like, ha- have a cool idea for lunch and try it and get some interesting results. So there was, there was this, this energy. And now everyone is very much doing some variation of the same thing. And the big labs also tried, uh, their hand on ARC, but because they got bad results, they didn't publish anything. Like, you know, people only publish, uh, positive results.

Dwarkesh Patel 1:08:38 Um, I wonder how much effort people have put into trying to prompt or scaffold, do some sort of maybe Devin-type approach into getting the frontier models, and the frontier models of today, not just a year ago because a lot of post-training has gone into making them better, so Claude 3 Opus or GPT-4o into g- getting good solutions on ARC. I, um, I, I hope that one of the things this episode does is get people to try out this open competition where they have to put in an open source model to compete. But also to, like, figure out if there... maybe the late capability is latent in Claude Opus and just sh- see if you can show that.

Dwarkesh Patel 1:09:20 I, I think that would be super interesting. So let's talk about the prize. How much do you win if you solve it? Uh, you know, get whatever percent on ARC-

Francois Chollet 1:09:27 Yeah.

Dwarkesh Patel 1:09:28 How much do you get if you get the best submission but don't crack it?

Mike Knoop 1:09:30 So we got a million dollar pl- actually a little over a million dollars in the prize pool. We're running the contest, uh, on an annual basis. We're gonna... we're starting it today, um, through the, uh, middle of November. And the goal is to get 85%. That's the lower bound of the human average that you guys talked about earlier. And, uh, there's a $500,000 prize for the first team that can get to the 85% benchmark. We're also gonna run... we don't expect that to happen this year actually. Um, one of the early, um, s- uh, statisticians at Zapier gave me this line that has always stuck with me, uh, that the, the longer it takes, the longer it takes. So my prior is that, like, ARC is gonna take years to solve. Um, and so we're gonna keep do-... we're also gonna break down and do a progress prize this year. So we're g-... there's $100,000 progress prize which we will pay out to, uh, the top scorers. So $50,000 is gonna go to the top objective scorers this year on the Kaggle leaderboard, which is... we're hosting it on Kaggle. And then we're gonna have a $50,000 pot set for, uh, a paper award, um, for the best paper that explains conceptually the, the scores that they were able to achieve. And one of the, I think, interesting things we're also gonna be doing is, um, we're gonna be requiring that in order to win the prize money that you put the solution or your paper out into public domain. Um, the reason for this is, you know, t- tend to... typically with contests you see a lot of, like, closed up sharing. People are kind of private, secretive, they want to hold their alpha to themself (laughs) during the contest period. And because we expect it's gonna be multiple years, um, we, we want an early game here. So the, the plan is, you know, at the end of November we will award the $100,000 prize money to the top progress prize and then use the downtime between December, January, February to, uh, share out all the knowledge, um, from the top scorers and the approaches folks were taking in order to re-baseline the community up to whatever the state of the art is, and then run the contest again next year, and keep doing that on a yearly basis until we get 85%.

Dwarkesh Patel 1:11:16 I, I'll give some people some context on why I think this prize is, uh, very interesting. I was having conversations with my friends who are very much believers in models as they exist today and first of all, it was intriguing to me that they didn't know about ARC. These are experienced ML researchers and so you show them the... this is, this happened a couple of nights ago. We went to dinner and I showed them an example problem and they said, "Of course an, uh, LLM would be able to solve something like this." And then we take a screenshot of it, we just put it into our ChatGPT app and it doesn't get the pattern. And so I think it's very interest-... like, i- it is a notable fact. I was sort of playing devil's advocate against you on these kinds of questions, but this is a very intriguing fact and I'm extrem-... I think this prize is extremely interesting because we're gonna learn, we're gonna learn something fascinating one way or another.

Francois Chollet 1:12:00 Yeah.

Dwarkesh Patel 1:12:00 Um, so with regards to the 85%, separate from this prize, I'd be very curious if somebody could replicate that result because obviously in psychology and other kinds of fields which this result seems to be analogous to when you run test on some small sample of people, often they're hard to replicate. So I'd be very curious if you try to replicate this how... wh- what does an average human perform on ARC? Um, as for the difficulty and how long it will take to crack this benchmark, it's very interesting because the other benchmarks that are now fully saturated, like MMLU-Math, actually the people who made them, Dan Hendricks and Colin Burns who did MMLU and Math, I, I think they were grad students or college students when th- they made it. And the goal when they made it just a couple of years ago was that this will be a test of AGI, and of course it got totally saturated. And I, I know you argue that th- these are a test in memorization, but I think the pattern we've seen... in fact, Epoch AI has a very interesting graph that I'll sort of overlay for the YouTube version here, where you see this almost exponential where it gets, you know, 5%, 10%, 30%, 40% as you increase the compute across models and then it just shoots up. And in the GPT-4 technical report, they had this interesting graph of the human eval problem set, which was, uh, 22 coding problems. And they had to graph it on the mean log path curve basically because early on in training or if even smaller models can have the right idea-... of how to solve this problem, but it takes a lot of reliability to make sure they stay on track to solve the whole problem. And so you really wanna upweight the signal where they get it right at least some of the time, maybe one in a hundred times or one in a thousand, and then so they go from, like, one in 1,000, one in 100, one in 10, and then they just, like, totally saturate it. I guess the question I have what th- this is all leading up to is why won't the same thing happen with ARC where people had to try really hard, big, b- bigger models, um, and then now they've figured out these techniques that Jack Cole has figured out with only a 240 million parameter, uh, language model that can get 35%. Shouldn't we see the same pattern we saw across all these other benchmarks where you just, like, sort of eke out and then once you get the general idea then you just go all the way to 100?

Francois Chollet 1:14:09 That's an empirical question, so we'll see in practice what happens. Um, but what, uh, what Jack Cole is doing is actually very unique. It's not just pre-training an LLM and- and- and then prompting it. He's actually trying to do active inference. And then-

Mike Knoop 1:14:22 He's doing it at test time, right?

Francois Chollet 1:14:23 Yeah.

Mike Knoop 1:14:23 He's doing, like, test-time fine-tuning.

Francois Chollet 1:14:24 Like test-time fine-tuning.

Mike Knoop 1:14:25 Yeah.

Francois Chollet 1:14:25 And this is actually trying to lift one of the key limitations of LLMs, which is that at inference time they cannot learn anything new. They cannot adapt on the fly to what they are seeing. And he's actually trying to, uh, uh, learn. So what he's doing is effectively a form of program synthesis, um, because the LLM contains a lot of useful building blocks, like programming building blocks, and by fine-tuning it on the task at test time, you are trying to assemble these building blocks into the right pattern that matches the... that matches the task. This is exactly what, uh, what program synthesis is about. And the way we'd contrast this approach, uh, with, uh, discrete program search is