Why Can't AI Make Its Own Discoveries? — With Yann LeCun

1. 0:00 – 1:12

   Introduction to Jan LeCun and AI's limitations

   1. AKAlex Kantrowitz0:00

      Why has generative AI ingested all the world's knowledge, but not been able to come up with scientific discoveries of its own? And is it finally starting to understand the physical world? We'll discuss it with Meta chief AI scientist and Turing Award winner, Yann LeCun. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. I'm Alex Kantrowitz, and I am thrilled to welcome Yann LeCun, the chief AI scientist, Turing Award winner, and a man known as the godfather of AI to Big Technology Podcast. Yann, great to see you again. Welcome to the show.

   2. YLYann LeCun0:36

      Pleasure to be here.

   3. AKAlex Kantrowitz0:37

      Let's start with a question about scientific discovery and why AI has not been able to come up with it until this point. This is coming from Dwarkesh Patel. He asked it a couple months ago, "What do you make of the fact that AIs, generative AI, basically have the entire corpus of human knowledge memorized, and they haven't been able to make a single new connection that has led to discovery? Whereas if even a moderately intelligent person had this much stuff memorized, uh, they would notice, oh, this thing causes this symptom, this other thing causes this symptom, there might be a medical cure here." So shouldn't we be expecting that type o- that type of stuff

2. 1:12 – 5:40

   Why LLMs can't make scientific discoveries

   1. AKAlex Kantrowitz1:12

      from AI?

   2. YLYann LeCun1:13

      Well, from AI, yes. From large language models, no. You know, there's several types of AI architectures, right? And, and all of a sudden, when we talk about AI, we imagine chatbots. Chatbots, LLMs, are trained on an enormous amount of, of knowledge, which is purely text, and they're trained to basically regurgitate, to retrieve, to, uh, essentially produce answers that are conformed to the statistics of whatever text they've been trained on. And it's amazing what you can do with them. It's very useful, there's no question about it. Uh, we also know that they can elucidate facts that ar- aren't true, uh, but they're really, in their purest form, they are incapable of inventing new things.

   3. AKAlex Kantrowitz1:55

      Let me throw out this perspective that, uh, Tom Wolf from Hugging Face shared on LinkedIn over the past week. I know you were involved in the discussion about it. It's very interesting. He says, "To create an Einstein in a data center, we don't just need a system that knows all the answers, but rather one that can ask questions nobody else has thought or dared to ask. One that writes, 'What if everythi- everyone is wrong about this when all texperts, textbooks, experts, and common knowledge suggest otherwise?'" Is it possible to teach LLM to do that?

   4. YLYann LeCun2:29

      No. No, not, not in their current form. Uh, I mean, and, and whatever form of AI would be able to do that will not be LLMs. [chuckles] They might use LLM as one component. LLMs are useful to turn, uh, you know, to produce text, okay? So we might, in the future AI systems, we might use them to turn abstract thoughts into, into language. Uh, in the h- human brain, that's done by a tiny little brain area right here called the Broca area. It's about this big. Um, that's our LLM, okay? Uh, but we don't think in language. We think in, uh, you know, mental representations of a situation. We have mental models of everything we think about. We can think even if, if we can speak. Uh, and that takes place here. That's like, you know, where real intelligence is, and that's the part that we haven't reproduced, uh, certainly with, with LLM. So the question is, um, you know, is, are we going to have eventually AI architectures, AI systems that are capable of, of not just answering questions that are already there, but solving, giving new solutions to problems that we specify? The answer is yes, eventually. Not with current LLMs. Um, and then the next question is, are they going to be able to ask their own questions, like figure out what are the good questions to answer? And the answer is eventually yes, but that's gonna take a while before we get machines that are capable of this. Like, you know, in, in humans, we, we have all the characteristic. We have people who are, who have extremely good memory. They can, um, you know, retrieve a lot of things. They have a lot of accumulated knowledge. Um, we have people who are problem solvers, right? You give them a problem, they'll solve it. And I think, uh, Tomas was actually talking about this kind of stuff. He said, like, you know, "If you're good at school, you're a good problem solver. We give you a problem, you can solve it, um, and you score well in math or physics or whatever it is." Um, but then in research, the, the most difficult thing is to actually ask the right, is to ask the, the, the good questions. What are the important questions? It's not just solving the problem, it's also asking the right questions, kind of framing, uh, a problem, you know, in the right way, so, so you have kind of, uh, new insight. And, and then after that comes, okay, I need to turn this into equations or into something, you know, practical, a model, um, and that may be a different skill from the one that asks the, the, the right questions. It might be a different s- skill also to, um, solve equations. The people who write the equations are not necessarily the people who write them, who, who solve them. Um, and, uh, or the people who remember that there is, you know, some textbook from hundred years ago where similar equations were solved, right? Those are three different skills. So LLMs are really good at retrieval. They're not good at solving new problems, get, you know, finding new solutions to new problems. They can retrieve existing solutions. And they're certainly not good at all at asking the right questions.

3. 5:40 – 10:13

   Reasoning in AI systems: limitations of chain of thought

   1. AKAlex Kantrowitz5:40

      And for those tuning in and, and learning about this for the first time, LLMs is the technology behind things like the GPT model that's within, baked within, uh, ChatGPT. But let, let me ask you this, Yann. So the AI field does seem to have moved from standard LLMs to LLMs that can reason, uh, and go step by step. And I'm curious, can you program this The, this sort of counterintuitive or, uh, this heretical thinking by imbuing a reasoning model with an instruction to question its directives.

   2. YLYann LeCun6:14

      Well, so y- we have to figure out what reasoning really mean. Okay? And there are, um, you know, obviously everyone is trying to, um, get LLMs to reason to some extent to perhaps be able to check whether the answer they produce are, are correct. Um, and the way people are approaching the problem at the moment is that they basically are trying to do this by modifying the current paradigm without completely changing it. Okay? So can you, you know, bolt a couple of words on top of LLM so that you kind of have some primitive, um, reasoning function? And that's essentially what a, a lot of the reasoning, uh, systems are, are doing. You know, one simple way of, uh, getting LLMs to kind of appear to reason is chain of thought, right? So you, you basically tell them to generate more tokens than they really need to in the hope that in the process of generating those tokens, they're gonna devote more computation to answering your question. And, uh, and to some extent that works surprisingly, but, but it's very limited. You don't actually get, uh, real reasoning, um, uh, out of this. Uh, reasoning, at least in classical AI and in, in many domain, um, involves a search through a space of potential solutions. Okay? So you have a, a problem to solve. You can characterize whether the problem is solved or not. So you have some way of telling whether the problem is solved. Um, and then you search through a space of solutions for when that actually, uh, satisfies the, the constraints or, y- you know, is, is identified as being a solution. Um, and that, you know, that's how, that's kind of the most general form of, of reasoning you can imagine. There is no mechanism at all in LLMs for this search mechanism. What you have is you have to kind of build this on top of it, right? So one way to do this is you get an LLM to produce lots and lots and lots of sequences of answers, right? So sequences of tokens, which, you know, represent answers. And then you have a, a separate system that picks which one is good. Okay? This is a bit like writing a program by sort of randomly more or less generating instructions, um, you know, while maybe, uh, respecting the grammar of the, of the language, um, and then checking all of those programs for when that actually works. It, not a good way, not a very efficient way of producing [chuckles] correct, uh, pieces of code. It's not a good way of reasoning, uh, either. So, um, a, a big issue there is, is that when, when humans or animals reason, we don't do it in token space. I- in other words, when we reason, we don't have to, you know, generate a text that expresses our solution, um, and then generate another one, and then generate another one, and then among all the ones we, we produce, pick the one that is good. We reason internally, right? We have a mental model of the situation, and we manipulate it in our head, and we find kind of a good solution. When we plan a sequence of actions to, I don't know, you know, build a table or something, um, we, we plan the sequence of action. Uh, you know, we have a mo- mental model of that in our head. If I, if I tell you-- And this has nothing to do with language, okay? So if I tell you, imagine a cube floating in front, in front of us right now. Now, rotate that cube 90 degrees along a vertical axis. Okay? You can imagine this, this thing taking place, and you can readily observe that it's a cube. If I rotate it 90 degrees, it's gonna look just like the cube that I started with, okay? Um, 'cause you have this mental model of a, of a cube. Um, and that reasoning is in some abstract continuous space. It's not in text. It's not related to language or, or anything like that. Um, and humans do this all the time, animals do this all the time, and this is what we yet cannot reproduce, uh, with, with machines.

4. 10:13 – 16:29

   LLMs approaching diminishing returns and the need for a new paradigm

   1. AKAlex Kantrowitz10:13

      Yeah. It reminds me, you're talking through chain of thought and how it doesn't produce much novel insights. And, um, when DeepSeek came out, one of the big screenshots that was going around was someone asking DeepSeek for a novel insight on the human condition. And as you read it, it's one-- another one of these very, like, clever tricks the AI pulls because it does seem like it's running through all these different, like, very interesting observations about humans, how we take our, uh, our, our hate, like our violent side, and we channel it towards cooperation instead of competition, and that helps us build more. And then you're like, as you read the chain of thought, you're like, "This is kind of just like you read Sapiens and maybe some other books, and that's your chain of thought."

   2. YLYann LeCun10:51

      Pretty much, yeah. I mean, yeah, a lot of it is, uh, regurgitation.

   3. AKAlex Kantrowitz10:57

      I'm now gonna move a part of the conversation I had later closer up, which is the wall, uh, effectively is training standard large language models coming close to hitting a wall, whereas before there was somewhat predictable, uh, returns. If you put a number, a certain amount of data and a certain amount of compute towards training these models, you can make them predictably better. Um, as we're talking, it seems to me like you believe that that is eventually not gonna be true.

   4. YLYann LeCun11:29

      Well, it, I don't know if I would call it a wall, but it's certainly diminishing return in the sense that, you know, we've kind of run out of natural text data to train those, those LLMs where they're already trained with, you know, on the order of, uh, you know, 10 to the 13 or 10 to the 14, uh, tokens.

   5. AKAlex Kantrowitz11:47

      That's a lot.

   6. YLYann LeCun11:48

      That's a lot, and that's the entire-

   7. AKAlex Kantrowitz11:49

      It's like the whole internet

   8. YLYann LeCun11:50

      ... that's all the-

   9. AKAlex Kantrowitz11:51

      And then some

   10. YLYann LeCun11:51

       ... publicly available internet, and then, you know, some companies license, uh, uh, content that is not publicly available. And then there is talks about like, you know, generating artificial data, uh, and then hiring thousands of people to kinda, you know, uh, generate more data to-

   11. AKAlex Kantrowitz12:05

       Buy all their knowledge, the PhDs and professors

   12. YLYann LeCun12:08

       Yeah, but in fact, uh, it could be even simpler than this because most of the systems actually don't understand basic logic, for example, [chuckles] right? So, um, so to some extent, you know, there's gonna be slow progress, uh, along those lines with, uh, synthetic data, with, you know, hiring more people to, you know, plug the holes in the, in the sort of n- you know, knowledge background of, uh, of, uh, of tho- of those systems. But, but it's diminishing return, right? The, the costs are building of, uh, generating the data and the, the returns are, are, are not, not that great. So we need a new paradigm, okay? We need, um, a new kind of architecture of systems that at the, you know, at the core are capable of those, uh, search and, uh, um, you know, searching for a good solution, checking whether that solution is good, uh, planning for a sequence of actions to arrive at a particular goal, which is what you would need for an agentic system to really work. Uh, everybody's talking about agentic system. Nobody has any idea how to build them other than, uh, basically regurgitating plans that have the system has already been trained on. Okay. So, you know, it's like, it's like everything in computer science. You, you can, you can engineer a, a solution, uh, which is limited for, for in the context of AI. Uh, you can make a system that is, you know, based on, on learning or retrieval with enormous amounts of data. But really the complex things, uh, the complex thing is, uh, how you build a system that can solve new problems without being trained to solve those problems. We are capable of doing this. Animals are capable of doing this. Facing a new situation, we can either, uh, solve it zero shot without, without training ourselves to handle that situation, just the first time we encounter it, or we can learn to solve it extremely quickly. So for example, um, you know, we, we, we can learn to drive in, in, you know, a couple dozen hours of practice. Um, and to the point that after twenty, thirty hours it becomes kinda second nature, right? It's become kinda subconscious. We don't even-

   13. AKAlex Kantrowitz14:23

       You don't even think about it

   14. YLYann LeCun14:24

       ... you don't need to think about it. You can-

   15. AKAlex Kantrowitz14:25

       Speaking of system one, system two, right?

   16. YLYann LeCun14:27

       That's right. So, you know, the, recalls the discussion we had with, with Danny Kahneman a few years ago. So, um, you know, the, the first time you drive, your system two is all, uh, present. You have to use it. You imagine all kind of catastrophe scenarios and stuff like that, right? You, you, your full attention is devoted to, to driving. But then after a number of hours, you know, you can talk to someone at the same time, like you don't need to think about it. It's become sort of, uh, subconscious and more, more or less automatic. Um, it's become system one. And pretty much every task that we, you know, learn, that we accomplish the first time, we have to use the full power of our, of our minds and then eventually if we repeat them, uh, sufficiently many times they get, they get kind of subconscious. I have this, uh, vivid memory of, uh, once being in a, a workshop where one of the participants was a chess grandmaster and he played a simultaneous game against like fifty of us, right? You know, going from, from-

   17. AKAlex Kantrowitz15:22

       [laughs]

   18. YLYann LeCun15:23

       ... one person to another. You know, I got wiped out in ten turns or something. I'm terrible at chess, right? But, um, so he would come, you know, come to my, my table and, you know, I w- I had time to think about this 'cause he, you know, he was playing the other fifty tables or something. So I make my move in front of it. He goes like, "What?" [laughs] And then immediately plays, right? He doesn't have to think about it. Um, I was not a, a challenging enough opponent that he had to actually call his system two. His system one was sufficient to beat me. Um, and uh, what that tells you is that when you become familiar with the task and you, you train, you train yourself, you know, it, it kind of, uh, becomes subconscious. But the, but the essential ability of humans and many animals is that when you face a new situation, you, you can think about it, figure out a sequence of actions, a course of action to accomplish a goal, um, and you don't need to know much about, about the situation other than your common knowledge of how the world works basically. That's what we're missing, okay, with, uh, with AI systems.

5. 16:29 – 21:36

   "A PhD next to you" vs. actual intelligent systems

   1. AKAlex Kantrowitz16:29

      And it, okay, now, now I really have to blow up the order here because you've said some very interesting things that we have to talk about. Um, you talked about how basically LLMs have hit the point of diminishing returns, large language models, the things that have gotten us here, and we need a new paradigm. But it also seems to me that that new paradigm is, isn't here yet, and I know you're working on the research for it and we're gonna talk about that, what the next new paradigm might be. But there's a real timeline issue, don't you think? Because I'm just thinking about the money that's been raised and put into this.

   2. YLYann LeCun17:01

      Yes.

   3. AKAlex Kantrowitz17:02

      Last year, six point six billion to OpenAI. Uh, last week or a couple weeks ago, another three and a half billion to Anthropic after they raised four billion, uh, last year. Elon Musk is putting, uh, ano- you know, another small fortune into, into building Grok. These are all LLM first companies. They're not searching out the next... I mean, maybe OpenAI is, but that six point six billion that they got was because of ChatGPT. So where's this field gonna go? Because if that money is being invested into something that is at the point of diminishing returns, requiring a new paradigm to progress, that sounds like a real problem.

   4. YLYann LeCun17:42

      Well, um, I mean, we have some ideas about what this paradigm sh- uh, is. The, the difficulty that... I mean, what we're working on is trying to make it work. [laughs] Um, and it's, you know, it's not simple. That may take, that may take, uh, that may take years. And so the question is, um, is the, the, are the capabilities we're talking about perhaps through this new paradigm that we're thinking of, that we're working on, uh, is it gonna come, uh, quickly enough- To justify all, all of this, uh, investment. Uh, and if it doesn't come quickly enough, is the investment still justified? Okay, so the first thing you can say is we are not going to get to human-level AI by just scaling up LLMs. This is just not going to happen, okay?

   5. AKAlex Kantrowitz18:27

      That's your perspective.

   6. YLYann LeCun18:29

      There's no way, okay? Absolutely no way. Um, and, and whatever you can hear from some of my, uh, more adventurous colleagues, uh-

   7. AKAlex Kantrowitz18:39

      [chuckles]

   8. YLYann LeCun18:39

      ... it's not going to happen within the next two years. There's absolutely no way in hell to, you know, pardon my French, um, the y- y- you know, the idea that we're, we're gonna have, you know, a country of genius in a data center, that's complete BS, right? There's absolutely no way. What we're gonna have maybe is systems that are trained on sufficiently large amounts of data that any question that any reasonable person may ask will, will find an answer through those systems, and it would feel like you have, you know, a PhD sitting next to you. But it's not a PhD you have next to you, it's, you know, a, a system with, uh, a gigantic, uh, memory and retrieval ability, not, not a system that can invent, uh, solutions to, uh, to new problems. Um, which is really what a, a PhD is [chuckles] . Okay, this is actually connect-

   9. AKAlex Kantrowitz19:23

      That's the whole deal

   10. YLYann LeCun19:24

       ... it's, it's, you know, connected to this post that, uh, uh, Tomas, Tomas Wolf, uh, uh, made that, uh, um, you, you, you, you... Inventing new things, you know, requires, uh, uh, the, the a type of, uh, of skill and abilities that, uh, you're not gonna get from, from, uh, from LLMs. So, um, so there's a big question, which is the investment that is being done now is not done for tomorrow. It's not, it's, it's done for, you know, the next few years and most of the investment, at least for, from the Meta side, is investment in, uh, infrastructure for inference. Okay, so let's imagine that by the end of the year, which is really the plan at Meta, we have one billion users of Meta AI through smart glasses, you know, standalone app and, and whatever. Um, you gotta serve those people, and that's a lot of computation. So that's why you need, you know, a lot of investment in infrastructure to be able to scale this up and, you know, build it up over months or, or years. Um, and so that, you know, that's where most of the money is going, um, uh, at least on, on, you know, on the side of companies like, like, like Meta and Microsoft and, and, and Google and potentially Amazon. Um, then there is... So this is just operations, essentially. Now, is there going to be the, the market for, um, you know, one billion people using those things regularly even if there is no change of paradigm? And the answer is probably yes. So, you know, even if the revolution or the new paradigm doesn't come, you know, within three years, this infrastructure is going to be used. Is, there's very little question about that, okay? So it's a good investment, and it takes so long to set up, you know, data centers and all that stuff that you need to, to get started now and plan for, you know, progress to be continuous, uh, so that, uh, you know, eventually the investment is, uh, is justified. But you can't afford not to do it, right? Because-

   11. AKAlex Kantrowitz21:32

       Right

   12. YLYann LeCun21:32

       ... um, because that would be too much of a, of a risk to take if

6. 21:36 – 25:37

   Consumer AI adoption vs. enterprise implementation challenges

   1. YLYann LeCun21:36

      you have the cash.

   2. AKAlex Kantrowitz21:37

      But let's go back to what you said. The stuff today is still deeply flawed, and there have been questions about whether it's going to be used. Now, Meta is making this consumer bet, right? The consumers want to use the AI. That makes sense. OpenAI has 400 million users of ChatGPT. The, and Meta has three, four billion. I mean, basically if you have a phone-

   3. YLYann LeCun21:56

      Six

   4. AKAlex Kantrowitz21:56

      ... on Earth.

   5. YLYann LeCun21:56

      Well, you have three, three point something-

   6. AKAlex Kantrowitz21:58

      Yeah

   7. YLYann LeCun21:58

      ... billion, uh, users. Uh, 600 million users of Meta AI.

   8. AKAlex Kantrowitz22:04

      Right. Okay. So more than ChatGPT.

   9. YLYann LeCun22:06

      Yeah. But they, they, but it's not used as much as ChatGPT.

   10. AKAlex Kantrowitz22:09

       Right.

   11. YLYann LeCun22:09

       So the users are not as intense-

   12. AKAlex Kantrowitz22:11

       As active

   13. YLYann LeCun22:11

       ... if you will.

   14. AKAlex Kantrowitz22:11

       But basically, the idea that, that Meta can get to a billion consumer users-

   15. YLYann LeCun22:15

       Yeah

   16. AKAlex Kantrowitz22:15

       ... that seems reasonable. But the thing is, a lot of this investment has been made with the idea that this will be useful to enterprises, uh, not just a consumer app. And there's a problem because, like we've been talking about, it's not good enough yet. Uh, you look at deep research, this is something Benedict Evans has brought up. Deep research is pretty good, but it might only get you 95% of the way there, and maybe 5% of it hallucinates. So if you have a hundred-page research report and 5% of it is wrong and you don't know what 5%, that's a, that's a problem. And similarly, in, in enterprises today, all, every enterprise is trying to figure out how to make, uh, AI useful to them, uh, generative AI useful to them and other types of AI. Uh, but only 10% or 20% maybe of proof of concepts make it out the door into production because they're, it's either too expensive or it's fallible. So if this is, if we are getting to the top here, uh, what do you anticipate's gonna happen with, uh, with everything that's, that, that has been pushed in the anticipation that it is gonna get even better f- from here?

   17. YLYann LeCun23:27

       Well, so again, it's a question of timeline, right? [chuckles] When, when are those systems going to become sufficiently reliable and intelligent so that the deployment is made easier? Um, but, but you know, uh, I mean, this, the situation you're describing that, you know, beyond the impressive demos, actually deploying systems that are reliable is where things tend to falter in, in the use of computers and technologies, and particularly AI. This is not new. Um, it's, it's basically, um, you know, why we, we had super impressive, you know, autonomous driving demos 10 years ago Um, but we still don't have level five [laughs] self, self-driving cars, right? Um, it's the last mile that's really difficult, uh, so to speak, uh, for cars.

   18. AKAlex Kantrowitz24:21

       [laughs]

   19. YLYann LeCun24:22

       You know, it's, you know, the last-

   20. AKAlex Kantrowitz24:23

       I see what you did there. Yeah. [laughs]

   21. YLYann LeCun24:23

       The last few... That was not deliberate. [laughs]

   22. AKAlex Kantrowitz24:27

       [laughs]

   23. YLYann LeCun24:27

       The, the, you know, the last few, few percent of, of reliability, which makes a system, uh, practical, um, and how you integrate it with sort of existing systems and, and, and blah, blah, blah, and, you know, how it makes, uh, users of it more efficient if, if you want, or more, uh, reliable or, or whatever. Um, that's where, that's where it's, that's where it's difficult. Um, and, you know, this is why, uh, if we take, if we go back several, several years and we look what happened with, uh, IBM Watson, okay? So Watson was going to be the thing that, you know, IBM was, was gonna push and generate tons of revenue by, by having Watson, uh, you know, learn about medicine and then be deployed in every, um, uh, every hospital. And it was, it was basically a complete failure and was sold for parts, right? Um, and cost a lot of money [laughs] to, to IBM, including the CEO. And the... What happens is that actually deploying those systems in, in situations where they are reliable and, and actually help people and don't, like, hurt the natural conservatism of the,

7. 25:37 – 29:37

   Historical parallels: expert systems and the risk of another AI winter

   1. YLYann LeCun25:37

      of the labor force, um, this is where things become complicated. We- we're seeing the same, you know, the process we're seeing now with the difficulty of deploying AI system is not new. It's, it's, it's happened, uh, absolutely at all, at all times. This is also why, you know, some, some of your listeners perhaps are too young to remember this, but there was a big wave of interest in AI in the 1980s, early 1980s, um, around expert systems. Um, and, you know, the, the hottest job in the 1980s was gonna, was going to be knowledge engineer, and your job was going to be to sit next to a, an expert and then, you know, turn the knowledge of the expert into rules and facts that would then be fed to a, um, inference engine that would be able to kind of derive new facts and, and answer questions and blah, blah, blah. Um, big wave of interest. Uh, the Japanese government started a big program called Fifth Generation Computer. The hardware was going to be designed to actually take care of that and blah, blah, blah. You know, mostly, mostly a failure. There was kind of a, a, a, a... You know, the wave of interest kind of died in the, the mid '90s, um, about this and, and, you know, a few companies were successful, but basically for a narrow set of applications for which you could actually reduce human knowledge to a bunch of rules and for which, um, uh, it was economy, economically feasible to do so. Um, but the, the, the wide-ranging impact on all of, uh, society and industry was just not there. And so that's the danger of, uh, of, of AI all the time. Um, I mean, the, the signals are clear that, you know, still, um, LLMs, uh, with all the bells and whistles actually play an info-im- important role, if nothing else, for information retrieval. Uh, you know, most companies, uh, want to have some sort of internal, um, experts that know all the internal documents so that any employee can ask any question. We have one at Meta. It's called Meta Mate. It's really cool. It's very useful.

   2. AKAlex Kantrowitz27:38

      Yeah. And I'm, I'm not suggesting that AI is gonna, that modern AI is not, or modern generative AI is not useful or, uh... I'm, I'm asking purely that there's been a lot of money that's been invested into expecting this stuff to effectively achieve God-level capabilities. And we both are talking about how, like, there's, you know, potentially diminishing returns here, and then what happens if there's that timeline mismatch like you mentioned.

   3. YLYann LeCun28:05

      Yeah.

   4. AKAlex Kantrowitz28:05

      And, um, this is the last question I'll ask about it because I feel like we have so much else to cover, but I feel like timeline mismatches, uh, that might be personal to you. You and I first spoke nine years ago, which is crazy now, nine years ago, uh, uh, and you know, about how in the early days you had an idea for how AI should be structured and you couldn't even get a seat at the conferences. Um, and then eventually with the right amount of com... When, when the right amount of compute came around, those ideas started working and then the entire AI field took off based off of your idea that you, you worked on with, uh, Bengio and Hinton. Um, but-

   5. YLYann LeCun28:45

      And a bunch of others. [laughs]

   6. AKAlex Kantrowitz28:46

      And many others. Uh, and, but you know, for the sake of efficiency-

   7. YLYann LeCun28:50

      [laughs]

   8. AKAlex Kantrowitz28:50

      ... we'll say go look it up. Um, but just talking about those mismatched timelines, when there have been over-hyped moments, uh, in the AI field, maybe with expert systems that you were just talking about, and they don't pan out the way that people expect, the AI field goes into what's called AI winter.

   9. YLYann LeCun29:06

      Well, there's a backlash. Yep.

   10. AKAlex Kantrowitz29:07

       Correct. And so if we're gonna, if we are potentially approaching this moment of mismatched timelines, do you fear that there could be another winter now, given the amount of investment, uh, given the fact that there's gonna be potentially diminishing returns with the main way of training these things? And maybe we'll add in the fact that the market is, is, the stock market looks like it's going through-

   11. YLYann LeCun29:29

       [laughs]

   12. AKAlex Kantrowitz29:29

       ... a bit of a downturn right now. Now, that's a variable, uh, probably the third most important variable of what we're talking about, but it has to factor.

8. 29:37 – 33:19

   Four critical capabilities AI needs for true understanding

   1. YLYann LeCun29:37

      So, uh, yeah, I, I think, um, I mean, there's certainly, uh, uh, a question of timing there, but I think, uh, if we try to dig a little bit deeper, um, as I said before, if you think that we're gonna get to human level AI by just training on more data and scaling up LLMs, you're making a mistake. So if you're, if you're an investor and you invested in a company that told you, "We're gonna get to human le- level AI and PhD level by just, you know, training on more data and with a few tweaks," um, I don't know if you're gonna use your shirt, but that was probably not a good idea. Um, however, there are ideas about how to, uh, go forward and have systems that are capable of doing what, what every intelligent Animal and, and human are capable of doing and that, uh, currently AI systems are not capable of doing, and I'm, I'm talking about understanding the physical world, um, having persistent memory and being able to reason and plan. Those are the four characteristics that, that, you know, need to be there. Um, and that requires systems that, you know, can acquire common sense, that can learn from, uh, natural sensors like video as opposed to just text, j- just human produced, uh, uh, data. Um, and that's a big challenge. I mean, I've been talking about this for many years now and, uh, and saying, "This is, this is where the challenge is. This is what we, we have to, uh, to figure out." And, and my group and I have, or people working with me and others who, who have listened to me, are making progress, uh, along, along this line, uh, of, uh, systems that can be trained to understand how the world works on video, for example. Systems that can use mental models of how the world, the physical world works to plan sequences of actions to arrive at a particular goal. So we, we have kind of early results of these kind of systems. Uh, and there are people at DeepMind working on similar things, and there are, you know, people in various universities working on this. Uh, so, um, the question is, you know, when is this going to go from, uh, interesting research papers, uh, demonstrating a, a new capability with a new architecture to, you know, architectures at scale that, you know, are practical for a lot of applications and can find solutions to new problems without being trained to do it, um, et cetera. And, you know, it, it's not gonna happen within the next three years, but it may happen with, you know, between three to five years, something like that, and that's kind of corresponds to k- you know, the, the sort of ramp up that we see in, uh, uh, in, in investment. Now, whether other... So, so that, that's the first thing. Now, the, the second thing that's important is that there's not going to be one secret magic bullet that one company or one group of people is going to invent that is going to just solve the problem. Um, it's gonna be a lot of different ideas, a lot of effort, some principles around which to base this that, that some people may, may not subscribe to and will, will go, um, in a direction that is, you know, will turn out to be a dead end. Uh, so there's not gonna be, like, a day before which there is no AGI and after which we, we have AGI. This is not going to be an event. Um, it's gonna be continuous conceptual ideas that, as time goes by, are gonna be made bigger and to scale and gonna work better, and it's not gonna come from a single entity. It's gonna come from the entire research community across the world, and the people who share their research are gonna move faster than the ones that don't. And so if you think that there is some startup somewhere with five people who has discovered the secret of AGI and you should invest five billion in them,

9. 33:19 – 37:24

   Testing AI's physics understanding with the paper test

   1. YLYann LeCun33:19

      you're making a huge mistake.

   2. AKAlex Kantrowitz33:20

      You know, Yann, first of all, I always enjoy our conversations because we start to get some real answers. And I remember even from our last conversation, I was just, you know, always looking back to that conversation saying, "Okay, this is what Yann says. This is what everybody else is saying. I'm pretty sure that this is the grounding point," and that's been correct, and I know we're gonna do that with this one as well. And now you've set me up for two, uh, interesting threads that we're gonna pull out, um, as we go on with our conversation. First is the understanding of physics and the real world, and the second is open source. So we'll do that when we come back right after this. And we're back here with Yann LeCun. He is the chief AI scientist at Meta and the Turing Award winner that we're thrilled to have on our show, luckily, for the third time. Um, I wanna talk to you about physics, Yann, because there's sort of this famous moment in Big Technology Podcast history, and I say famous with our listeners. [chuckles] I don't know if it really extended beyond, but you had me, uh, uh, write to ChatGPT, "If I hold a paper horizontally with both hands and let it, let it go, let go of the paper with my left hand, uh, what will happen?" And, uh, I write it and it convincingly says, like it writes the, the physics will, will happen and the paper will float towards your left hand, and I read it out loud convinced, and you're like, "That thing just hallucinated and you believed it."

   3. YLYann LeCun34:34

      [laughs]

   4. AKAlex Kantrowitz34:35

      And that is what happened. So listen, it's been two years. I put the test to ChatGPT today. Uh, it says, um, "When you let go of the paper with your left hand, grav- gravity will cause the left side of the paper to drop while the right side, still held up by your right hand, remains in place. This creates a pivot effect where the paper rotates around the point where your right hand is holding it."

   5. YLYann LeCun34:58

      Yeah.

   6. AKAlex Kantrowitz34:58

      So now it gets it right.

   7. YLYann LeCun34:59

      It learned the lesson. You know, it's quite possible that this, uh, um, you know, some- someone hired by OpenAI to solve the problem was fed that question and sort of fed the answer, and the system was fine-tuned with the answer. I mean, you know, obviously you can imagine an infinite number of such questions and, and this is where, you know, uh, the, the, the so-called post-training of LLM becomes expensive, um, which is that, you know, how much coverage of all those style of questions do you have to do for the system to basically cover 90% of or 95% or whatever percentage of all the questions that people may ask it? Um, but there, you know, it's, there is a long tail and there is no way you can train the system to answer all possible questions because there is an essentially infinite number of them and, and there is way more questions the system cannot answer that, uh, than questions it can, it can answer. You cannot cover the set of all possible training, uh, you know, questions in the training set, the training set.

   8. AKAlex Kantrowitz36:00

      Right. So because I think our conversation last time was saying you, you said that because these actions of, like, what's happening with the paper if you let go of it with your hand has not been covered widely in text, the model won't really know how to handle it because unless it's been covered in text-

   9. YLYann LeCun36:16

      Right

   10. AKAlex Kantrowitz36:16

       ... the model won't have that understanding, won't have that inherent understanding-

   11. YLYann LeCun36:19

       Right

   12. AKAlex Kantrowitz36:19

       ... of the real world, and I've kinda gone with that for a while. Uh, then I said, "You know what? Let's, let's try to generate some AI videos." And one of the interesting things that I've seen with the AI videos is there is some understanding of how the physical world works there. Um- In a way that i-in our first meeting nine years ago, you said, um, "One of the hardest things to do is you ask an AI what happens if you hold a pen vertically on a table and let go. Uh, will it fall?" And there's, like, an unbelievable amount of permutations, uh, that can occur, and it's very, very difficult for the AI to figure that out because it, it just doesn't in-inherently understand physics. But now you go to something like Sora, uh, and you say, um, "Show me a, a video of a man sitting on a chair kicking his legs," and you can get that video. And the person sits on the chair, and they kick their legs, and the legs, you know, don't fall out of their sockets or stuff. They bend at the joints.

   13. YLYann LeCun37:17

       And they don't have three legs.

   14. AKAlex Kantrowitz37:19

       And they don't have three legs. So wouldn't that suggest an improvement of the capabilities here with these large,

10. 37:24 – 43:33

    Why video generation systems don't equal real comprehension

    1. AKAlex Kantrowitz37:24

       large models?

    2. YLYann LeCun37:24

       No.

    3. AKAlex Kantrowitz37:25

       Why?

    4. YLYann LeCun37:26

       Because you still have those videos produced by those, uh, video generation system where, you know, you spill a, a glass of wine and, and the wine, like, floats in the air or, like, flies off or disappears or whatever. And, um, so, you know, of course, for every specific situation, you can always collect more data for that situation and then train your model to handle it. But that's not really understanding the underlying, uh, reality. This is just, you know, compensating, uh, the, the lack of understanding by, uh, uh, increasingly large amounts of data. Um, you know, children understand, uh, you know, simple concepts like, like gravity, um, with a surprisingly small amount of data. Um, so in fact, there is an interesting c-calculation you c-you can do, which I've talked about publicly before. But, um, if you take a LLM, typical LLM trained on thirty trillion tokens, something like that, right, three ten to the thirteen tokens. A token is about three bytes, so that's point nine ten to the fourteen tokens. Let's say ten to the fourteen tokens to, to round this up. Um, that text would take any of us probably some-- on the order of four hundred thousand years to read, right?

    5. AKAlex Kantrowitz38:49

       No problem. [laughs]

    6. YLYann LeCun38:51

       At twelve hours a day, okay? [laughs] Um, now, um, a four-year-old has been awake a total of sixteen thousand hours. Uh, you can multiply by three thousand six hundred to give number of seconds, and then you can put a number on how, like, how much data has got into your visual cortex through the optic nerve. Optic nerve, each optic nerve, we have two of them, carries about one megabyte per second, roughly, right? So it's two megabytes per second, uh, times three thousand six hundred times sixteen thousand, and that's just about ten to the fourteen bytes. Okay, so in four years, a child has seen through vision or touch, for that matter, as much data as the biggest LLMs. And, and it tells you clearly that we're not gonna get to human level AI by just training on text. It's just not a rich enough source of information. Um, and by the way, sixteen thousand hours is not that much video. It's thirty minutes of YouTube uploads, okay?

    7. AKAlex Kantrowitz39:47

       Mm-hmm.

    8. YLYann LeCun39:48

       Uh, we can get that pretty easily. Now, in nine months, uh, a baby has seen, you know, um, let's say ten to the thirteen bytes or something, which is not, not much again. Um, and in that, in that time, baby has learned basically all of intuitive physics that, that, uh, that, that, that we know about. Um, you know, cons-conservation of momentum, gravity, conservation of momentum, the fact that object don't spontaneously disappear, the fact that they still exist even if you hide them. I mean, there's all kinds of stuff, you know, very basic stuff that we learn about the world in the first few months of life. Um, and this is what we need to reproduce with machine, this type of learning of, you know, figuring out, uh, what is possible and impossible in the world, what will result from an action you take, um, so that you can plan a sequence of actions to arrive at a particular goal. That's the idea of a world model. And now, connected with your question about, uh, video generation systems, is the right way to approach this problem to train better and better video generation systems? And my answer to this is absolutely no. Um, the, the problem of understanding the world does not go through the solution to the, to, to generating video at the pixel level, okay? I don't need to know, um, if I, if I take this, uh, this glass of, uh, [laughs] of this cup of, um, of water and I spill it, I cannot entirely predict, you know, the exact path of-- that the water will, will, uh, uh, follow on the table and what shape it's gonna take and all that stuff, what noise it's going to make. Um, but at a certain level of abstraction, I can make a prediction that the water will spill, okay? And it, it, it'll probably make my phone wet and everything. So, um, so at a, at a, at a-- I can't predict all the details, but I can predict at some level of abstraction. And I think that's really a critical concept, the fact that if you want a system to be able to learn to comprehend the, the world and understand how the world works, it needs to be able to learn an abstract representation of the world that allows it to make those prediction. And, um, what that means is that those architectures will not be generative.

    9. AKAlex Kantrowitz42:14

       Right. And, uh, I wanna get to your solution here in a moment, but I just wanted to also, like, what would a conversation between us be without a demo? So I wanna just show you, I'm gonna put this on the screen when we do the video, but there's-- This is a video I was pretty proud of. I got this guy sitting on a chair kicking his legs out, and the legs stay attached to his body. And I was like, "All right, this stuff is making real progress." And then I said, "Can I get a car going into a haystack?" And so it's two bales of haystacks, and then a haystack magically emerges from the hood of a car-

    10. YLYann LeCun42:45

        [laughs]

    11. AKAlex Kantrowitz42:46

        ...that's stationary. And I just said to myself, "Okay." Yann wins again

    12. YLYann LeCun42:51

        It's a nice car though. Yeah. I mean, the thing is those systems have been fine-tuned with a huge amount of data for humans because that's what people are asking.

    13. AKAlex Kantrowitz43:02

        Right.

    14. YLYann LeCun43:02

        Most videos they ask to do. So there is a lot of data of humans doing various things to train those systems. So that's why it works for humans, but not for a situation that the people training that system had not anticipated.

    15. AKAlex Kantrowitz43:17

        So you said that the model can't be generative to be able to understand the real world.

    16. YLYann LeCun43:22

        That's right.

    17. AKAlex Kantrowitz43:22

        You are working on something called V-JEPA.

    18. YLYann LeCun43:26

        JEPA.

    19. AKAlex Kantrowitz43:26

        JEPA, right. V is the video.

    20. YLYann LeCun43:28

        Yeah.

    21. AKAlex Kantrowitz43:28

        You also have I-JEPA for images.

    22. YLYann LeCun43:30

        Right.

    23. AKAlex Kantrowitz43:31

        That is-

    24. YLYann LeCun43:31

        We have JEPA for all kinds of stuff.

11. 43:33 – 51:10

    Self-supervised learning and its limitations for understanding

    1. YLYann LeCun43:33

       Text also.

    2. AKAlex Kantrowitz43:34

       And text. So explain how that will solve the problem of being able to allow a machine to abstractly represent what is going on in the real world.

    3. YLYann LeCun43:43

       Okay. So what has made the success of, uh, uh, AI and particularly, um, natural language understanding and chatbot in the last few years, but also to some extent computer vision, is self-supervised learning. So what is self-supervised learning? It's, um, take an input, be it an image, a video, a piece of text, whatever, uh, corrupt it in some way and train a big neural net to reconstruct it. Basically recover the uncorrupted version of it or the undistorted version of it, or a transformed version of it that would result from taking an action. Okay? Um, and that would mean, um, for example, in the context of text, take a piece of text, remove some of the words, and then train some big neural net to predict the words that are missing. Take an image, remove some pieces of it, and then train a big neural net to recover the full image. Take a video, remove a piece of it, train a neural net to predict what's missing. Okay? So LLMs are a special case of this where, um, you, you, you take a, a text and you train the system to just reproduce the text, and you don't need to corrupt the text because the system is designed in such a way that to predict one particular word or token in the text, it can only look at the tokens that are to the left of it. Okay? So, so in effect, the system has hardwired into its architecture the fact that it cannot look at the present and the future, um, to predict the present. It can only look at the past. Okay. So, but basically, you train that system to just reproduce its input on its output. Okay? Um, so this kind of architecture is called a causal architecture, and this is what an LLM is, a large language model. That's what all the chatbots in the world are, are based on. Um, take a piece of text and train that system to just reproduce that piece of text on its output. Um, and to predict a particular word, it can only look at the word to the, to the left, uh, of it. And so now what you have is a system that given a piece of text, can predict the word that, that follows, um, that text. And you can take that, uh, that word that is predicted, uh, shift it into the input, and then predict the second word.

    4. AKAlex Kantrowitz45:58

       Mm-hmm.

    5. YLYann LeCun45:58

       Shift that into the input, predict the third word. That's called autoregressive prediction. It's not a new concept. Very old. Um, so self-supervised learning does not train to do a particular t- does not, uh, train a system to accomplish a particular task other than capture the internal structure of the, of the data. It doesn't require any labeling by, by a human. Okay? So apply these two images. Um, take an image, mask a chunk of it, like a bunch of patches from it if you want, and then train a big neural net to reconstruct that, that what is missing. And now use the internal representation of the image learned by the system, um, as input to a subsequent downstream task for, I don't know, image recognition, segmentation, whatever it is. It works to some extent, but not great. Um, so there was a big project like this, uh, to do this at FAIR. It's called MAE, Masked Autoencoder. It's a special case of denoising autoencoder, which itself is the sort of general framework from which I, um, I derive this, uh, this idea of s- self-supervised learning. So, um, it doesn't work so well. Um, and there's various ways to... If you apply this to video, also, I've been working on this for almost 20 years now. Take a video, show just the, a piece of the video, and then train a system to predict what's gonna happen next in the video. So same idea as for text, but just for, for video. And that doesn't work very well either. Um, and the reason it doesn't work, why does it work for text and not for video, for example? Um, and the answer is it's easy to predict a word that comes after a text. You cannot exactly predict which word follows a particular text, but you can produce something like a probability distribution over all the possible words in your dictionary, all the possible tokens. There's only about 100,000 possible tokens, so you, you just produce a big vector with 100,000 different numbers that are positive and sum to one. Okay. Um, now what are you gonna do to represent a probability distribution over all possible frames in a video or all possible missing parts of an image? We don't know how to do this properly. In fact, it's mathematically intractable to represent distributions in high dimensional continuous spaces. Okay? We don't know how to do this in a kind of useful way, if you want. Um, and so, and I've tried to do this for video for a long time. Um, and so that is the reason why those idea of self-supervised learning using generative models have failed so far. And this is why, you know, using a, you know, trying to train a video generation system as a way to understand, to get a system to understand how the world works. That's why it's, it can't succeed. Um- So what's the alternative? The alternative is something that is not a generat- generative architecture, uh, which we call JEPA. So that means joint embedding predictive architecture. And we know this works much better than attempting to reconstruct. So we, we've had, uh, experimental, uh, results on learning good representations of, of images going back many years, where instead of taking an image, corrupting it, and attempting to reconstruct this image, we take the original full image and the corrupted version. We run them both through neural nets. Those neural nets produce representations of those, of those two, uh, images, the initial one and the corrupted one. And we train another neural net, a predictor, to predict the representation of the full image from the representation of the corrupted-

    6. AKAlex Kantrowitz49:54

       Corrupted one. Mm-hmm

    7. YLYann LeCun49:55

       ... okay. And if you train a system, if you're successful, you train a system of this type, this, this is not trained to reconstruct anything. It's just trained to learn a representation so that you can make prediction within the representation layer. And you have to make sure that the representation contains as much information as possible about the input, which is where it's difficult, actually. That's the difficult part of training those systems. So that's called a JEPA, joint embedding predictive architecture. Um, and to, to train a system to learn good representations of images, those joint embedding architectures work much better than the ones that are generative, that are trained by reconstruction.

    8. AKAlex Kantrowitz50:33

       Mm-hmm.

    9. YLYann LeCun50:34

       Um, and now we have a version that works for video, too. So w- we take a video, we corrupt it by masking a big chunk of it. We run the full video and the corrupted one through encoders that are identical. And then w- and simultaneously, we train a predictor to predict the representation of the full video from the partial one. And the representation that the system learns of videos, when you feed it to a system that you train to tell you, for example, what action is taking place in the video or whether the video is possible or impossible or things like that, it actually works

12. 51:10 – 54:33

    JEPA: Building abstract representations for reasoning and planning

    1. YLYann LeCun51:10

       quite well. Um-

    2. AKAlex Kantrowitz51:11

       That's cool. So it gives that abstract thinking-

    3. YLYann LeCun51:13

       Yeah

    4. AKAlex Kantrowitz51:13

       ... in a way.

    5. YLYann LeCun51:14

       Right. And, and we have experimental result that shows that this, uh, joint embedding training, we have several methods for doing this. Uh, there's one that's called DINO. Another one that's called VCREG. Another one that's, or VCREG. Another one that's called, uh, iJEPA, but which is sort of, uh, a distillation method. Um, and so we ha- you know, several different ways to, to approach this, but one of those is going to lead to a recipe that basically gives us a, a general way of training those JEPA architectures. Okay, so it's not generative because the system is not trained to regenerate the part of the input. It's trained to generate a representation, an abstract representation of the input. And what that allows it to do is to ignore all the details about the input-

    6. AKAlex Kantrowitz51:56

       Mm-hmm

    7. YLYann LeCun51:56

       ... that are really not predictable. Like, you know, the, the pen that you put on the table vertically, and when you let it go, you cannot predict in which direction it's gonna fall. But at some abstract level, you can say that the pen is gonna fall.

    8. AKAlex Kantrowitz52:08

       It's falling.

    9. YLYann LeCun52:09

       Right?

    10. AKAlex Kantrowitz52:09

        Right.

    11. YLYann LeCun52:09

        Without representing the direction. Um, so, so that's the, that's the idea of JEPA. And, and we're starting to have, you know, good results on sort of, um, uh, having systems. So the VJEPA system, for example, is trained on natural, lots of natural videos, and then you can show it a video that's impossible, like a video where, for example, an object disappears or changes shape, okay? You can generate this with a game engine or something. Or a situation where you have a ball rolling and it rolls and it stops behind a screen, and then the screen comes down and the ball is not there anymore.

    12. AKAlex Kantrowitz52:45

        Right.

    13. YLYann LeCun52:46

        Okay? Um, so things like this, and you measure the prediction error of the system. So the, the system is ta- trained to predict, right? Um, not necessarily in time, but, but, like, basically to predict, you know, the, the, the sort of coherence of the video. And so you, you measure the prediction error as you show the, the video to the system. And when something impossible occurs, the prediction error goes, goes through the roof.

    14. AKAlex Kantrowitz53:11

        Goes up.

    15. YLYann LeCun53:11

        And so you can detect if the system has integrated some idea of what, you know, is possible physically or what's not possible, by just being trained with physically possible natural videos. Um, so that, that's really interesting. That's sort of the first hint-

    16. AKAlex Kantrowitz53:26

        Mm-hmm

    17. YLYann LeCun53:26

        ... that a system has acquired some level of common sense. And-

    18. AKAlex Kantrowitz53:29

        Yes

    19. YLYann LeCun53:30

        ... um, we have versions of those systems also that are so-called action conditions. So basically, we have things where we have, um, a chunk of video or an image of, you know, the state of the world at time T, and then an action is being taken, like, you know, a robot arm is being moved or whatever. And then, of course, we can observe the, the result, um, uh, resulting from this action. So now what we ha- when, when we train a JEPA with this, um, the, the model basically can say, "Here is the state of the world at time T. Here is an action you might take. I can predict the state of the world at time T plus one in this abstract representation space."

    20. AKAlex Kantrowitz54:06

        So there's, there's this learning of, of how the world works.

    21. YLYann LeCun54:07

        Of how the world works.

    22. AKAlex Kantrowitz54:08

        That's cool. Yes.

    23. YLYann LeCun54:08

        And, and the cool thing about this is that now you can imagine, you can have the system imagine what would be the outcome of a sequence of actions. And if you give it a goal, saying, like, "I want the world to look like this at the end. Can you figure out a sequence of actions to get me to that point?" It can actually figure out by search for a sequence of actions that will actually produce that result. That's planning.

    24. AKAlex Kantrowitz54:31

        That's crazy.

    25. YLYann LeCun54:31

        That's reasoning. That's actual reasoning

13. 54:33 – 58:57

    Open source vs. proprietary AI development

    1. YLYann LeCun54:33

       and actual planning.

    2. AKAlex Kantrowitz54:34

       Okay, Yann, I have to get you out of here. We're, we are over time, but can you give me, like, 60 seconds, your reaction, uh, to DeepSeek and sort of has open source overtaken the propriety mo- proprietary models at this point? And we got a limit to 60 seconds, otherwise I'm gonna get, uh, killed by your team here. So give your take.

    3. YLYann LeCun54:52

       Overtaken is a, is a strong word. I think, uh, progress is faster in the open source world, that's for sure. But of course, you know, uh, the propri- proprietary shops are profiting from the progress of the open source world, right? They get access to that information like everybody else. Um, so what, what's clear is that there is many more interesting ideas coming out of the open source world that any single shop, as big as it can be, cannot Come up with. You know, nobody has a monopoly on good ideas, and so the magic efficiency of the open source world is that, uh, it recruits talents from all over the world. And so what we've seen with DeepSeek is that if you set up a small team with a relatively long leash and few constraints on coming up with just the next generation of, of, of LLMs, they can actually come up with new ideas that nobody else had come up with, [laughs] right?

    4. AKAlex Kantrowitz55:45

       Mm-hmm.

    5. YLYann LeCun55:45

       They can sort of reinvent a little bit what, what, uh, you know, how you do things. And then if they share that with the rest of the world, then the entire world progresses. Okay, and so, um, uh, the, the, it's, it, it clearly shows that, um, you know, open source progresses faster, um, and, um, you know, a lot more innovation can, can take place in the open source world, which the proprietary world may have a hard time catching up with. Uh, it's cheaper to run. What we see is, uh, for, you know, partners who we, we talk to, um, they say, "Well, our clients, when they prototype something, they may use a proprietary API, but when it comes time to actually deploy the product, they actually use Llama or open source-

    6. AKAlex Kantrowitz56:28

       Mm-hmm

    7. YLYann LeCun56:28

       ... or other open source engines because it's cheaper and it's more, uh, secure, you know, it's more controllable, you can run it on premise, you know, there's all kinds of advantages." So, um, y- we've seen also a big evolution in the, the thinking of some people who are initially worried that, uh, open source efforts, uh, were going to, I don't know, for example, you know, help the Chinese or something, if you have like some geopolitical reason to think it's a bad idea. Um, but what DeepSeek has shown is that the Chinese don't need us. I mean, they can come up with really good ideas, right? I mean, we all know that there are really, really good scientists in, in China, and, uh, one thing that is not widely known is that the single most cited paper in all of science is a paper on deep learning from ten years ago, from 2015, and it came out of Beijing.

    8. AKAlex Kantrowitz57:23

       Oh.

    9. YLYann LeCun57:24

       Okay? It's ca- uh, the paper is called, uh, ResNet. So it's a particular type of architecture of neural net where basically by default, every stage in a deep learning system computes the identity function. It just copies its input on its output, and what the neural net does is compute the deviation from this identity. Okay? So that allows to train extremely deep neural net with, you know, dozens of layers, perhaps, uh, 100 layers. And it was, uh, the first author of that paper is a gentleman called Skaming- uh, called Kaiming He. At the time he was working at Microsoft Research Beijing.

    10. AKAlex Kantrowitz57:59

        Mm-hmm.

    11. YLYann LeCun57:59

        Um, soon thereafter, the publication of, of that paper, he joined FAIR in California, so I hired him, um, and worked at FAIR for eight years or so, and, uh, recently left and is now a professor at MIT. Okay, so, uh, there are really, really good scientists, uh, everywhere in the world. Nobody has a monopoly on good ideas. Certainly Silicon Valley does not have a monopoly on good ideas. Um, or another example of that is actually the first Llama came out of Paris. It came out of, uh, the FAIR labs in Paris, a, a small team of 12 people. Um, so, um, you have to take advantage of the diversity of ideas, backgrounds, uh, creative juices of the entire world if you want, uh, science and technology to progress fast, and that's enabled by, by open

14. 58:57 – 59:22

    Conclusion

    1. YLYann LeCun58:57

       source.

    2. AKAlex Kantrowitz58:58

       Yann, it is always great to speak with you. Uh, appreciate it. This is our, I think, fourth or fifth time speaking again going back nine years ago. You always help me see through all the hype and the buzz and actually figure out what's happening, and I'm sure that's gonna be the case for our listeners and viewers as well. So Yann, thank you so much for coming on. Hopefully do it again soon.

    3. YLYann LeCun59:15

       Thank you, Alex.

    4. AKAlex Kantrowitz59:16

       All right, everybody. Thank you for watching. We'll be back on Friday to break down the week's news. Until then, we'll see you next time on Big Technology Podcast.

Episode duration: 59:22