Yann LeCun: Meta’s New AI Model LLaMA; Why Elon is Wrong about AI; Open-source AI Models | E1014

1. 0:00 – 0:32

   Introduction

   1. YLYann LeCun0:00

      AI is going to bring a new renaissance for humanity, a new form of enlightenment, if you want. Because AI is going to amplify everybody's intelligence. It's like every one of us will have a staff of people who are smarter than us and know most things about most topics. So it's going to empower every one of us.

   2. HSHarry Stebbings0:17

      Yann, I am so excited for this. I heard so many great things from our mutual friends, obviously David Marcus and then Matthieu at PhotoRoom. So thank you so much for joining me today.

   3. YLYann LeCun0:26

      It's a pleasure.

   4. HSHarry Stebbings0:27

      Now, I would love to start, I heard some of the early stories, but I want to start with one from

2. 0:32 – 6:25

   Yann LeCun's Journey to Chief AI Scientist at Meta: A History of AI

   1. HSHarry Stebbings0:32

      David Marcus. How did you first enter the world of AI and make that first foray?

   2. YLYann LeCun0:37

      I was still an undergraduate, uh, engineering student in France, and I stumbled on a philosophy book which was a debate between, uh, Jean Piaget, you know, the cognitive psychologist, and, uh, Noam Chomsky, the famous linguist. And they were arguing about nature versus nurture for lang- for language, whether language is acquired or innate. So Chomsky was on the side of innate and Piaget on the side of acquired with, you know, some innate structure. And on the side of Piaget was, uh, a guy called Seymour Papert, who was a professor at MIT. In his argument, he talked about something called a Perceptron, which was an early, uh, machine learning system. And I- I read this and discovered that people had been working on, uh, machines that could learn and I was fascinated, and I started digging the literature. Soon discovered that much of that literature was in the 1950s and '60s and basically stopped in the late '60s because of a book, that they killed it, and Seymour Papert was a co-author of that book. Um, so strangely enough. And- and here he was 10 years later actually, uh, praising the Perceptron as kind of a- a amazing concept. So I was hooked. I had s- you know, started getting interested in what was not yet called machine learning, but eventually became neural nets and now deep learning.

   3. HSHarry Stebbings1:55

      Can I ask you? David asked this as well. How long did it take to get... uh, in terms of, like, the major breakthroughs, how long did it take you to get to the major breakthroughs that you're at the origin of when you look back over that time to get to those major breakthroughs?

   4. YLYann LeCun2:09

      Well, so there's a- a few breakthroughs. So the first one was, uh, in- in the... when I was still an undergrad basically finishing my engineering studies, uh, I figured out that the- the way forward to kind of lift the limitations of the old systems that were abandoned in the '60s was to find learning algorithms that could train multi-layer neural nets essentially. And people had all but abandoned this, uh, type of research except for a handful of people in Japan.

   5. HSHarry Stebbings2:36

      (laughs) .

   6. YLYann LeCun2:36

      And one guy I heard, I heard about called Geoff Hinton-

   7. HSHarry Stebbings2:42

      Good.

   8. YLYann LeCun2:42

      ... um, who had published a paper in 1983, so this was just, uh, the year I graduated, on something called the Boltzmann machine which, uh, was clearly a- a method to go beyond those- those limitations. And so I had, on my side, kind of developed a- a method for training multi-layer nets which was very close to what we now call backpropagation, but not exactly the same. It was closer to what we call target prop actually nowadays. And then, you know, published a few papers in French and eventually met Geoff at a meeting in France in 1985, and we realized we'd been working on the same thing and we were thinking alike. And, but I was, you know, in the middle of my PhD and he was a associate professor at Carnegie Mellon. So we, we started, you know, a discussion and then, you know, visited him at Carnegie Mellon for a summer school he organized, and then I, when I finished my PhD I did a postdoc with him and then joined Bell Labs. And- and when I was in Toronto, I developed what- what's called convolutional nets now, convolutional networks, which, you know, is a major method for image and speech, uh, processing nowadays. And so that's- that's what I'm- I'm best known for but, uh, um, but it started much earlier.

   9. HSHarry Stebbings3:52

      I- I have to ask. Yoshua described kind of the- the hype cycles within, uh, AI and neural nets like deserts when you're not in them, and he asked the question, "How did Yann not get discouraged when for a solid decade we were in a desert where no one really cared about neural nets?" How do you keep the enthusiasm bluntly when, as Yoshua said, no one really cared?

   10. YLYann LeCun4:16

       Both Yoshua, Geoff, and I had in the back of our minds that those methods would eventually come to the fore and that, you know, we would have to kind of snap people out of their preconceived ideas about, uh, about neural nets. So yes, there was... Um, so Yoshua and I were actually working together at, uh, AT&T, uh, Bell Labs in the early '90s and then the interest of the community for those methods started waning around 1995 or so. And it was indeed about 10 years when not only nobody was interested in neural nets but people were even making fun of it, you know, talking about it in, uh, sort of disparaging terms. Now there- there is, uh, something though, in 1996 I kinda changed job. I- I stayed in the same company, I was still working at AT&T in the research labs but I became a- a department head, and this was the early days of the internet. And, uh, my group and I started working on something completely different that had nothing to do or not much to do at least with machine learning, it was, uh, image compression. I had this- this idea that, uh, with the internet coming up we should have a way of scanning existing paper documents and then, you know, put them on the internet so that everybody could- could have access to them. And so I worked on this for five or six years together with Léon Bottou who's, had been a- a long, long-term, uh, collaborator. Yoshua was also involved, uh, peripherally and a bunch of other people, Patrick Heffner, et cetera. And- and that project ended when all of us basically left, uh, AT&T, that's when I restarted working on- on deep learning and, uh, Geoff also kind of came back to- to Canada, he had been in the UK for a while. And Yoshua, Geoff, and I decided in the early 2000 to basically start a conspiracy (laughs) to, uh, you know, revive the interests of the community in- in neural nets, um, by making them work, discor- discovering new- new algorithms.... and, you know, it took almost 10 years but it succeeded be- beyond our wildest dreams basically (laughs) .

   11. HSHarry Stebbings6:11

       I, uh, so I'm gonna ask you a range of varying questions in terms of depth, breadth, and kind of obvious and non-obvious, so forgive me if some are obvious. I just wanna ask, when I hear the historical context there from you over

3. 6:25 – 12:31

   The Rapid Progress of AI Today

   1. HSHarry Stebbings6:25

      many decades, how do you feel today when we look at what's happening today? Are we at a new inflection point in development or is this merely the continuation of what we've seen for many decades?

   2. YLYann LeCun6:39

      Um, it's a combination of the two. So on the one hand, a lot of what we see today when, when you are kind of down in the trenches of, of research, looks at a logical extension. I was not as enthralled by the sort of, uh, recent progress as the, the, the public was because, you know, I've seen this progress happen over the last several years. Now there are things that have, have been very surprising. The fact that self-supervised learning methods applied to transformer architectures work amazingly well, and they work, you know, way beyond what we could have expected. The fact that we can do basically train systems to understand language, translate language in multiple languages, and then, you know, continue text if you, if you train them to do this or answer questions if you train them to do this, works amazingly well to an extent that, you know, people didn't quite expect-

   3. HSHarry Stebbings7:29

      Mm-hmm.

   4. YLYann LeCun7:30

      ... that, uh, you know, was gonna happen by just making them bigger and training them on more data. So that certainly has been surprising for everybody. But that revolution occurred two years ago, right? So whereas the, the wider public, you know, has learned about it through, uh, ChatGPT, that, you know, was made available, for us, you know, it's, it's been more continuous. And you see this in, you know, a lot of marking events in, in technological progress or in AI in particular are marked by kind of splashy events that the public pays attention to. But to many of us, like, it looks like more like a continuous thing and, and generally what it, what those progress require is a bunch of people to take the techniques that already exist, push them a little further, do a bit of engineering, and then make a, a demo that demonstrate that it works. So that was the case for, you know, Deep Blue, the chess player that IBM built, uh, in the, in the mid '90s, that, that beat, you know, Garry Kasparov. You know, same thing with the DARPA Grand Challenge that Sebastian Thrun team at, uh, Stanford, uh, won. A car that could drive itself in the desert, right, for 100 miles. And then, you know, AlphaGo and, and, you know, the IBM Jeopardy. This, there's a number of those things, right? And, you know, with ChatGPT just being the, the latest one. And it looks like kind of jumps when you look at it from, from far away, but when you're in the field it's more like a continuous evolution.

   5. HSHarry Stebbings8:52

      Can I ask, has there been any other surprising, on the positive side, developmental things you've seen over the last year or so? You said about self-supervised learning and the efficiencies there. Is there anything else where you're like, "I didn't expect it to go as well as it has done in the last year"?

   6. YLYann LeCun9:07

      Yeah. So I already mentioned it. Uh, you know, the, the fact that merely, uh, training a language model to predict the n- the, the, the last word in a sequence of, of words, if you do it properly, you get a system that has capabilities that are somewhat unexpected and they, um, emerge a- as, as you make those systems, uh, bigger and you train them on larger amounts of data. That's, that's clearly, um, clearly been a surprise for everyone. Uh, now the thing is, you know, as, uh, researchers and scientists, we, we're always looking for the next, the next thing. So what I'm interested in at the moment is, you know, what goes beyond that? Like, you know, a lot of people are gonna work on applications of autoregressive large-language models, which is great. There's gonna be a lot of, uh, you know, products and, and new ways for people to do things and it's gonna be wonderful. But I'm already, you know, I've already been thinking about the next, the next stage for the last, uh, three- three/four years, four/five years even. Actually more. Um, which is, like, what's missing from those systems?

   7. HSHarry Stebbings10:10

      What are your thoughts on what's missing from those systems? In, in that logical next step, where does that lead you in your thinking?

   8. YLYann LeCun10:16

      So those systems do not have anywhere close to human-level intelligence. Okay?

   9. HSHarry Stebbings10:21

      Hm.

   10. YLYann LeCun10:21

       Despite what you might think. We are kind of fooled into thinking it because those systems are very fluent with language, but their ability to, to think, to understand how the world works, to plan are very, very limited, and their understanding of the world is very superficial. And the reason for it is that they are strictly trained on language and language only contains a small, uh, proportion of all, all human knowledge. Most of human knowledge is not linguistic at all, and all of animal knowledge is non-linguistic and we take it for granted. You know, this is the, the Moravec paradox, right? All the capabilities and abilities that we take for granted, like, you know, planning a motion or something or very simple things that everyone can, can do. Uh, a 10-year-old can, you know, clear up the dinner table and fill up the dishwasher. Any, uh, 17-year-old can learn to drive.

   11. HSHarry Stebbings11:14

       If, if-

   12. YLYann LeCun11:14

       We still don't have self-driving cars. We don't have domestic robots.

   13. HSHarry Stebbings11:16

       If they're non-linguistic, like, the, the majority, I'm sorry for the base questions, but then what are they and, uh, is that, that we don't have able to be ingested by AI models and engines over time?

   14. YLYann LeCun11:31

       Well, so first of all there's no question that eventually AI systems will understand the world in similar ways that, that humans do, uh, perhaps better ways. Uh, but they will not be autoregressive large-language models of the type that we're now, uh, talking about. They will be different, uh, for a number of different, different reasons. But, but to answer your question more directly, anything that has to do with sort of an intuition of the real world requires an experience of the real world or, or a simulated version of it, uh, which, uh, those large-language models don't have. They're purely trained from text. So you can a- you-... there's a number of questions that, th- about the physical world that they'll be able to answer because there's a, there's a template for it in the... or something very similar in the data that they've been trained on. Same for planning. You can ask them to, you know, plan a trip or something and they will adapt a, a template that they've, they've been trained on. But they don't really have sort of a model of, a mental model of how the world works that allows them to plan complex action sequences or, or use

4. 12:31 – 21:19

   Prophecies of Doom: Debunking AI Misconceptions

   1. YLYann LeCun12:31

      tools or things like that.

   2. HSHarry Stebbings12:32

      Can I ask, is that why you said that AI researchers facepalm when they hear prophecies of doom?

   3. YLYann LeCun12:38

      No. That's a different question. Those are kinda orthogonal concepts, so, I mean, there is some, some weak connection. There is a, a flaw in, uh, current auto-regressive LMs which is that you can only control their answer in two ways. The first way is you modify the statistics of the training data that you train them on, possibly using, uh, human feedback for, you know, specific answers, and the second one is you change the prompt. And the combination of the prompt that, you know, the question you, you ask them, the form in which you, you ask the question, and those statistics of the training data, entirely determines the answer that the system will produce.

   4. HSHarry Stebbings13:17

      Huh.

   5. YLYann LeCun13:18

      So there is no persistent memory, first of all. But second of all, you cannot control the system. You cannot impose constraints on it, like be factual, be understandable by a 13-year-old. You can try to put this in a prompt but then, you know, you rely on whether those statistics of the training data is appropriate for, for taking, taking that into account. There's no direct way to constrain the answer of those systems to satisfy certain objectives and that makes them very difficult to, to control and steer. And so that creates some fears because people are kind of extrapolating, if we let those systems do whatever, we connect them to internet and they can do whatever they want, they're gonna do crazy things and stupid things and perhaps dangerous things, and we're not gonna be able to control them and this, they're going to escape our control and they're gonna become intelligent just because they're bigger, right? And that's nonsense. First of all because this is not the type of system that we are going to give agency to.

   6. HSHarry Stebbings14:15

      S- (laughs)

   7. YLYann LeCun14:15

      The systems that will eventually be given agency, that are going to be able to plan sequences of actions, are systems that are gonna have objectives that they're gonna have to satisfy. And because of those objectives, they're gonna be controllable, so they're gonna be much more controllable than the current systems. Okay? So my prediction is that within a few years, nobody in their right mind would use auto-regressive LMs. They'll go away in favor of something more sophisticated and controllable that can plan its answer as opposed to just produce one word after the other, uh, reactively. Okay. That's, that's the first fallacy. The second fallacy is that there is this idea somehow that the desire to, and the ability to dominate is linked with intelligence, right? So this is a statement that a lot of people are, are making including, you know, my friend Geoff Hinton recently, that somehow as soon as a machine becomes intelligent, it becomes uncontrollable because, you know, it's, it being smarter than us, it can influence us in ways that we can't even, uh, imagine. Now I think this is a gigantic fallacy, uh, because even within the human species, it is not the smartest among us that want to dominate the others. Okay? To dominate other entities, you don't necessarily need to be smarter than them, but you need to want to dominate them. This is not something that every intelligent d- entity is going, is going to do spontaneously. We do it as humans because the desire to influence others was built into us by evolution 'cause we are a social species. Okay? Same as baboons and chimpanzees and wolves and dogs and et cetera. It's not the case for orangutans. Orangutans don't have the desire to dominate anybody because they are non-social animals, they are solitary animals, they're territorial, in fact. So we need to separate this, uh, those two concepts, the, the, the will, the desire, and the ability to dominate on one hand, and intelligence on the other hand. The fact that we're g- gonna have super intelligent machines at our disposal means that every one of us is gonna be like a business leader, politician, or an academic with a staff of people working for them that are more intelligent than themselves. I mean, it's great. It's not... Like, if you feel threatened by, you know, being the boss of other people who work, work with you but are smarter than you, you're not being a good leader. (laughs)

   8. HSHarry Stebbings16:43

      (laughs) Can I, can I ask then, how do we instill values within models where they don't have a desire to dominate?

   9. YLYann LeCun16:50

      Right. So these objectives that we're, we're telling you about... So, okay, so let me describe the, the sort of architecture of future AI systems as I see it. We're gonna have AI systems that basically are going to plan their actions, and actions can include sequences of words that you tell someone, but they're gonna plan those sequence of actions or, or words so as to optimize a series of objectives that we set them.

   10. HSHarry Stebbings17:15

       Yeah.

   11. YLYann LeCun17:15

       Okay? So one objective is, uh, does this answer the question I just asked? Okay. Another objective might be, when you're talking to a 13-year-old, make that answer understandable by a 13-year-old. Another objective might be, you know, I ask you to answer, uh, a question about the world, so be factual. Or it's a question about, you know, yesterday's political event, you know, can you kinda be compatible with everything you've read in the press, uh, this morning, uh, things like that, right? Uh, so you, you'll, you'll have those systems that have, you know, a series of objectives and their output, their answer, by construction, is going to have to satisfy those objectives. And some of those objectives will be hardwired to make those systems safe. Like, if it's a domestic robot that can, you know, cook, uh, cook dinner and can wield, you know, kitchen knife in its, uh, in its arm, there's gonna be a term in there that says, like, stop moving your arm when there is people around 'cause, you know, you might hurt them. So that's gonna be an objective that the system cannot violate because, by construction, it's gonna have to satisfy them. So that's the way to build safe AI system-... you make them produce answers that by construction have to satisfy your objectives, and you design those objectives so that their actions are safe. Now how precisely to do this is not a completely solved question, but you try it, you deploy it at a small scale, you see what the effect is, and you correct it when it doesn't work, and you, you fix it progressively. And it's not like if, if you get it wrong, it's going to destroy humanity. (laughs)

   12. HSHarry Stebbings18:49

       (laughs) How... (laughs) Depen- depends on that cooking robot. You never know. Um, how do you determine who's able to set the objectives? 'Cause that could be right or wrong depending on who sets them.

   13. YLYann LeCun19:00

       That's true. So that's gonna have to be, uh, there's gonna have to be a process by which, you know, we, we allow people to do this, so some vetting process. You know, the same way that, you know, there's a vetting process for, you know, people to take care of, uh, take care of your health or cut your hair, fix your plumbing, or your car, right? Um, so there's some, you know, some vetting process, certainly some testing and, you know, market deployment procedure with regulating agencies for things that have, you know, that are potentially, uh, dangerous, probably not for all applications, but for many applications, certainly in healthcare, transportation, and things like that. And then perhaps also, it could be that, um, you know, let, let's take the example of, of intelligent assistants. So let's imagine a future where everyone can, you know, talk to, to their intelligent assistant. That sy- that system will have pretty close to human level intelligence, probably more accumulated knowledge than most, most humans. You know, they could translate in any language and prob- you know, give you a quick summary of, you know, yesterday's newspaper and things like that, right? Explain mathematical concepts to you, things like that. So people are probably going to use this almost exclusively in the future for their interaction with the digital world. You know, you're not gonna go to Google or Wikipedia. You're just gonna talk to your assistant. And the only way to do this properly is for the base infrastructure for those assistant, I mean, they, they would be so pervasive, so much will ride on, on those systems that I don't think anyone will accept that those assistant being behind a event horizon in a private company. They will insist that the infrastructure is open. They will insist also that the vetting process by which those systems are, are trained be something maybe like Wikipedia, right? We tend to trust Wikipedia sometimes with a grain of salt, but we tend to trust Wikipedia because there is a vetting process so that whenever a- an article is modified, you know, some editor kind of check on it, and, and then the changes are accepted or not, things like that. So you can imagine that the sort of common repository of all human knowledge that will be our assistants will be constructed through some sort of crowdsourcing process perhaps similar to Wikipedia, where you're gonna have a bunch of people training those systems and fine-tuning them so that, you know, whatever they... and so they produce

5. 21:19 – 25:01

   Open vs Closed-Models of AI; Where does the value go?

   1. YLYann LeCun21:19

      are, are correct.

   2. HSHarry Stebbings21:20

      It's so- it's so funny you say about that kind of the benefits there of the, the open a- approach over the closed approach 'cause I've, uh... And that's where I've been kind of stuck, which is like, where does value accrue? Is it to the closed model or the open model? And then we have the leaked internal memo stay from the Google employee who said, "You know, we're not ahead. OpenAI are not ahead. There's this third, uh, being which is actually far more significant, and we haven't taken notice of," um, summarized.

   3. YLYann LeCun21:46

      Right.

   4. HSHarry Stebbings21:46

      How do you-

   5. YLYann LeCun21:46

      And that was triggered, that was triggered by, by LLaMA, which is the, uh, the model, uh, that was put together by my, uh, my colleagues, uh, at FAIR, which was, uh, the code was open sourced. The model, sadly, uh, was distributed only for research and non-commercial purpose. Uh, and the, the reason for that is, is basically complicated legal issues of what's, what's the status of the data that those system have been trained on and, and things like that. So it's more kind of... It's not a lack of desire from the, from Meta to open source. Uh, it's, it's more kind of complex legal issues that go beyond my, uh-

   6. HSHarry Stebbings22:27

      I'm, I'm super, I'm super naive, Yann. Why does open win against a more controlled, tight-knit, well-funded OpenAI or other large corporate with a big balance sheet and a very rigorous but streamlined team?

   7. YLYann LeCun22:43

      It's very simple. It's because no outfit as powerful as they may be has a monopoly on good ideas. So if you do it in the open, you basically recruit the entire world's intelligence to contribute to things and, and having ideas, and ideas that you may not have, you know, thought about, which, you know, an outfit with 400 people has no chance, uh, thinking about, or even a large company with 50,000 employees may not want to devote any resource, uh, resources to because they may not think it's, uh, useful in the long term or, or they have, you know, more urgency to take care of. So you give it, you give it away, and then you have, you know, tons and tons of people, some of whom are, you know, undergraduate students or people, you know, you know, in their parents' basement, so coming up with amazing ideas that you would never have thought about or willing to spend the time to crunch down the, you know, s- 7 billion weight LLaMA so that it runs, uh, on a Mac, on a laptop. Like, ah, that's pretty amazing. So I, I think that's why, you know, open source, uh, projects succeed particularly when they concern basic infrastructure. So if you think about it, the, the early days of the internet, there was a, a battle between Microsoft and Sun Microsystems to provide the basic infrastructure for the internet, uh, you know, the operating system, the web server, you know, things like that, right? Um, so on Sun Microsystems, it was Solaris and, you know, whatever web server and Java, and then on the, on the Microsoft side, it was Windows with IIT or whatever, you know, and ASP, which was their kind of, uh, uh, server and client-side, uh, protocol.... both of them lost. In fact, Sun Microsystems pretty much went bankrupt and was, you know, sold for parts to Oracle. One was Linux and Apache, which is completely open source. And, uh, you might, you might ask why. You know, the entire internet and the entire tech industry runs on Linux, right? And your phone probably runs on Linux too if you have Android.

   8. HSHarry Stebbings24:44

      (laughs)

   9. YLYann LeCun24:44

      So that's three-quarters of the phones in, uh, you know, in the world. Uh, so the reason for this is that, you know, it's just a much better way of gathering competence and talent around a common project, even if it's not motivated necessarily by, by profit.

   10. HSHarry Stebbings25:00

       Yann,

6. 25:01 – 29:50

   How does Meta win the AI race?

   1. HSHarry Stebbings25:01

      I, I, I agree and I, I love this. You work with Meta. My question, and David Marcus's question, was how does Meta win then?

   2. YLYann LeCun25:10

      So it's been the case that Meta in the past has open sourced pretty much everybody everything that it's, it's ever produced (laughs) , uh, in terms of basic infrastructure, right? So you have, you know, React for, uh, you know, the framework for web and mobile apps. You have PyTorch. PyTorch is not even owned by Meta anymore. The ownership was transferred to the Linux Foundation-

   3. HSHarry Stebbings25:34

      Huh.

   4. YLYann LeCun25:34

      ... because it's so essential, um, to the, you know, AI R&D infrastructure nowadays. You know, ChatGPT was developed on PyTorch, okay? All OpenAI runs on PyTorch. Uh, the entire world, in fact, runs on PyTorch, except Google. (laughs)

   5. HSHarry Stebbings25:46

      (laughs)

   6. YLYann LeCun25:47

      Uh, because they have their own, their own thing, right? But it goes beyond that, right? Uh, Meta open sources its hardware server backplane design so that hardware manufacturers can, can build to its, uh, specifications. And pretty much everything, aside from sort of, uh, legal issues that are sometimes due to kind of recent laws or, or, or court decisions, uh, pretty much everything is, uh, has been open sourced. It is not because other people can use your technology that you can't exploit it to the same extent, right? Who can use smart NLP systems for, you know, translation or content moderation on Facebook other than Facebook? (laughs) It doesn't matter if other people have access to the same technology.

   7. HSHarry Stebbings26:31

      I mean, it's, I, I totally agree with you. And this kind of led to my next question, which you actually tweeted about, which comes to the size of, like, data moats and size of data availability. Is it simply a case that the largest model wins? And how do you think about value in small models as well?

   8. YLYann LeCun26:48

      Yeah, so it's not the case. The, uh... And this is really what, uh, LLaMA has demonstrated and, and really kind of shown people. So the people behind LLaMA, Edouard Grave and Guillaume Lampe and then their collaborators, uh, mostly in, at FAIR Paris, actually, ma- many of them are in Paris, they demonstrated that you don't need those models to be very large to, to work really well. I think it, it, it caused a bit of a epiphany for a lot of people, realizing, oh, you know, you don't need... Okay, maybe you need a thousand GPUs, you know, running for 10, you know, a couple weeks to train it, the base system. Uh, in fact, this, that number is going down too (laughs) because people are configuring how to do this more efficiently. But once it's pre-trained, you can use it for all kinds of stuff and you can fine-tune it really easily. Uh, and, uh, and then at the end you can run it on your laptop, right? That's kind of amazing. Or maybe on a, on a, you know, desk, desktop machine with, uh, a GPU in it or a couple GPUs. So, uh, I, I think, you know, it sort of opened the minds of people to the fact that there is, like, enormous opportunities that really weren't thought to be possible before. And I think it's gonna make even more progress because if we go towards the design of AI systems perhaps along the lines of what I described with, uh, objectives and planning-

   9. HSHarry Stebbings28:03

      Yeah.

   10. YLYann LeCun28:03

       ... uh, I think those systems could actually be even smaller, uh, to some extent.

   11. HSHarry Stebbings28:08

       How would they be even smaller? Sorry, unpack that for me.

   12. YLYann LeCun28:11

       Well, because the, the current models, you... For them to work, you have to train them on gigantic amounts of data.

   13. HSHarry Stebbings28:17

       Yep.

   14. YLYann LeCun28:17

       Way more data than any human's ever been trained on, right? So the, the amount of data LLaMA is trained on, for example, is something like, uh, 1.4 trillion, uh, tokens, which is a, you know, it's like a quarter of the internet or something. (laughs) Uh, it's something absolutely enormous. It would take someone reading eight hours a day at normal speed about 22,000 years, um, to read through that.

   15. HSHarry Stebbings28:41

       (laughs)

   16. YLYann LeCun28:41

       Okay? So obviously those systems can accumulate a lot of knowledge from text, but they don't do it the same way humans do it, because we don't need that much time to be that smart and to learn, uh, that much. So obviously, we are much more efficient. Our brains are much more efficient than those models at learning things. Like, how is it that a teenager can learn to drive a car in about 20 hours of practice? We still don't have Level 5 self-driving cars. So obviously, we're missing something really big. And what we're missing, I think, is abilities for, for AI systems to learn how the world works by observation mostly, and then this ability to plan so as to satisfy your objectives. And then beyond that, the ability to set sub-objectives that in the satisfaction of a bigger one. Okay, that's called hierarchical planning.

   17. HSHarry Stebbings29:28

       That's-

   18. YLYann LeCun29:29

       And, and we do this, humans do this. Some animals do this to some extent. Every animal is... You know, mammal and bird is capable of, of some level of planning. Ultra-rigressive LLMs basically don't do planning, or a very simple form of it.

   19. HSHarry Stebbings29:41

       Yann, you mentioned the efficiency that can come from actually smaller models than expected, and how actually size, uh, of models isn't everything. Uh, the other thing,

7. 29:50 – 36:41

   Incumbents vs Startups: Profiting in the AI Era

   1. HSHarry Stebbings29:50

      we spoke about open and closed. The other thing that I've been thinking and everyone's been thinking about, and I've interviewed many kind of leading AI experts and they say the value will accrue to the incumbents. Startups, they don't have the data, they don't have the models, it'll accrue to the incumbents. Is that right? Will the value accrue to the incumbents or do you believe that given what you just said about size not being everything in terms of models, it could be startups as well?

   2. YLYann LeCun30:16

      So it depends on which scenario you, you believe in. So the scenario, um, I think will happen and I'm certainly rooting for is the scenario I described earlier where you have some sort of open, uh, platform for base LLMs. So base LLMs basically would be seen as a basic infrastructure.... uh, like, you know, TCPIP Linux Apache essentially.

   3. HSHarry Stebbings30:38

      Uh-huh.

   4. YLYann LeCun30:38

      Um, completely open. And then there will be an ecosystem of companies building stuff on top of it, which for vertical applications, for specific things, right? To specialize those systems for particular application, to offer support, to make it, you know, customized for, for enterprise applications, for personal things. I mean, there's, there'll be, like, a whole economy around this, which will create jobs, by the way, not make them disappear. So this is the se- the scenario that I believe will happen. And the reason I think it will happen is because there is, uh, essentially a need to use essentially millions of contributions for, for making those systems, uh, factual and correct and et cetera. So W- Wikipedia style. So I think the proprie- proprietary approaches will actually fall behind. So that's one point, okay? The second point is, you can ask yourself the question, how is it that the companies that were best positioned to produce something that ChatGPT, namely Google and Meta-

   5. HSHarry Stebbings31:33

      Mm-hmm.

   6. YLYann LeCun31:34

      ... didn't? Why is it OpenAI? Okay. A small outfit with, you know, 400 people. I mean, more now, but pretty small outfit. And the answer is, it's not because Google or, or, or Meta did not have the competence or the technology. It's just that they didn't have the, the pressure to produce new products, completely new products that had a lot of risk attached to them, and the risk were... And we know what the risks are, because a few weeks before Ja- Chi- ChatGPT, my colleagues at, at, uh, at FAIR produced an large language model called Galactica, which was an experimental system, and they put out a demo, and the, the demo was to demonstrate that... So Galactica was a, a large language model trained to train on the entirety of the scientific literature and it was basically designed to help scientists write papers. So you would start writing a paragraph or, or, or something like that to describe the topic of the paragraph, and then Galactica would basically complete the paragraph and it wouldn't, it wouldn't be factually correct. You would have to kind of fix it, but it would like, you would ask it to, like, build a table of result and it would just, you know, put the LaTe commands to kind of build the thing and populate it with the known results on the literature about the topic that you're working on, or you would type a chemical formula for something and it would, you know, turn it into an actual name for it or things of that type, right? Very useful for scientists. As soon as the demo was put out, it was murdered by the social network Twitter sphere.

   7. HSHarry Stebbings33:05

      Why?

   8. YLYann LeCun33:06

      People said, "Oh, this is going to, you know, destroy scientific publication because now, you know, any random person can write a authoritatively sounding scientific paper that is, you know, nonsense." And there was so much vitriol thrown at the, at the system that the, the people at, at Meta who, who built it, like, couldn't take it. They, they took ou- they took down the demo because they said like, "We, we can't sleep at night." So here is an example of a very useful system, a system that could have been extremely useful, uh, particularly for writers of scientific papers who are not native English speakers, that basically was destroyed by AI doomers, people who just did not think about the risk-benefit analysis. The risk of flooding the literature with nonsense is ridiculous. I mean, uh, because, you know, the scientific publications are vetted and, and things like that. So, um, there was not a, a significant danger. Um, and then ChatGPT came two weeks later and was welcomed as the second coming of the Messiah, right? Um, so what does that tell you? Uh, oh, and then, you know, a few months later, uh, Google came out with, uh, uh, Bard and, and in the demo, Bard made a, a tiny, you know, minor factual mistake about some astronomical fact, and, you know, Google's, uh, uh, stock went down by 8%. Now, what that tells you is that when something is produced by a large company that has, uh, a reputation, particularly a reputation to defend, they can put out things that's pure nonsense, but it's okay for a small company. So that's the landscape of what happens now, which, which is why I think there's a bit of a paradox, which is that the companies that have, you know, the, the best technology basically can't... have difficulties putting it out because of those legal issues and, uh, sort of public image.

   9. HSHarry Stebbings35:00

      Uh, do you not also think there's this core business model challenge though, which is, it's the classic innovator's dilemma. Like, why didn't Google do this? 'Cause it would have killed that absolute cash cow of Google Ads, the cost to service a query versus the costs of this is so significantly different. You'd be killing your core cash cow with this, with unknown upside, versus retaining what is a great business.

   10. YLYann LeCun35:26

       You don't have a choice. I mean, there's no question that, you know, within some time, you know, it could take a, it could take a while, but there is no question that people will interact mostly with the digital world using, uh, AI assistance. And, you know, they may run into your, your augmented reality glasses, okay? So (laughs) , uh, or, or something of that type, like, you know, you know, like in the Spike Jonze movie, uh, Her, that's, uh-

   11. HSHarry Stebbings35:52

       Yeah.

   12. YLYann LeCun35:52

       ... that's not, not a bad depiction of what, you know, the, the way things could develop. And so if you, if you take the assumption, make the assumption this is gonna happen, you, you, you have to build it as qui- as quickly as you can. And it might cannibalize your, you know, your newsfeed algorithm or, or whatever, or in the case of Google, your, your search engine. But you have to do it. You know, it's like, um, uh, I mean, Meta has been known to make those choices, uh, in the past, like the move to, to mobile, for example, um, and the, the move to, uh, you know, short, uh, short form video, for example, you know, which, you know, obviously TikTok has been, uh, very successful at. Uh, Meta has entered that, that, that business in kind of a, a big way, despite the fact that the amount of revenue it derives from it is lower than a traditional newsfeed.... 'cause it's hard to-

   13. HSHarry Stebbings36:40

       So-

   14. YLYann LeCun36:40

       ... put, you know,

8. 36:41 – 43:36

   AI Will Create More Jobs Than It Destroys

   1. YLYann LeCun36:41

      put ads in videos, basically. (laughs)

   2. HSHarry Stebbings36:42

      (laughs) Uh, uh, you mentioned the job creation element there. I do just wanna touch on the job side, 'cause it's, uh, it's the classic AI doom, uh, that s- we're all gonna be unemployed, and we're gonna have universal basic income in an optimistic world.

   3. YLYann LeCun36:55

      With hope.

   4. HSHarry Stebbings36:55

      You said about job creation there. We didn't hear about job creation through AI. How do you see what jobs will be created through this new ecosystem and what that world of employment could look like?

   5. YLYann LeCun37:08

      So, 100 years ago or maybe 120 years ago, uh, most people in most of the world worked, uh, in the fields, in, uh, food production. Um, that was pretty much a majority of the population. Uh, today, in developed country- countries is between 1 and 2%. Um, and that has caused, you know, a migration of people into the cities and, uh, you know, the, uh, development of, uh, service, business, uh ... You know, the same thing 20 years ago or, you know, 20, 30 years ago, there was a, a big movement towards automation of manufacturing, and a lot of manufacturing jobs disappeared in third world countries, but they were replaced by other things. So 20 years ago, like who would have thought that you could make a living with a podcast? (laughs)

   6. HSHarry Stebbings37:58

      (laughs)

   7. YLYann LeCun37:59

      Or YouTube.

   8. HSHarry Stebbings37:59

      I just, I, I didn't think I could five years ago, Yann. I'm as surprised as everyone else.

   9. YLYann LeCun38:03

      (laughs)

   10. HSHarry Stebbings38:04

       (laughs)

   11. YLYann LeCun38:04

       Right. Uh, so, you know, a lot of jobs appear, like, you know, uh, 30 years ago, there was no such thing as, as web designer, and now it's, you know, we have engi- engineers in the world basically do this, right? So, you know, the number of economists that I have talked to, which is pretty large, about ... where I ask that question, who tell me, "Well, we're gonna run out of jobs because, you know, we're all gonna be replaced by IT," is exactly zero. Like, no economist believes this. No economist believes we're gonna run out of job because no economist believes that we're gonna ru- run out of problems to solve or requirement for human creativity and, and human communication and stuff like that. So, you know, this is gonna create as many jobs as it makes disappear. Now, the question is, though ... And, and those jobs, by the way, are gonna be more productive. So overall, technology makes people more productive. In other words, for the same amount of hours worked, you produce more wealth, okay?

   12. HSHarry Stebbings38:56

       Uh-huh.

   13. YLYann LeCun38:57

       Uh, but every, uh, technological revolution, unless it's accompanied by sort of, you know, political changes and s- and social changes, generally profit a small number of people, at least temporarily, right? That, that happened in the Industrial Revolution in the late 19th century where, you know, a few people became extremely rich, and then other people were exploited, and then, you know, society changed, and there were, like, social programs and, and, you know, income tax and, and high tax for richer people and stuff like that, which still the US has backpedaled on this, but not Europe. Uh, or the UK to some extent too, but not, not the rest of Europe. Um, so there is a, a question of, you know, how you distribute the wealth, if you want, okay? How do you organize society so everyone profits from it? But that's a political question. There's no technology que- question, and it's not new. It's not caused by AI. It's just caused by technological evolution, right? It's not a recent, uh, a recent phenomenon.

   14. HSHarry Stebbings39:51

       This is so unfair of me to ask, but what do those jobs look like? Like, w- what are they? Are they cr- uh, they're creative-oriented, but what does that actually mean? Like, sorry, I know that's a really hard question, but I'm just trying to understand how, how we actually spend our time. And my children, which I don't have, by the way, Yann, but-

   15. YLYann LeCun40:07

       (laughs)

   16. HSHarry Stebbings40:07

       ... what, what, what do they do?

   17. YLYann LeCun40:09

       (laughs)

   18. HSHarry Stebbings40:09

       Like, sculpt or paint? I do- I don't know.

   19. YLYann LeCun40:13

       I don't know. That's a good question. Uh, but it's not because I don't know that it won't happen.

   20. HSHarry Stebbings40:18

       (laughs)

   21. YLYann LeCun40:18

       Because, I mean, look at, like how many people exercise their creative juices, uh, today, right? With all the tools that are available that, you know, weren't available, uh, 10, 20, or 30 years ago.

   22. HSHarry Stebbings40:29

       Noth-

   23. YLYann LeCun40:29

       Like 3D artists or something like this, you know, game designers, you know, all kinds of things. Uh, you know, I think creative jobs are the- are the ones ... So there, there, there are two types of jobs that, that, you know, have a bright future. Creative jobs, whether they are scientific, technical, educational, or artistic. ACI has to do with communication, right? And communication of human emotions, which is, you know, intrinsically human, if you want. Uh, so that's one category. And then the other one is personal services, so w- where you need, you know, actual people to interact with you.

   24. HSHarry Stebbings40:59

       I, I totally agree and get you, and I lo- I love that. We shall see. Uh, can I ask, the only thing that I worry about is like the speed of transition. Like when you look at past, you know, Industrial Revolution, when you get even the introduction of PCs into kind of, you know, working environments, these were multi-decade introductions. Bluntly, what AI feels like in some industries today, we use it at the media company, and it's cutting our employ. Like the speed of transition is much, much more compressed in this timeline, which will lead to short-term significant high unemployment. Do you concede that or do you not concede that?

   25. YLYann LeCun41:36

       So this is something I used to be really worried about, that, uh, the, the speed of, uh, progress of technology was gonna leave a certain number of people behind who, you know, cannot be basically retrained fast enough or be- or maybe they are too old to retrain themselves for the new, uh, the new world. I was worried about this. And then I talked to a bunch of economists, and they say, "Oh, you know, not really, because the speed at which a technology disseminate in the economy is actually limited by how fast people can learn to use it." Um, so, uh, a good person to talk to about this is Erik Brynjolfsson at, at Stanford, and what he says is that when a new technology is introduced, let's say the, the PC, right, uh, with, you know, a graphical user interface, a mouse, et cetera, right, in the mid '90s, how long did it take to have a measurable effect on, uh, productivity, you know, which is, uh, an amount of wealth produced by per hour worked?

   26. HSHarry Stebbings42:33

       Yeah.

   27. YLYann LeCun42:33

       And he says, you know, typically it's 15, 20 years, and the reason is that that's what it takes for people to learn to use that new technology basically.

   28. HSHarry Stebbings42:40

       But you, but you buy that here, like, people are, are pretty good at prompts. You know, social media content managers are using prompts very efficiently to produce content plans, to create content ideas in under half an hour after watching a couple of TikToks.

   29. YLYann LeCun42:54

       Yeah. But like, what is gonna be the effect of this on, uh, first of all, on measurable productivity? Second of all, on the, the job market? Like, is it gonna make people lose their job, like, right away? And no, it's gonna take a while, it's gonna take 10, 15 years, you know, possibly more. Right, it depends when you start counting, right? Because the AI revolution maybe started 10 years ago. So if you start counting then, then it might only take, you know, another 10 years. But, you know, uh, I mean, I don't think you want to underestimate the degree of conservativeness of, of the business world, right? I mean, things (laughs) tend to change not that quickly.

   30. HSHarry Stebbings43:29

       Can I ask-
9. 43:36 – 45:40

   Why Humans Love AI Doom Scenarios

   1. YLYann LeCun43:36

      be, or become more productive themselves.

   2. HSHarry Stebbings43:38

      Uh, why do you think we love the doom, Yann? Your, y- you know, I, I love your approach and mindset and I agree with it. But why do you think we are kind of magneticized to, like, oh, we're all gonna be unemployed in the doom?

   3. YLYann LeCun43:51

      Well, because I think for a number of reasons. So I'm not a, you know, a social psychologist (laughs) or a sociologist, but, uh, but clearly, I think we're hardwired to pay attention to things that occur or may occur that could be dangerous to us, because it means that there is something about the world that we don't completely understand, and we do have to pay attention to it and be careful about it. So for example, take a young, a young child, five month old, and show a scenario to this small child of a little car that is sitting on a platform, and then you push the car off the platform, and instead of falling, the, the car appears to float in the air. The five months old will barely pay attention to it. But if you show this to a, a 10 months old, the 10 months old will look at it with huge eyes and stare at it for a long time, wondering what's going on. Because in the meantime, babies around the age of, you know, between, between six and, and nine months learn about gravity. They learn that objects that are not supported are supposed to fall. And so their mental model is that an object is not supported should fall, and they see this object that appears to float in the air and they say like, "This can't be." It's like, "You know, there's something I didn't, I didn't, I don't understand about the world. I need to look at this and investigate." Okay, so we, we're hardwired for this because that's the way we learn the, our, our internal mental model of the world that allows us to predict what's gonna happen, allows us to plan. Uh, that's what makes us smart. That's the basis of intelligence, the a- the ability to predict. And so we naturally pay attention to stuff that are, that is surprising or dangerous, or both. Which is why, you know, you see a outrageous piece of news, you know, a clickbait at the bottom of some, uh, you know, website and, like, y- y- y- y- you have to convince yourself not to click on it. (laughs)

   4. HSHarry Stebbings45:36

      Can I ask you a couple of, uh, direct questions? I'm just too interested,

10. 45:40 – 51:38

    Jeff Dean's Exit from Google & His AI Warning

    1. HSHarry Stebbings45:40

       and we can take them out if, if needed. Um, what did you say to Jeff when you heard that he was obviously making the moves that he did? Did, I'm sure you had a conversation with him. What did you say to him?

    2. YLYann LeCun45:51

       We haven't spoken yet, actually. We're going to speak to kinda get, you know, each other's opinion on it. Uh, I don't think he knows my, uh, my, my opinion on this 'cause I don't think he follows, you know, what I post on Twitter or whatever, even though he is on Twitter himself. But, and so I think we have, you know, a discussion to have. I've had this discussion before with Yoshua Bengio, um, but not with Jeff. And, and to me, the fact that he left Google is not particularly a surprise. The fact that he leaves Google to be able to speak his mind I think is not surprising. So I have a very different view at Meta, which is that I say whatever I want, (laughs) okay? I'm not under the, uh, tight control of, uh, you know, the co- communications department or, or, or anything. I just, I just say what I think, all right? Um, but-

    3. HSHarry Stebbings46:45

       How did y- how did you get that deal, Yann? Like, no one has-

    4. YLYann LeCun46:47

       (laughs)

    5. HSHarry Stebbings46:48

       No, seriously. I, I'm, many of my friends at, at Meta in very high positions, as you know, we have mutual friends, they don't have that deal. (laughs)

    6. YLYann LeCun46:57

       So they, there is, I mean, I'm in a particularly, uh, sweet spot because I have a, quite a bit of a, of a following, people who trust me or, or believe me or, or, or want to hear what I have to say even if they don't, don't trust me at all.

    7. HSHarry Stebbings47:13

       (laughs)

    8. YLYann LeCun47:14

       Um, and at the same time, I'm not, I'm not an officer, so I, it's not like, you know, there are things I can't say because of, uh, legal issues of, you know, financial blah, blah, blah, right? I'm a vice president, but I'm just below the, the level where you had to be really, really careful and sort of control, control your message.

    9. HSHarry Stebbings47:31

       (laughs)

    10. YLYann LeCun47:31

        And I think there is a cost-benefit trade-off here of, you know, AI is such a complicated, ev- fast-evolving, uh, issue that you basically, you need someone to be able to, you know, speak freely.

    11. HSHarry Stebbings47:46

        Okay.

    12. YLYann LeCun47:46

        And I, I think Jeff didn't feel like he had that option at, at Google, maybe, uh, you know, for, for various reasons. So I understand why, why he might, he might have wanted to, to leave. But I don't, I don't agree with him at all with the, uh, the whole sort of, you know, probability of, uh, human extinction or, or whatever.

    13. HSHarry Stebbings48:04

        Have you ever felt your role at Meta has impeded your ability to be impartial?

    14. YLYann LeCun48:10

        I don't believe so, no.

    15. HSHarry Stebbings48:11

        Right.

    16. YLYann LeCun48:12

        I mean, there are certain things that I would, uh, post on social media that are kind of, you know, kind of propping up the work of my, my colleagues and-

    17. HSHarry Stebbings48:19

        Sure.

    18. YLYann LeCun48:20

        ... you know, I'm obviously biased about this because, you know, I know about the work and they are friends and-

    19. HSHarry Stebbings48:24

        Sure.

    20. YLYann LeCun48:24

        ... or colleagues. And, uh, you know, I think it's interesting probably because, you know, I've, I've followed the politics and-

    21. HSHarry Stebbings48:30

        You work with them. I totally get that.

    22. YLYann LeCun48:31

        So yeah.

    23. HSHarry Stebbings48:32

        Yeah.

    24. YLYann LeCun48:32

        For this kind of stuff, I might be biased. Take this with a grain of salt. You don't have to believe me, you know, things like that. But it's given me a vision also of, you know, how things are built, what the problems are. So, you know, for example, there, there's a narrative, a very, very common narrative...... that AI is the culprit for a lot of the bad side effects of social networks, uh, in the past. And in fact, it's completely backwards. AI is the solution to those problems. (laughs) Um, so, uh, you know, let me, let me tell you, uh, uh, you know, go back like, you know, backpedal 12 years ago or something, you know, even before I joined, uh, Meta, when Meta, you know, started experimenting with the newsfeed. And, uh, and the newsfeed was, you know, an algorithm that would pick, like, which piece of news to show to everyone. And, you know, originally it was decided by, you know, how, how friends are you with the person making the post and things like that, right? How many interactions you have with that person. Eventually, a bit of machine learning was put into it, shortly before I, I joined Meta. It was very, very simple. It was something like logistic regression, something like the simplest method you can imagine, uh, with a lot of engineering behind it and a lot of, you know, hacks by hand and, and special cases. But basically it was something like logistic regression. Uh, you know, some big factor that describes, you know, what you click on, like how many times you, you know, how much time you spent on a particular piece of content and blah, blah, blah. And then it would, it would decide like, you know, give, give a, a rating to everything. So that was deployed and people ended up, you know, spending more time on, on, on Facebook. But then also it created problems that were quickly identified, like, you know, like information bubbles in the context of political, uh, disc-, uh, discourse. And, uh, and the fact that, uh, what I was talking about earlier, that people tend to click on things that is more outrageous, right? (laughs) So it caused the appe- you know, the appearance of, uh, clickbait companies that basically were just like farms of, you know, teenagers in Montenegro or some place, uh, making false news to get people to click on them and, and get money from the ads that they show them. So then, you know, this, this was realized. There were, like, big groups at, at Facebook at the time kind of studying the, what the effect of, of those things are, and this was corrected. So that, that's the way you, you make a system work, right? You, you try it out on a small scale, you see what the effect is. If there is bad side effect, you correct it, and then you sort of compare, you know, two, two systems. And then sometimes something unexpected are, uh, occurs and you have to backpedal and completely change the way you do things. That's what happened in 2017 after the presidential election, American presidential election in 2016. The main newsfeed algorithm was completely changed so that, you know, there was no clickbaits anymore. There was no, like, you know, news outlets that could, like, push their content that was propaganda basically. You know, much more effort to take down false accounts and attempts to corrupt the democratic system and stuff like that, right? So you, you, you correct it. And then what... The, the progress of AI over the last few years basically allowed systems to be deployed to do things like taking, take down hate speech relatively, uh, uh, reliably in

11. 51:38 – 54:40

    Elon Musk Is Wrong About AI

    1. YLYann LeCun51:38

       hundreds of different languages, which was basically impossible to do before.

    2. HSHarry Stebbings51:41

       You mentioned correct it. I promise, last question, then we'll do a quick fire. You mentioned correct it. Elon Musk said on, with Tucker Carlson, "The trouble with AI is you can't release and then correct. Unlike all prior technological developments, once released, it is too powerful to be able to bring back into the box. It cannot be amended in that way." Is that not true?

    3. YLYann LeCun52:02

       That's not true. That's completely false. It makes an assumption which, uh, Elon and perhaps some other people may have, uh, become convinced of by reading, you know, Nick Bostrom's book, Superintelligence, or, or reading, you know, some of Eliezer Yudkowsky's, uh, writing. So this is predicated on an assumption that is just false, which is, uh, the existence of a hard takeoff, right? So the fact that the minute you turn on a superintelligent AI system, it's gonna take over the world, and it's going to escape your control, and it's going to refine itself to be even more intelligent. And so, you know, and the world will be destroyed. Uh, and that's just ridiculous. It's just completely ridiculous because there is no process in the real world that is exponential for very long.

    4. HSHarry Stebbings52:58

       Huh. (laughs)

    5. YLYann LeCun52:59

       Um, you know, those systems would have to, like, recruit all the resources in the world. They would have to be given, uh, you know, limitless power agency. Like, why would we, we do this? And what's more, they would have to be built so that they have a desire to take over. Like, you know, systems are not going to take over just because they are intelligent because again, you know, in, uh, even within the human species, it is not the most intelligent among us that want to dominate others.

    6. HSHarry Stebbings53:28

       So his desire and many other leaders' desire to prevent any further development and to regulate intensity right now and stop all progression is BS, basically.

    7. YLYann LeCun53:39

       It's, uh, obscurantism.

    8. HSHarry Stebbings53:40

       Yeah.

    9. YLYann LeCun53:41

       Right? It's like, it's like people who wanted to stop the printing press and the diffusion of printed books because, you know, if people could read the Bible for themselves, they wouldn't have to talk to priests anymore and then would have their own idea about religion. And that's exactly what happened. People read the Bible for themselves, and that created a Protestant movement in Europe, and that created 200 years of religious conflicts. But it also brought to us the enlightenment, science, rationalism, philosophy, uh, ideas of democracy, and then the, uh, French and American revolutions. And then, you know, you can compare this with the, uh, Ottoman Empire, which for reasons of being able to control their population, you know, basically stopped, forbid the use of the printing press, and it started 300 years of decline. They were dominating science in the Middle Ages, the Muslim world, which is why, you know, every star in the sky has a Arabic name.

    10. HSHarry Stebbings54:35

        (laughs) I love this. I'm gonna do a quick fire round with you now. So I say a short statement, and you give me your immediate

12. 54:40 – 1:06:02

    Quick-Fire Round

    1. HSHarry Stebbings54:40

       thoughts, and then we'll rock and roll. Does that sound okay?

    2. YLYann LeCun54:43

       Sounds good.

    3. HSHarry Stebbings54:43

       So which regions most need to change their modus operandi (laughs) when it comes to the practice of scientific research and incentive mechanisms?

    4. YLYann LeCun54:54

       Which region? (laughs)

    5. HSHarry Stebbings54:55

       None.

    6. YLYann LeCun54:57

       Uh, oh, wow, uh, pretty much every region (laughs) , I'm afraid, but for different reasons.

    7. HSHarry Stebbings55:01

       Fair.

    8. YLYann LeCun55:02

       Uh, so let me start with China. And so China has, uh, a bit of a epidemic of bad science. There's da- a lot of very smart people in China, a lot of very good researchers, a lot of very good work coming out, uh, of China, particularly in AI, particularly in computer vision, uh, but a lot of absolutely terrible work that has to be retracted a few months later it's, it's been published.

    9. HSHarry Stebbings55:24

       (laughs)

    10. YLYann LeCun55:24

        And it's partly because of the incentive mechanisms in the, uh, academic and, and, uh, system, uh, in China. So there's, there's a problem to fix there. I can move to Europe. So in Europe, there are good things. So the education system for, like, undergraduate education in Europe is great. It's fantastic because it's partly free, so that allows, uh, talented people to go to good schools even if they're, uh, if they're not rich, right?

    11. HSHarry Stebbings55:54

        Sure.

    12. YLYann LeCun55:55

        Uh, which is not the case in the US, for example. At least not to the same extent. That's good for Europe. A lot of, you know, European engineers and scientists are, are great, are top, best in the world. But then what are the opportunities for people who want to, you know, go into science and research? And, and there, uh, most of, most European countries actually are... don't have systems that really encourage this and motivate the most talented people and students to get... go into, into science. And so some of them go to North America, like me 35 years ago. There are opportunities now that are really good in research labs like, like FAIR in Paris, or, uh, Google also has, has labs, uh, in Paris. Actually, my brother works at Google in Paris. (laughs) So there are other outfits. So that gives opportunities for, for, you know, people who really wanna be productive and don't think that they can in the sort of public research and academic system, uh, in, uh, in France and the rest of Europe. Um, the only European countries that can rival... country that rival the US in terms of the quality of, uh, job for an academic or a scientist is Switzerland.

    13. HSHarry Stebbings57:03

        Ah. Wh- what do you think they do to rival that?

    14. YLYann LeCun57:07

        Uh-

    15. HSHarry Stebbings57:07

        What is it about their incentive mechanism structure that gives them that ability?

    16. YLYann LeCun57:12

        Two things. They pay people better.

    17. HSHarry Stebbings57:14

        Good.

    18. YLYann LeCun57:16

        Second thing is they, they, they give them resources for research. They can get extra resources through grants and stuff like that, but, but they could. And then they also attract some of the best students in the world, so you get the ideal, uh, combination that you only get in the, you know, top 30 universities in North America.

    19. HSHarry Stebbings57:32

        So we've got China. We've got Europe. What about the US? What could they do differently or improve?

    20. YLYann LeCun57:38

        Well, so there's a lot that the US does right, uh, in terms of, of research, which is, to a large extent, uh, a, a bit of a, you know, partial explanation for the success of the technological industry, the, the tech industry in the US. I think, you know, partly because the- you know, the US devotes a kind of, you know, significant amount of, uh, resources to fundamental research through NSF and NIH, and you know, various other outfits, probably more than, uh, Europe. Universities, uh, pays their, their faculty pretty well, um, particularly in areas like, uh, computer science and AI. Now, this comes with, uh, a downside, and the downside is that, uh, studying in the US is expensive.

    21. HSHarry Stebbings58:19

        Fair enough.

    22. YLYann LeCun58:19

        It's a trade-off, right? So can you do one without the other? Switzerland figured out how to pay academics, uh, pretty well while actually offering (laughs) free education to their, to their students, so, you know, there is a way to do it. Canada also figured out a pretty good trade-off as well. So a lot of things the US does right, but they... one thing that the US system or, or lack thereof does, does right also is the willingness to take risk and invest on ideas that seem, you know, a little crazy, but, but basically, you know, the, the sort of vibrant startup scene in, uh, Silicon Valley and other, uh, places, uh, in the US, in New York, and in the Boston area is, is, you know, leading, uh, leading the world. Now you start seeing a similar thing in Europe now. There's been like an enormous growth, for example, of, uh, of tech startups in, uh, in Paris, uh, in Paris area, in France more generally, and continental Europe a little widely, m- more widely, and, uh, in the UK as well. And, and so I think that's, that's a, that's a good thing, but it's still a little more difficult to have access to investment money in Europe than it is in the US.

    23. HSHarry Stebbings59:25

        That's why I'm here, Yann. I'm happy to provide. (laughs)

    24. YLYann LeCun59:29

        (laughs)

    25. HSHarry Stebbings59:29

        Um, I, I'm gonna do a penultimate one for you. When you think about what you'd most like someone listening to take away, what would it be? When they hear this, what do you want them to take away as the number one thing?

    26. YLYann LeCun59:41

        AI is going to bring a, a new, a new renaissance for, for humanity, a new, a new kind of... a new form of enlightenment, if you want, because AI is going to amplify everybody's intelligence, right? It's like every one of us will have a staff of people who are smarter than us and know most things about, uh, you know, most things and, and most topics. So it's going to empower every one of us. It's gonna make us more creative because we're gonna be able to produce, uh, text, art, music, videos without necessarily having all the technical, uh, skills that are currently required for, for doing those things, and, and so exercise our creative juices. So that's, that's the positive side. There are risks. There's no question, but it's not like those risks... Don't believe the people who tell you that those risks are inevitable or that they will inevitably lead to catastrophe. That's just not true. It's like-

    27. HSHarry Stebbings1:00:36

        Right.

    28. YLYann LeCun1:00:36

        ... you know, place yourself in 1920. Like, who would have thought that a mere 50 years later you could, you know, cross the Atlantic in a few hours in complete safety, you know, at near the speed of sound. You know, would people seriously want to ban aviation or call for regulation of jet engines before jet engines existed? I mean, that's kind of insane.So, I'm not against regulation. Uh, the- there should be regulation of AI products, particularly the ones that involve making critical decisions for people. But regulating or slowing down research is complete nonsense, and it's just obscurantism.

    29. HSHarry Stebbings1:01:14

        Whose incumbent team do you most respect and admire when you look at Amazon, Facebook, Google, in terms of their approach and talent internally? Outside of Meta, obviously.

    30. YLYann LeCun1:01:27

        So this is changing a lot. (laughs) And the reason it's changing is because, uh, a lot of people are leaving large companies and large labs.

Episode duration: 1:06:02