François Chollet: Keras, Deep Learning, and the Progress of AI | Lex Fridman Podcast #38

1. 0:00 – 15:00

   The following is a…

   1. LFLex Fridman0:00

      The following is a conversation with Francois Chollet. He's the creator of Keras, which is an open source deep learning library that is designed to enable fast, user-friendly experimentation with deep neural networks. It serves as an interface to several deep learning libraries, most popular of which is TensorFlow, and it was integrated into the TensorFlow main code base a while ago, meaning if you want to create, train, and use neural networks, probably the easiest and most popular option is to use Keras inside TensorFlow. Aside from creating an exceptionally useful and popular library, Francois is also a world-class AI researcher and software engineer at Google, and he's definitely an outspoken, if not controversial personality in the AI world, especially in the realm of ideas around the future of artificial intelligence. This is the Artificial Intelligence podcast. If you enjoy it, subscribe on YouTube, give us five stars on iTunes, support it on Patreon, or simply connect with me on Twitter at Lex Fridman, spelled F-R-I-D-M-A-N. And now here's my conversation with Francois Chollet. You're known for not sugarcoating your opinions and speaking your mind about ideas in AI, especially on Twitter. It's one one of my favorite Twitter accounts. So what's one of the more controversial ideas you've expressed online and gotten some heat for? How do you pick?

   2. FCFrançois Chollet1:32

      (laughs) How do I pick? Yeah, no, I think if you have, um, if you go through the trouble of maintaining a Twitter account, you might as well speak your mind, you know? Otherwise it's, you know, what- what's even the point of having a Twitter account? It's like having a nice car and just leaving it- leave it in the- in the garage. Uh, yes, so what's one thing for which I got a lot of pushback, perhaps, you know, uh, that time I wrote something about, uh, the idea of intelligence explosion, and I was questioning, uh, the idea and the reasoning behind this idea, and, uh, I got a lot of pushback on that, uh, got a lot of flak for it. So yeah, so intelligence explosion, I'm sure you're familiar with the idea, but it's the idea that if you were to build general AI problem-solving algorithms, well, the problem of building such an AI, that itself is a problem that could be solved by your AI, and maybe it could be solved better than, uh, than what humans can do.

   3. LFLex Fridman2:33

      Right.

   4. FCFrançois Chollet2:33

      So your AI could start tweaking its own algorithm, could, uh, start being a better version of itself and so on, iteratively, in a- in a recursive fashion, and so you would end up with, um, an AI with exponentially increasing intelligence.

   5. LFLex Fridman2:50

      That's right.

   6. FCFrançois Chollet2:51

      And I was basically questioning this idea, first of all, because the notion of intelligence explosion uses an implicit definition of intelligence that doesn't sound quite right to me. It considers intelligence as a property of a brain that you can consider in isolation, like the height of a building for instance.

   7. LFLex Fridman3:15

      Right.

   8. FCFrançois Chollet3:16

      But that's not really what intelligence is. Intelligence, uh, emerges from the interaction between a brain, a body, like embodied intelligence, and an environment. And if you're missing one of these pieces, then you cannot really define intelligence anymore. So just tweaking a brain to make it smarter and smarter doesn't actually make any sense to me.

   9. LFLex Fridman3:39

      So first of all, you're crushing the dreams of many people, right? So there's, uh, let's look at, like, Sam Harris, actually a lot of physicists, Max Tegmark, people who think, you know, the universe is a information processing system, our brain is kind of an information processing system.

   10. FCFrançois Chollet3:57

       It is.

   11. LFLex Fridman3:57

       So what's the theoretical limit? Like, wh- it doesn't make sense that there should be some, uh... It seems naive to think that our own brain is somehow the limit of the capabilities of this information, it's just I'm playing devil's advocate here, uh, this information processing system, and then if you just scale it, if you're able to build something that's on par with the brain, you just, the process that builds it just continues and it'll improve exponentially. So that- that's the logic that's used actually by almost everybody that is worried about superhuman intelligence.

   12. FCFrançois Chollet4:37

       Yeah.

   13. LFLex Fridman4:37

       So you're- you're trying to make... So most people who are skeptical of that are kind of like, (sighs) "This doesn't..." Their thought process is, "This doesn't feel right." Like, that's for me as well. So I'm more like, it doesn't... (sighs) The whole thing is shrouded in mystery where you- you can't really say anything concrete, but you could say this doesn't feel right, this doesn't feel like that's how the brain works. And you're trying to, with your blog post and now making it a little more explicit. So one idea is that the brain isn't... It exists alone, it exists within the environment. So you can't exponentially... You would have to somehow exponentially improve the environment and the brain together almost, yeah, in order to create something that's much smarter in some kind of, uh... Of course, we don't have a definition of intelligence. But-

   14. FCFrançois Chollet5:30

       That's correct. That's correct. I- I- I don't think... If you look at very smart people today, even humans, not even talking about AIs, I don't think their brain and the performance of their brain is the bottleneck to their- to their expressed intelligence, to their achievements. You cannot just tweak one part of the system, like of this brain-body-environment system and expect the capabilities, like what emerges out of this system, to just, you know, uh, explode exponentially.... because, um, any time you improve one part of a system with many interdependencies like this, uh, there's a new bottleneck that arises, right? And I don't think even today, for very smart people, their brain is not the bottleneck, uh, to the sort of problems they can solve, right? In fact, many various smart people today, uh, you know, they're, they're not actually solving any big scientific problems. They're not Einstein. They're like Einstein, but, you know, the, the patent clerk days. Um-

   15. LFLex Fridman6:29

       (laughs)

   16. FCFrançois Chollet6:29

       ... like Einstein became Einstein because this was a meeting of a genius with a big problem at the right time, right? But maybe this meeting could have, you know, never happened and then Einstein would have just been a patent clerk, right? It's ... and in fact many people today are probably like genius level smart, but you wouldn't know because they're not expressing any of that.

   17. LFLex Fridman6:54

       Wow, that's brilliant. So, we can think of the world, Earth, but also the universe as just ... as a space of problems. So all these problems and tasks are roaming it of various difficulty and there's agents, creatures like ourselves and animals and so on that are also roaming it, and then you, you get coupled with a problem and then you solve it. But without that coupling, you can't demonstrate your "intelligence."

   18. FCFrançois Chollet7:22

       Exactly. Intelligence is the meeting of great problem-solving capabilities-

   19. LFLex Fridman7:27

       Mm.

   20. FCFrançois Chollet7:27

       ... with a great problem, and if you don't have the problem, you're not really expressing intelligence. All y- all you're left with is potential intelligence-

   21. LFLex Fridman7:34

       Mm.

   22. FCFrançois Chollet7:34

       ... like the performance of your brain or, you know, how high your IQ is, which in itself, uh, is just a number, right?

   23. LFLex Fridman7:41

       Right. So, you mentioned problem-solving capacity.

   24. FCFrançois Chollet7:46

       Yeah.

   25. LFLex Fridman7:47

       What, what do you think of as problem-solving capa- ... what ... can you try to define intelligence? Like what does it mean to be more or less intelligent? Is it completely coupled to a particular problem? Or is there something a little bit more universal?

   26. FCFrançois Chollet8:05

       Yeah, I do believe all intelligence is specialized intelligence, even human intelligence has some degree of generality. Well, all intelligent systems have some degree of generality, but they're always specialized in, in one category of problems. So the, the human intelligence is specialized in the human experience and that shows at various levels. That shows in some prior knowledge that's innate that we have at birth, knowledge about, um, things like agents, uh, goal-driven behavior, uh, visual priors about what makes an object, priors about time, and so on. Uh, that shows also in the way we learn. For instance, it's very, very easy for us to pick up language, it's very, very easy for us to learn certain things because we are basically hardcoded to learn them, and we are specialized in solving certain kinds of problem and we are quite l- useless when it comes to other kinds of problems. For instance, we, we are not really designed to handle very long-term problems. We have no capability of seeing the, the, the very long-term. Um, we don't have, um, very much working memory, you know?

   27. LFLex Fridman9:18

       So, how do you think about long-term? Do you think long-term planning? Are we talking about a scale of years, millennia? What do you mean by l- long-term, we're not very good?

   28. FCFrançois Chollet9:28

       Well, human intelligence is specialized in the human experience and human experience is, is very short. Like one lifetime is short. Even within one lifetime, uh, we have a, a, a, a very hard time envisioning, you know, uh, things on a scale of years. Like it's very difficult to project yourself at a, at a scale of five year, at a scale of 10 years, and so on.

   29. LFLex Fridman9:46

       Right.

   30. FCFrançois Chollet9:47

       We can solve only fairly narrowly scoped problems. So, when it comes to solving bigger problems, larger scale problems, we are not actually doing it on an individual level. So it's not actually our brain doing it. We, we, we have this thing called civilization, right, which is itself a sort of problem-solving system, a sort of, uh, artificial intelligent system, right?
2. 15:00 – 30:00

   Right. Let me linger…

   1. FCFrançois Chollet15:00

      to a recursively self-improving superhuman AI. And you can just observe, you know, is science, is scientific progress today exploding? Which, you know, itself is- is- is an interesting question. You can use that as a basis to try to understand what will happen with a superhuman AI that has s- uh, uh, science-like behavior.

   2. LFLex Fridman15:20

      Right. Let me linger on it a little bit more. What is your intuition why an intelligence explosion is not possible? Like taking the scientific... all the s- semi-scientific revolutions, why can't we slightly accelerate that process?

   3. FCFrançois Chollet15:38

      So you- you can absolutely, uh, uh, accelerate any problem-solving process.

   4. LFLex Fridman15:42

      Yep.

   5. FCFrançois Chollet15:43

      So, uh, recursively, uh, uh, recursive self-improvement is absolutely a real thing, but what happens with a recursively self-improving system is typically not explosion because no system exists in isolation. And so tweaking one part of the system means that suddenly another part of the system becomes a bottleneck, and if you look at science for instance, which is clearly recursively self-improving, clearly a problem-solving system, scientific progress is not actually exploding. If you look at science, what you see is the picture of a system that is consuming an exponentially increasing amount of resources-

   6. LFLex Fridman16:20

      Right.

   7. FCFrançois Chollet16:20

      ... but ha- uh, it's having a linear output in terms of scientific progress, and may- maybe that- that will seem like a very strong claim. Many people are- are- are actually saying that, you know, s- scientific progress is exponential, but when they're claiming this, they are actually looking at indicators of, uh, uh, resource consumptions, resource consumption by science. For instance, the number of, um, uh, papers being published, the number of pa- patents being filed and so on, which are just- just completely correlated with how many people are working o- on the, uh, on- on science today.

   8. LFLex Fridman16:57

      Yeah.

   9. FCFrançois Chollet16:58

      Right? So it's actually an indicator of resource consumption, but what you should look at is the output, is um, progress in terms of the knowledge that science generates, in terms of the- the scope and significance of the problems that we solve, and, uh, some people have actually been trying to measure that.

   10. LFLex Fridman17:16

       Mm-hmm.

   11. FCFrançois Chollet17:16

       Like, um, Michael Nielson, for instance.

   12. LFLex Fridman17:19

       Mm-hmm.

   13. FCFrançois Chollet17:20

       He had a- a very nice paper, uh, I think it was last year about it. So his approach to measure scientific progress was to, uh, look at the timeline, uh, of scientific discoveries over the past, you know, 100, 150 years, and for, um, each measured discovery, ask a panel of experts to rate the significance of the discovery.

   14. LFLex Fridman17:46

       Hmm.

   15. FCFrançois Chollet17:46

       And if the output of science as an institution were exponential, you would expect the temporal density of significance to go up exponentially, maybe because there- there's a faster rate of discoveries, maybe because the discoveries are, you know, increasingly more important.And, uh, what actually happens if you, if you plot this temporal density of significance measured in this way is that you see very much a flat graph. You see a flat graph across all disciplines, across physics, biology, medicine, and so on. And it actually makes a lot of the- of sense if you think about it, because think about the progress of physics, uh, uh, 110 years ago, right? It was a time of crazy change.

   16. LFLex Fridman18:29

       Mm-hmm.

   17. FCFrançois Chollet18:30

       Think about the progress of technology, you know, uh, 130 years ago when we started having, you know, replacing horses with cars, when we started having electricity and so on.

   18. LFLex Fridman18:39

       Yeah.

   19. FCFrançois Chollet18:40

       It was a time of incredible change, and today is also a time of very, very fast change, but it would be, uh, an unfair characterization to say that today technology and science are moving way faster than they did 50 years ago or 100 years ago. And if you do try to rigorously plot the temporal density of the significance-

   20. LFLex Fridman19:04

       Significant ideas, yeah.

   21. FCFrançois Chollet19:04

       ... yeah, of significance idea, of significant idea, sorry, you do see very flat curves.

   22. LFLex Fridman19:09

       That's fascinating.

   23. FCFrançois Chollet19:10

       And, and, and you can check out the paper that Michael Nielson had, uh, about this idea. And so the way I interpret it is as you make progress in a, in a given field or in any given sub-field of science, it becomes exponentially more difficult to make further progress.

   24. LFLex Fridman19:30

       Mm-hmm.

   25. FCFrançois Chollet19:30

       Like, the very first, uh, uh, person to work on information theory, if you enter a new field and it's still the very early years, there's a lot of, uh, low-hanging fruit you can pick.

   26. LFLex Fridman19:41

       That's right, yeah.

   27. FCFrançois Chollet19:42

       But the next generation of researchers is gonna have to, uh, dig much harder actually, um, to make smaller discoveries, uh, probably larger number of smaller discoveries. And to achieve the same amount of impact, you're gonna need a much greater headcount, and that's exactly the picture you're seeing with science is that the number of scientists and engineers is in fact increasing exponentially. The amount of computational resources that are available to science is increasing exponentially and so on, so the resource consumption of science is exponential, but the output in terms of progress, in terms of significance is linear. And the reason why is because... And even though science is recursively self-improving, meaning that scientific progress-

   28. LFLex Fridman20:27

       Mm-hmm.

   29. FCFrançois Chollet20:28

       ... uh, turns into technological progress which in turn helps science. If you look at, um, computers for instance are a product of science, and computers are tremendously useful in speeding up science. The internet, same thing, the internet is a technology that's made possible by, uh, very recent scientific advances, and itself, uh, uh, because it enables, you know, scientists to, to, to network, to communicate, to exchange papers and ideas much faster-

   30. LFLex Fridman20:55

       Mm-hmm.
3. 30:00 – 45:00

   Right. …

   1. FCFrançois Chollet30:00

      it at some point. I also believe that... You know, it's... The problem with, with talking about human-level intelligence is that implicitly you're considering like an axis of intelligence with different levels.

   2. LFLex Fridman30:14

      Right.

   3. FCFrançois Chollet30:14

      But that's not really how intelligence works. Intelligence is very, uh, multidimensional. And so does the question of...... uh, capabilities, but there's, uh, also the question of being human-like and it's two very different things. Like, you can build potentially very, uh, uh, advanced intelligent agents that are not human-like at all, and you can also build very, uh, human-like agents. And these are very, two very different things, right?

   4. LFLex Fridman30:37

      Right. Let's go from the philosophical to the practical. Uh, can you give me a history of Keras and all the major deep learning frameworks that you kind of remember in relation to Keras and in general? TensorFlow, Theano, the old days. Can you give a brief overview, Wikipedia-style history and your role in it before we return to AGI discussions? (laughs)

   5. FCFrançois Chollet30:59

      Yeah, that's, that's a broad topic. So I started working on Keras ... It wasn't named Keras at the time. I actually picked the name, like, uh, just the day I was gonna release it. So I started working on it in February 2015, and so at the time, there weren't too many people working on deep learning, maybe like fewer than 10,000. The software tooling was not really developed. So the main deep learning library was Caffe, which was mostly C++.

   6. LFLex Fridman31:30

      Why, why do you say Caffe was the main one?

   7. FCFrançois Chollet31:32

      Caffe was vastly more popular than Theano in, uh, in late 2014, early 2015. Caffe was the one library that everyone was using for computer vision.

   8. LFLex Fridman31:43

      And computer vision was the most popular problem-

   9. FCFrançois Chollet31:46

      Absolutely.

   10. LFLex Fridman31:47

       ... in deep learning at the time.

   11. FCFrançois Chollet31:47

       Uh, com- Like, convnets was, like, the subfield of deep learning that everyone was working on.

   12. LFLex Fridman31:52

       Right.

   13. FCFrançois Chollet31:53

       So myself ... So in,in, in late 2014, I was actually interested in, uh, RNNs, in recurrent neural networks, which was a very niche topic at the time, right? It really, it really took off arou- around 2016. And so I was looking for good tools. I had l- I had used, uh, Torch 7. I had used Theano, used Theano a lot, uh, in, uh, Kaggle competitions. Hmm, I had used Caffe, and, uh, th- there was no, like, good solution for RNNs at the time. Like, there was no reusable open source implementation of an LSTM, for instance. So I decided to build my own, and at first, uh, the pitch for that was it was gonna be mostly around, uh, LSTM, recurrent neural networks. It was gonna be in Python. An important decision, uh, at the time that was kinda not obvious is that the models w- would be defined via, uh, Python code, which was kind of, like, going against, uh, the mainstream at the time because Caffe, PyTorch and so on, like, all the big libraries, were actually, uh, going with the approach of having static configuration files in YAML to define models.

   14. LFLex Fridman33:05

       Yeah.

   15. FCFrançois Chollet33:05

       So some libraries were using, uh, code to define models, like Torch 7 obviously, but that was not Python. Lasagne was, like, a Theano-based, uh, very early library that was, I think, developed, I'm not sure exactly, probably late 2014.

   16. LFLex Fridman33:20

       It's Python as well.

   17. FCFrançois Chollet33:21

       It's Python as well. It was, it was, like, on top of Theano. And so I started working on something and the, and the value proposition at the time was that not only did the, uh, what I think was the first reusable open source implementation of LSTM, you could combine RNNs and convnets with the same library, which was not really possible before. Like, Caffe was only doing convnets. And it was kinda easy to use because ... So before I was using Theano, I was actually using scikit-learn, and I loved scikit-learn for its usability, so, uh, I drew a lot of inspiration from scikit-learn when, when I made Keras. It's almost like scikit-learn for neural networks.

   18. LFLex Fridman34:05

       Yep. The fit function.

   19. FCFrançois Chollet34:06

       Exactly, the fit function. Like, reducing, uh, a complex training loop to a single function call, right?

   20. LFLex Fridman34:12

       Yeah.

   21. FCFrançois Chollet34:12

       And of course, you know, some people will say this is hiding a lot of details, but that's exactly the point, right?

   22. LFLex Fridman34:18

       Right.

   23. FCFrançois Chollet34:18

       The magic is the point.

   24. LFLex Fridman34:20

       Right.

   25. FCFrançois Chollet34:20

       So it's magical, but in a good way. It's magical in the sense that it's delightful, right?

   26. LFLex Fridman34:25

       Yeah, yeah. I'm, I'm actually quite surprised. I didn't know that it was born out of desire to, uh, implement RNNs and LSTMs.

   27. FCFrançois Chollet34:32

       It was. It was.

   28. LFLex Fridman34:32

       That's fascinating. So you were actually one of the first people to really try to attempt, um, to get the major architectures together, and it's also interesting, you made me realize that that was a design decision at all is defining the model in code. Just I'm, I'm putting myself in your shoes, whether the YAML, especially if Caffe was the most popular.

   29. FCFrançois Chollet34:53

       It was the most popular by far at the time.

   30. LFLex Fridman34:56

       If I was ... If I were ... Yeah, I don't ... It ... I didn't like the YAML thing, but it makes more sense that you would put in a configuration file the definition of a model. That's an interesting gutsy move to stick with defining it in code, just if, if you look back.
4. 45:00 – 1:00:00

   Mm-hmm. …

   1. FCFrançois Chollet45:00

      so that they- they- they have, uh, an- an API surface that is, uh, as small as possible, right?

   2. LFLex Fridman45:07

      Mm-hmm.

   3. FCFrançois Chollet45:07

      And- and you want, uh- uh, this modular hierarchical architecture to reflect the way that domain experts think about the problem. 'Cause like as- as a domain expert, when- when you're reading about a new API, you're reading a tutorial or- or some docs pages, um, you already have a way that you're thinking about the problem.

   4. LFLex Fridman45:28

      Right.

   5. FCFrançois Chollet45:28

      You already have like, uh- uh, certain concepts in mind, uh, and- and- and you're thinking about, uh, how they relate together, and when you're reading docs you're trying to build, uh, as quickly as possible a mapping between the concepts...

   6. LFLex Fridman45:42

      Mm-hmm.

   7. FCFrançois Chollet45:43

      ... featured in new API and the concepts in your mind, so you're trying to map your mental model as a domain expert to the way things, uh, uh, work in the API.

   8. LFLex Fridman45:53

      Mm-hmm.

   9. FCFrançois Chollet45:53

      So, you need a- an API and an underlying implementation that are reflecting the way people think about these things.

   10. LFLex Fridman46:00

       So you're minimizing the time it takes to do the mapping?

   11. FCFrançois Chollet46:02

       Yes.

   12. LFLex Fridman46:03

       All right.

   13. FCFrançois Chollet46:03

       Minimizing the time, the cognitive load there is in ingesting this new knowledge about your API. An API should not be self-referential or- or ref- referring to implementation details; it should only be referring to domain-specific concepts that people already kn- uh, understand.

   14. LFLex Fridman46:23

       Brilliant. So, what's the future of Keras and TensorFlow look like? What does TensorFlow 3.0 look like?

   15. FCFrançois Chollet46:30

       So, that's kind of too far in the future for me to answer, especially, uh- uh, since I'm not- I'm not even the one making these decisions.

   16. LFLex Fridman46:37

       Okay.

   17. FCFrançois Chollet46:39

       But, so from my perspective, which is, you know, just one perspective among many different perspectives on the TensorFlow team, I'm really excited by developing, uh, even higher level APIs. Higher level than Keras. I'm really excited by hyper-parameter tuning, by, uh, automated machine learning, AutoML. I think the future is not just, you know, defining a model like, uh- uh, like you were assembling Lego blocks and then

   18. NANarrator47:07

       (laughs)

   19. FCFrançois Chollet47:08

       ... calling fit on it. It's more like an automagical model that will just look at your data and optimize the objective you- you're after, right? So that's- that's, uh- uh, what- uh, what I'm looking into.

   20. LFLex Fridman47:23

       Yeah, so you, uh, put the baby into a room with the problem and come back a few hours later, uh, with a f- with a fully solved problem.

   21. FCFrançois Chollet47:31

       Exactly. It's not like a box of Legos.

   22. LFLex Fridman47:33

       Right.

   23. FCFrançois Chollet47:33

       It's more like the combination of a kid that's really good at Legos...

   24. LFLex Fridman47:37

       (laughs) Yeah.

   25. FCFrançois Chollet47:37

       ... and a box of Legos.

   26. LFLex Fridman47:38

       Yeah. Exactly.

   27. FCFrançois Chollet47:38

       And just building the thing on its own.

   28. LFLex Fridman47:41

       Nice. Uh, very nice. So that's- that's an exciting future and I think there's a huge amount of applications and, uh, revolutions to be had, uh, under the constraints of the discussion we previously had. But what do you think are the current limits of deep learning? If we look specifically at these, uh, function approximators that try to generalize from data. So you've, uh, you've talked about local versus extreme generalization. You mentioned the neural networks don't generalize well and humans do, so there's this gap. So w- and you've also mentioned that ex- generalization, extreme generalization requires something like reasoning to fill those gaps. So how can we start trying to build systems like that?

   29. FCFrançois Chollet48:27

       Right. Yeah, so this is- this is by design, right? Deep learning models are like huge biometric models.... differentiable, so continuous, uh, that go from an input space to an output space. And they're trained with gradiente descent. So they are trained pretty much point by point.

   30. LFLex Fridman48:46

       Mm-hmm.
5. 1:00:00 – 1:15:00

   Hmm. So let's talk…

   1. FCFrançois Chollet1:00:00

      over rule-based models is gonna be a cornerstone of AI research in the next century, right? And, um, that doesn't mean we are, we are gonna drop deep learning. Deep learning is immensely useful, like being able to learn these, uh, these, uh, uh, uh, very flexible, adaptable parametric models with .......................... That's, that's actually immensely useful. Like, all it's doing is pattern recognition, but being good at pattern recognition, given lots of data is, is, is just extremely powerful. So we are, we are still gonna be working on deep learning. We're gonna be working on program synthesis. We're gonna be combining the two in increasingly automated ways.

   2. LFLex Fridman1:00:35

      Hmm. So let's talk a little about, about data. You've tweeted (laughs) -

   3. FCFrançois Chollet1:00:40

      (laughs)

   4. LFLex Fridman1:00:42

      ... uh, about 10,000 deep learning papers have been written about hard coding priors about a specific task in a neural network architecture. It works better than a lack of a prior. Basically summarizing all these efforts, they- they put a name to an architecture, but really what they're doing is hard coding some priors that improve the performance of the system.

   5. FCFrançois Chollet1:01:00

      Yes, yes.

   6. LFLex Fridman1:01:01

      But we're... (laughs) Uh, get straight to the point is- is probably true, so you say that you can always buy performance, "buy" in quotes, performance by either training on more data, better data, or by injecting task information-

   7. FCFrançois Chollet1:01:15

      Yeah.

   8. LFLex Fridman1:01:15

      ... to the architecture as a pre-processing. Uh, however, this isn't informative about the generalization power of the techniques used, the fundamental ability to generalize. Do you think we can go far by coming up with better methods for this kind of cheating, for better methods of large-scale annotation of data, so building better priors?

   9. FCFrançois Chollet1:01:34

      If you- if you'd have made it, it's not cheating anymore.

   10. LFLex Fridman1:01:37

       Right. Uh, um, I'm joking-

   11. FCFrançois Chollet1:01:38

       (laughs)

   12. LFLex Fridman1:01:38

       ... about the cheating, but large scale... So basically, I'm asking, um, about something that hasn't, uh, from my perspective, been researched too- too much is, uh, exponential improvement in annotation of data.

   13. FCFrançois Chollet1:01:54

       Yeah.

   14. LFLex Fridman1:01:56

       Do you- have you often think about that?

   15. FCFrançois Chollet1:01:58

       I think it's- it's actually been- been researched quite a bit, you just don't see publications about it because, you know, people who publish papers are gonna publish about known benchmarks.

   16. LFLex Fridman1:02:08

       Right.

   17. FCFrançois Chollet1:02:08

       Sometimes they're going to raise a new benchmark.

   18. LFLex Fridman1:02:09

       Right.

   19. FCFrançois Chollet1:02:09

       People who actually have real-world large-scale-

   20. LFLex Fridman1:02:12

       Rules, yeah.

   21. FCFrançois Chollet1:02:12

       ... deep learning problems, they're gonna spend a lot of resources into data annotation and good data annotation pipelines, but you don't see any papers about it.

   22. LFLex Fridman1:02:19

       That's interesting. So do you think there are certainly resources, but do you think there's innovation happening?

   23. FCFrançois Chollet1:02:24

       Oh, yeah, definitely.

   24. LFLex Fridman1:02:25

       ... as

   25. FCFrançois Chollet1:02:25

       To clarify, uh, uh, and the point in the twist, so machine learning in general is the science of generalization. You want to generate knowledge that can be reused across different datasets, across different tasks.

   26. LFLex Fridman1:02:41

       Right.

   27. FCFrançois Chollet1:02:42

       And if instead you're looking at one dataset and then you are hard coding, uh, knowledge about this task into your architecture, this is no more useful than training a network and then saying, "Oh, I found these weight values, uh, uh, perform well." Right?

   28. LFLex Fridman1:03:00

       Right.

   29. FCFrançois Chollet1:03:02

       So, uh, uh, David Ha, I don't know if- if you know w- uh, uh, David, he had a paper the other day about, uh, weight agnostic neural networks.

   30. LFLex Fridman1:03:10

       (laughs)
6. 1:15:00 – 1:30:00

   Yeah. To me, that's…

   1. FCFrançois Chollet1:15:00

      them than real news, simply because they are not, um, constrained to reality, so they can be as atrocious, as, as, as, as surprising, as, as good stories as you want because they're artificial, right?

   2. LFLex Fridman1:15:15

      Yeah. To me, that's an exciting world because so much good can come. So there's an opportunity to educate people. You can, uh, balance people's worldview-

   3. FCFrançois Chollet1:15:28

      Mm-hmm.

   4. LFLex Fridman1:15:28

      ... with other ideas. So, the- there's so many objective functions. The space of objective functions that create better civilizations is large, arguably infinite. But there's also a large space that creates division and, uh, and, uh, and destruction, civil war, a lot of bad stuff. And the worry is, uh, naturally probably that space is bigger, first of all, and if we don't explicitly think about what kind of, uh, effects are going to be observed, uh, from different objective functions, then we're gonna get into trouble.

   5. FCFrançois Chollet1:16:10

      Mm-hmm.

   6. LFLex Fridman1:16:10

      But the question is, how do we... how do we get into rooms and have discussions, so inside Google, inside Facebook, inside Twitter, and think about, okay, how can we drive up engagement and at the same time create a good society?

   7. FCFrançois Chollet1:16:27

      Mm-hmm. Mm-hmm.

   8. LFLex Fridman1:16:28

      Is there, is it even possible to have that kind of philosophical discussion?

   9. FCFrançois Chollet1:16:31

      Um, I think you can definitely try. So from my perspective, I would feel rather uncomfortable with, uh, companies that are in control of these, uh, newsfeed algorithms, uh, with them making explicit decisions to manipulate, uh, uh, people's opinions or behaviors, even if the intent is good, because that's- that's a very totalitarian mindset. So instead, what I would like to see, and it's probably never gonna happen because it's- it's not super realistic, but that's actually something I really care about, I would like, uh, all these algorithms, uh, to present, uh, configuration settings to their users so that the users-

   10. LFLex Fridman1:17:11

       Ah, yeah.

   11. FCFrançois Chollet1:17:11

       ... can actually make the decision about how they want to be impacted, uh, by these, uh, uh, information recommendation, content recommendation algorithms. For instance, as a user of something like YouTube or Twitter-

   12. LFLex Fridman1:17:26

       Mm-hmm.

   13. FCFrançois Chollet1:17:26

       ... maybe I want to maximize learning-

   14. LFLex Fridman1:17:28

       Learning.

   15. FCFrançois Chollet1:17:28

       ... about a specific topic, right? So I want the, the algorithm, um, to, uh, uh, feed my curiosity, right, which is in itself a very interesting problem. So instead of maximizing my engagement, it will maximize how fast and how much I'm learning, and it will also take into account the accuracy, uh, hopefully, you know, of the information I'm learning. So yeah, uh, the user should be able to determine exactly how these algorithms are affecting their lives. I, I don't want actually any entity, uh, making decisions about, uh, in which direction they're gonna, uh, uh, try to manipulate me, right? I want, uh, I want technology. So AI, these algorithms are increasingly gonna be our interface to a world that is increasingly made of information.

   16. LFLex Fridman1:18:19

       Right.

   17. FCFrançois Chollet1:18:20

       And I want... I want everyone to be in control of this interface, to interface with the world on their own terms. So if someone wants, uh, these algorithms to serve, you know, their own personal growth goals, they should be able to configure these algorithms in such a way.

   18. LFLex Fridman1:18:41

       Yeah, but so I know it's painful to have explicit decisions, but there is underlying explicit decisions, which is some of the most beautiful fundamental philosophy that, uh, that we have before us, which is personal growth. If I want to watch videos from which I can learn, what does that mean? So if I have a checkbox that wants to emphasize learning...... there's still an algorithm with explicit decisions in it that would promote learning. What does that mean for me? Like for example, I've watched a documentary on, um, flat earth theory, I guess. It- it was very like, th- I learned a lot. I- I'm really glad I watched it. It was ... a friend recommended it to me. Not (laughs) 'cause I don't have s- such an allergic reaction to c- to crazy people as my fellow colleagues do, but it was very wi- it was very eye-opening and for others it might not be. For others, they- they might just get, uh, turned off with that.

   19. FCFrançois Chollet1:19:45

       Mm-hmm.

   20. LFLex Fridman1:19:45

       Same with Republican-Democrat, like what ... It's a non-trivial problem. And f- first of all, if it's done well, I don't think it's something that wouldn't happen, that, uh, that YouTube wouldn't be promoting or Twitter wouldn't be. It's just a really difficult problem, how do we do, how to give people control?

   21. FCFrançois Chollet1:20:05

       Well, it's mostly an- an interface design problem.

   22. LFLex Fridman1:20:08

       Right. (laughs)

   23. FCFrançois Chollet1:20:09

       The- the way I see it, you want to create technology that's like, um, a mentor or a coach-

   24. LFLex Fridman1:20:14

       Right.

   25. FCFrançois Chollet1:20:15

       ... or an assistant.

   26. LFLex Fridman1:20:16

       Right.

   27. FCFrançois Chollet1:20:16

       So that it's not your boss, right? You are in- in control of it.

   28. LFLex Fridman1:20:22

       Right.

   29. FCFrançois Chollet1:20:22

       You are telling it what to do for you. And if you feel like it's manipulating you, it's not actually, it's not actually doing what you want, you should be able to switch to a different algorithm, you know?

   30. LFLex Fridman1:20:34

       Right. So that fine-tune control, and you kinda learn, you're trusting the human collaboration. I mean that's how I see autonomous vehicles too, is giving as much information as possible and you learn that dance yourself.
7. 1:30:00 – 1:31:43

   Yeah. …

   1. LFLex Fridman1:30:00

      that mean? Uh, I'm impressing you with natural language processing. Maybe if you weren't able to see me, maybe this is a phone call.

   2. FCFrançois Chollet1:30:07

      Yeah.

   3. LFLex Fridman1:30:08

      So that kind of system.

   4. FCFrançois Chollet1:30:09

      Okay. So-

   5. LFLex Fridman1:30:10

      Companion.

   6. FCFrançois Chollet1:30:11

      So that- that's very much about building human-like AI and you're asking me, you know, is this- is this an exciting perspective?

   7. LFLex Fridman1:30:18

      Yes.

   8. FCFrançois Chollet1:30:19

      I think so, yes. Not so much because of- of- of what, uh, uh, artificial human-like intelligence could do but, you know, from an intellectual perspective, I think if you could build truly human-like intelligence, that means you could actually understand human intelligence, which is fascinating, right? Uh, human-like intelligence is gonna require emotions, it's gonna require consciousness, which is not things that- that would normally be required by an intelligent, uh, system. If you look at, you know, we were mentioning earlier, like science as- as super- superhuman problem-solving, uh, uh, um, agent or system, it does not have consciousness, it doesn't have emotions. In general, so emotions, I see consciousness as being on the same spectrum as emotions. It is, uh, a component of the subjective experience that is meant very much to, uh, guide, uh, behavior generation, right?

   9. LFLex Fridman1:31:18

      Mm-hmm.

   10. FCFrançois Chollet1:31:18

       It's meant to guide your behavior. Uh, in general, um, human intelligence and animal intelligence, uh, has evolved for the purpose of behavior generation, right?

   11. LFLex Fridman1:31:28

       Mm-hmm.

   12. FCFrançois Chollet1:31:29

       Uh, including in a social context, so that's why we actually need emotions. That's why we need consciousness.... an artificial intelligence system developed in different context may well never need them, may well- may well never be conscious, like science.

Episode duration: 1:59:49