Tomaso Poggio: Brains, Minds, and Machines | Lex Fridman Podcast #13

Lex Fridman 0:00 The following is a conversation with Tomaso Poggio. He's a professor at MIT, and is a director of the Center for Brains, Minds, and Machines. Cited over 100,000 times, his work has had a profound impact on our understanding of the nature of intelligence in both biological and artificial neural networks. He has been an advisor to many highly impactful researchers and entrepreneurs in AI, including Demis Hassabis of DeepMind, Amnon Shashua of Mobileye, and Christof Koch of the Allen Institute for Brain Science. This conversation is part of the MIT course on artificial general intelligence, and the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter, @LexFridman, spelled F-R-I-D. And now, here's my conversation with Tomaso Poggio. You've mentioned that in your childhood, you've developed a fascination with physics, especially the theory of relativity, and that Einstein was also a childhood hero to you. What aspect of Einstein's genius, the nature of his genius, do you think was essential for discovering the theory of relativity?

Tomaso Poggio 1:13 You know, Einstein was, uh, a hero to me, and I'm sure to many people, because he was able to make, uh, uh, of course, a major, major contribution to physics with, simplifying a bit, just a gedanken experiment, a thought experiment.

Lex Fridman 1:33 Mm-hmm.

Tomaso Poggio 1:35 You know, imagining, uh, communication with lights between a stationary observer and somebody on a train.

Lex Fridman 1:43 Mm-hmm.

Tomaso Poggio 1:43 And, uh, I thought that, um, you know, the, the, the, the fact that just with the force of y- of his thought, of his thinking, of his mind, it could get to some- something so deep in term of physical reality, how time depend on space and speed, is, was something absolutely fascinating. It was the power of intelligence, the power of the mind.

Lex Fridman 2:08 Do you think the ability to imagine, to visualize as he did, as a lot of great physicists do, do you think that's in all of us human beings? Or is there something special to that one particular human being?

Tomaso Poggio 2:23 I think, uh, you know, a- all of us can learn and have, uh, in principle similar b- breakthroughs. Uh, there is lesson to be learned from Einstein. Uh, he was one of five PhD students at ETH, uh, the Eidgenössische Technische Hochschule in, uh, Zurich, in physics, and he was the worst of the five. The only one who d- did not get an academic position when, uh, he, he graduated, when he finished his PhD, and he went to work, as everybody knows, for the patent office. And so it's not so much that he worked for the patent office, but the fact that obviously he was smart, but he was not a top student. O- obviously, he was the anticonformist, he was not thinking in the traditional way that probably his teachers and the other students were doing. So there is a lot to be said about, uh, you know, trying to be... to do the opposite or something quite different from what other people are doing. That's certainly true for the stock market. Never...

Lex Fridman 3:33 (laughs)

Tomaso Poggio 3:33 Never buy if everybody's buying. (laughs)

Lex Fridman 3:37 And also true for science.

Tomaso Poggio 3:38 Yes.

Lex Fridman 3:39 So you've also mentioned, staying on the theme of physics, that you were excited at a young age by the mysteries of the universe that, uh, physics could uncover. Such, as I saw mentioned, the possibility of time travel.

Tomaso Poggio 3:55 (laughs)

Lex Fridman 3:56 So the most out-of-the-box question I think I'll get to ask today, do you think time travel is possible?

Tomaso Poggio 4:03 Well, it would be nice if it were possible right now. Uh, you know, in science, you never say no, um...

Lex Fridman 4:12 But your understanding of the nature of time.

Tomaso Poggio 4:15 Yeah, it's very likely that it's not possible to travel in time. Um, you may be able to travel forward in time if we can, for instance, freeze ourselves or, uh, you know, go on some spacecraft traveling close to the speed of light. But in terms of actively traveling, for instance, back in time, I find probably very unlikely.

Lex Fridman 4:45 So do you still hold the, the underlying dream of the engineering intelligence that we'll build systems that are able to do such huge leaps, like discovering the kind of mechanism that would be required to travel through time? Do you still hold that dream or is- or echoes of it from your childhood?

Tomaso Poggio 5:07 Yeah, I, uh, you know, I don't think whether... Uh, there are certain problems that probably cannot be solved depending what, uh, what you believe about the physical reality, like, uh, you know, maybe totally impossible to create energy from nothing or to travel back in time. But, uh, um, about making machines that can, uh, think as well as w- we do or better, or more likely, especially in the short and mid-term, help us think better, which is in a sense is happening already with the computers we have, and it will happen more and more. Well, that I certainly believe, and I don't see in principle why computers at some point, uh, could not become more intelligent than we are. Although the word intelligence...... is a tricky one and one we should discuss- (laughs)

Lex Fridman 6:05 Yeah, for sure.

Tomaso Poggio 6:06 ... what I mean with that. (laughs)

Lex Fridman 6:08 Uh, in- intelligence, consciousness-

Tomaso Poggio 6:09 Yeah.

Lex Fridman 6:09 ... words like love, is, all these are very, uh-

Tomaso Poggio 6:14 Yeah.

Lex Fridman 6:14 ... need to be disentangled. So, you've mentioned also that you believe the problem of intelligence is the greatest problem in science, greater than the origin of life and the origin of the universe. You've also, uh, in the talk I've listened to, uh, said that you're open to arguments against, uh, against you. So, uh, what do you think is the most captivating aspect of this problem of understanding the nature of intelligence? Why does it captivate you as it does?

Tomaso Poggio 6:47 Well, originally, I think one of the motivation that I had as a, I guess, a teenager, when I was infatuated with theory of relativity, was really that I- I found that there was, uh, the problem of time and space and general relativity, but there were so many other problems of the same level of difficulty and importance that I could ... even if I were Einstein, it was difficult to hope to solve all of them. So, what about solving a problem whose solution en- allowed me to solve all the problems? And this was (laughs) what if we could find the key to an intelligence, you know, 10 times better or faster than Einstein?

Lex Fridman 7:37 So, that's sort of seeing artificial intelligence as a- as a tool to expand our capabilities, but is there just an inherent curiosity in you in just understanding what it is in our- in- in here that makes it all- all work?

Tomaso Poggio 7:54 Yes, absolutely. You are right. So, I was starting- I started saying this was the motivation when I was a teenager-

Lex Fridman 8:00 Right.

Tomaso Poggio 8:00 ... but, uh, you know, soon after, uh, I think the problem of human intelligence became a- a real focus of- f- you know, of- of my sci- of my science and my research because I think is, for me, the most interesting problem is really asking, uh, who we- we are, right? It's asking not only a question about science, but even about the very tool we are using to do science, which is our brain. How does our brain work? From where does it come from? What are its limitation? Can we make it better?

Lex Fridman 8:46 And that, in many ways, is the ultimate question that underlies this whole effort of science. So, you've made significant contributions in both the science of intelligence and the engineering of intelligence. In a hypothetical way, let me ask, how far do you think we can get in creating intelligence systems without understanding the biological, the understanding how the human brain creates intelligence? Put another way, do you think we can build a strong AI system without really, uh, getting at the core, the functional nat- uh, understanding the functional nature of the brain?

Tomaso Poggio 9:25 Well, this is a- a real difficult question. You know, we did, uh, um, solve problems like flying without, uh, really using too much of knowledge about how birds fly. It was important, I guess, to know that you could have, uh, things heavier than- than air being able to fly like- like birds. But beyond that, probably we did not learn very much, you know, some, you know. The Brothers Wright did learn a lot of observation about flo- birds and designing their- their aircraft, but, uh, you know, you can argue we did not use much biology in that particular case. Now, in the case of intelligence, I think that, um, uh, w- it's- it's a bit of a bet right now. If you a- if you ask, uh, okay, um, we- we all agree we'll get, at some point, maybe soon, maybe later, to a machine that is indistinguishable from my secretary, say, in terms of what I can ask the machine to do. I- I think we'll get there, and now the question is, and you can ask people, do you think we'll get there without any knowledge about, uh, you know, the human brain, or that the best way to get there is to understand better the human brain?

Lex Fridman 11:02 Yeah.

Tomaso Poggio 11:02 Okay. This is, I think, an educated bet that different people with different background will decide in different ways. The recent history of the progress in AI in the last, uh, I would say five years or 10 years is- has been the- the main, uh, breakthroughs, the main recent breakthroughs. I really start from neuroscience. I can mention reinforcement learning as one, is one of the algorithms at the core of AlphaGo, which is the system that beat the kind of an official world champion of Go, Lee Sedol, in two, three years ago in, uh, Seoul. Um...That's one, and that started really with the work of Pavlov, um... (laughs)

Lex Fridman 11:59 (laughs) And his dog.

Tomaso Poggio 11:59 ... 1900, Marvin Minsky in the '60s, and many ne- other neuroscientists later on. Um, and deep learning, uh, uh, started, uh, which is at the core again of AlphaGo and systems like, uh, autonomous, uh, driving systems for cars, like the systems that, uh, Mobileye, which is a company started by one of my ex-postdoc, Amnon Shashua-

Lex Fridman 12:26 Yes.

Tomaso Poggio 12:27 ... um, did. So that, that is the core of those things, and deep learning, re- really the initial ideas in terms of the architecture of these layered hierarchical networks started with work of Thorsten Wiesel and David Hubel at Harvard up the river in the '60s. So recent history suggests that neuroscience played a big role in these breakthroughs. My personal bet is that there is a good chance they continue to play a big role, maybe not in all the future breakthroughs, but in some of them.

Lex Fridman 13:03 At least in inspiration. So you-

Tomaso Poggio 13:05 At least in an inspiration, absolutely, yes.

Lex Fridman 13:07 So y- so you studied both, uh, artificial and biological neural networks. You said these, uh, mechanisms that underlie deep learning, deep, uh, and reinforcement learning, but there is nevertheless, uh, significant differences between biological and artificial neural networks as they stand now. So between the two, wh- what do you find is the most interesting, mysterious, maybe even beautiful difference as, as it currently stands in our understanding?

Tomaso Poggio 13:38 I must confess that until recently, I found the artificial networks too simplistic relative to real neural networks. But, uh, you know, recently, I've been started to think that, yes, they're a very big simplification of what you find in the brain, but on the other hand, uh, are much closer in terms of the architecture to the brain than other models that we had, that computer science used as model of thinking, which were mathematical logics, you know, LISP, Prolog, and those kind of things. (laughs)

Lex Fridman 14:19 (laughs) Yeah.

Tomaso Poggio 14:19 So in comparison to those, they're much closer to the brain. You have networks of neurons, which is what the brain is about, and the, the artificial neurons in the models are, as I said, caricature of the biological neurons, but they're still neurons, single units communicating with other units, something that is absent in, you know, the traditional, uh, computer-type models of mathematics, reasoning, and so on.

Lex Fridman 14:50 So what aspect do you th- would you like to see in artificial neural networks added over time as we try to figure out ways to improve them?

Tomaso Poggio 15:00 So one of the main differences, and, um, you know, problems, in terms of deep learning today, and it's not only deep learning, and the brain, is the need for deep learning techniques to have, uh, a lot of labeled examples. You know, for instance, for ImageNet, you have like a training set which is one million images, each one labeled by some human-

Lex Fridman 15:31 Mm-hmm.

Tomaso Poggio 15:32 ... in terms of which object is there, and, um, it's, it's clear that in biology, a baby, uh, may be able to see million of images in the first years of life, but will not have million of labels given to him or her by parents or take, take, uh, caretakers. So, uh, how do you solve that? You know, I think that there is this interesting challenge that today, uh, deep learning and related techniques are all about big data, big data meaning a lot of examples labeled by humans-

Lex Fridman 16:16 Mm-hmm.

Tomaso Poggio 16:17 ... um, whereas in, uh, nature you have, uh ... So the, the, th- this big data is n going to infinity, that's the best, you know, n meaning labeled data. But I think the biological world is more n going to one.

Lex Fridman 16:34 (laughs)

Tomaso Poggio 16:34 A- a child can learn-

Lex Fridman 16:36 It's a beautiful way to put it.

Tomaso Poggio 16:37 ... from very small number of, you know, labeled examples. Like you tell a child, "This is a car." You don't need to say, like in ImageNet, you know, "This is a car, this is a car, this is not a car, this is not a car" o- one million times. (laughs)

Lex Fridman 16:53 So... And of course, with AlphaGo and, or at least the-

Tomaso Poggio 16:57 Yeah.

Lex Fridman 16:57 ... AlphaZero variance, there's ... Because of the, because the world of Go is so simplistic, that you can actually learn by yourself through self-play, you could play against each other.

Tomaso Poggio 17:08 Yep.

Lex Fridman 17:08 And the real world, I mean, the visual system that you've studied extensively, is a lot more complicated than the game of Go. So-

Tomaso Poggio 17:16 Right.

Lex Fridman 17:16 ... uh, on the comment about children, which are fascinatingly good at learning new stuff, how much of it do you think is hardware and how much of it is software?

Tomaso Poggio 17:26 Yeah, that's a, a good, a deep question, is, in a sense, is the old question of nurture and nature.

Lex Fridman 17:33 Yeah.

Tomaso Poggio 17:33 How much is in, in the gene and how much is, uh, in the experience of an individual. Obviously, it's both that play a role, and, uh, I believe that the way evolution gives, put prior information, so to speak, hardwired, it's not really hardwired, but, um, uh, that's-Essentially an hypothesis. I think what's going on is that w- evolution, as, um, you know, almost necessarily if you believe in Darwin, is very opportunistic, and, and think about, uh, uh, our DNA and the DNA of Drosophila.

Lex Fridman 18:23 Mm-hmm.

Tomaso Poggio 18:24 Uh, our DNA does not have many more genes than Drosophila. Oh, now I'm-

Lex Fridman 18:28 The fly.

Tomaso Poggio 18:29 The fly.

Lex Fridman 18:30 Yeah.

Tomaso Poggio 18:30 The fruit fly.

Lex Fridman 18:31 Yeah.

Tomaso Poggio 18:32 Now, we know that the fruit fly does not learn very much during its individual existence. It looks like one of these machinery that it's really mostly, not 100%, but, you know, 95%, hardcoded by the genes. But since we don't have many more genes than Drosophila is, evolution could encode in us a kind of general learning machinery, and then had to give very weak priors.

Lex Fridman 19:08 Mm-hmm.

Tomaso Poggio 19:09 Um, like, for instance, let me take, give a, uh, a specific example which is a recent work by a member of our Center for Brains, Minds, and Machines. We know because of work of other people in our group and other groups that there are cells in a part of our brain, neurons-

Lex Fridman 19:29 Mm-hmm.

Tomaso Poggio 19:29 ... that are tuned to faces. They seems to be involved in face recognition. Now, this face area exist, uh, uh, seems to be present in young, uh, children and adults, um, and one question is, is there from the beginning, is hardwired by evolution, or, you know, somehow is learned very quickly?

Lex Fridman 19:55 So, what's your... By the way, a lot of the questions s- I'm asking, we, the answer is we don't really know. But as a person who has contributed s- some profound ideas in these fields, you're a good person to guess at some of these. So, of course there's a caveat before a lot of the stuff we talk about. But what is your hunch? Is the face, the part of the brain that, that seems to be concentrated on face recognition, are you born with that or are you just, it's designed to, to learn that quickly, like the face of the mother and so on?

Tomaso Poggio 20:26 My, my hunch, you know, my, my bias was the second one, learned very quickly. And, uh, turns out that Marge Livingstone at Harvard has done some, uh, amazing experiments in which she raised baby monkeys, depriving them of faces during the first weeks of life.

Lex Fridman 20:47 Mm-hmm.

Tomaso Poggio 20:48 So, they see technicians, but the technician have a mask.

Lex Fridman 20:53 Yes.

Tomaso Poggio 20:55 And, um, and so when they looked, uh, um, at, uh, the area in the brain of these monkeys that, where usually you find faces, they found no face preference. So, my guess is that what evolution does in this case is there is a plastic, an area which is plastic, which is kind of predetermined to be imprinted very easily. But the command from the gene is not a detailed circuitry for a face template.

Lex Fridman 21:32 Mm.

Tomaso Poggio 21:32 Could be. But this will require probably a lot of bits.

Lex Fridman 21:35 Mm-hmm.

Tomaso Poggio 21:36 'Cause you had to specify a lot of connection of a lot of neurons. Instead, the com- the command from the gene is something like imprint, memorize what you see most often in the first two weeks of life, especially in connection with food, and maybe nipples. (laughs) I don't know.

Lex Fridman 21:53 Right. Well, source of food.

Tomaso Poggio 21:56 Yeah.

Lex Fridman 21:56 And so... And then that area's very plastic at first-

Tomaso Poggio 21:59 Yeah.

Lex Fridman 21:59 ... and it solidifies. It'd be interesting if a variant of that experiment would show a different kind of pattern associated with food than a face pattern-

Tomaso Poggio 22:08 Right.

Lex Fridman 22:09 ... whether that could stick.

Tomaso Poggio 22:10 There are indications that during that experiment, what the monkeys saw quite often were, um, the blue gloves of the technicians that were giving to the baby monkeys the milk.

Lex Fridman 22:24 Mm-hmm.

Tomaso Poggio 22:25 And s- some of the cells, in- instead of being face-sensitive in that area, are hand-sensitive.

Lex Fridman 22:32 (laughs) Oh, that's fascinating.

Tomaso Poggio 22:34 Yep.

Lex Fridman 22:36 Can you, uh, talk about what are the different parts of the brain, and in your view, sort of loosely, and how do they contribute to intelligence? Do y- do you see the brain as a bunch of different modules and they together come in the human brain to create intelligence? Or is it all one, uh, mush of the same kind of fundamental-

Tomaso Poggio 23:00 (laughs)

Lex Fridman 23:00 ... uh, um, th-

Tomaso Poggio 23:02 Right.

Lex Fridman 23:02 ... uh, architecture?

Tomaso Poggio 23:04 Yeah. That's, um, you know, that's, uh, an important question. And, uh, w- there was a phase in, uh, neuroscience back in the 1950 or so, in which, uh, it was believed for a while that the brain was equipotential. This was the term. You could cut out a piece and, um, nothing special happened apart l- little bit less performance. There was a, um, surgeon, Lashley, who did a lot of experiments of this type with mice and rats, and concluded that every part of the brain was essentially equivalent to any other one. It turns out that that's, that's really not true. It's, uh, there are very specific modules in the brain, as you said-

Lex Fridman 24:01 Mm-hmm.

Tomaso Poggio 24:01 You know, people may lose the ability to speak if you have a stroke in a certain region, or may lose control over their legs in another region, or... so they are very specific. Th- the brain is also quite flexible and redundant, so often it can correct things, and, uh, you know, r- uh, kind of, uh, um, take over f- functions from one, uh, part of the brain to the other, but, uh, but, but really, there are s- specific modules. So, the answer that we know from this old work, uh, uh, which was basically on, based on lesions-

Lex Fridman 24:43 Mm-hmm.

Tomaso Poggio 24:43 ... either on animals or very often there were, uh, a mine of, um, well, th- there was a mine of very interesting data coming from, um, from the war, from different types of, um-

Lex Fridman 24:59 Injuries.

Tomaso Poggio 24:59 ... injuries that soldiers had in the brain. And, uh, more recently, um, functional MRI, which allow you to, to check which part of the brain are active when you are doing different tasks as, you know, r- r- can replace some of this. You can see that certain parts of the brain are involved, are active in certain tasks-

Lex Fridman 25:29 Vision, language. Yeah.

Tomaso Poggio 25:31 Yeah.

Lex Fridman 25:31 Yeah.

Tomaso Poggio 25:31 That's right.

Lex Fridman 25:32 But sort of taking a step back to that part of the brain that discovers, that, uh, specializes in the face, and how that might be learned, what's your intuition behind... you know, is it possible that the, sort of from a physicist's perspective, when you get lower and lower, that it's all the same stuff, and it just, when you're born, it's plastic, and, and it quickly figures out, "This part is gonna be about vision. This is gonna be about language. This is about common sense reasoning"? Do you have any intuition that that kind of learning is going on really quickly or is it really kind of solidified in hardware?

Tomaso Poggio 26:09 That's a great question. Uh, so there are parts of the brain, like the cerebellum or the hippocampus, that are quite different from each other. They clearly have different anatomy, different connectivity. They're... then there is, uh, the, the cortex, which is the, the most developed part of the brain in humans. And, uh, in the cortex, you have different regions of the cortex that are responsible for vision, for audition, for motor control, for language. Now, one of the big puzzles of, of this is that in the, the cortex is the cortex, is the cortex, is, is looks like it is the same in terms of, um, hardware-

Lex Fridman 27:01 Mm-hmm.

Tomaso Poggio 27:01 ... in terms of type of neurons and connectivity across these different modalities. So, for the cortex, letting aside these other parts of the brain like spinal cord, hippocampus, cerebellum and so on, for the cortex, I think your question about hardware and software and, uh, learning and so on, it's, it's, I think is rather open. And, uh, you know, it... I find very interesting, for instance, to think about an architecture, computer architecture, that is good for vision and at the same time is good for language. Seems to be, you know, so different problem, uh, areas that you have to solve.

Lex Fridman 27:49 But the underlying mechanism might be the same, and that's really instructive for-

Tomaso Poggio 27:52 It may be.

Lex Fridman 27:53 ... artificial neural networks.

Tomaso Poggio 27:55 Right.

Lex Fridman 27:55 So, you've done a lot of great work in vision, in human vision, m- computer vision, and you mentioned the problem of human vision is... will be as difficult as the problem of general intelligence, and maybe that connects to the cortex discussion. Uh, can you describe the human visual cortex and how the humans begin to understand the world, uh, through their raw sensory information? What's, uh, for folks who are not f- familiar (laughs) , especially in, on the computer vision side, we don't often actually take a step back except saying, well, the sentence or two that one is inspired by the other. Wh- wha- what is it that we know about the human visual cortex that's interesting?

Tomaso Poggio 28:40 So, we know quite a bit. At the same time, we don't know a lot. But the-

Lex Fridman 28:44 (laughs) .

Tomaso Poggio 28:44 ... the, the bit we know, you know, in a, uh, in a sense, we know a lot of the details, and, uh, um, and many we don't know, and, uh, we know a lot of the top level, um, the answer to top level question, but, uh, we don't know some basic ones, even in terms of general neuroscience, forgetting vision. You know, why do we sleep?

Lex Fridman 29:09 (laughs) .

Tomaso Poggio 29:09 (laughs) It's such a basic question, and we really don't have an answer to that. Uh-

Lex Fridman 29:14 Do you think... so taking a step back-

Tomaso Poggio 29:16 Yeah.

Lex Fridman 29:16 ... on that, so sleep, for example, is fascinating. Do you think that's a neuroscience question? Or if we talk about abstractions, what do you think is an interesting way to study intelligence or most effective on the levels of abstraction? Is it chemical? Is it biological? Is it electrophysical, mathematical, as you've done a lot of excellent work on that side, which... psychology sort of like a... which level of abstraction do you think?

Tomaso Poggio 29:43 Well, in terms of levels of, um, abstraction, I think we need all of them.

Lex Fridman 29:48 All of them.

Tomaso Poggio 29:48 It's one... uh, you know, it's like, uh, if you ask me, "What does it mean to understand a computer," right? And, and th- that's much simpler, but in a computer, I could say, "Well, I understand how to use PowerPoint."

Lex Fridman 30:03 Right.

Tomaso Poggio 30:04 ... that's my level of understanding a computer. It's, it tends reasonable, you know, it give me some power to produce slides, and beautiful slides, and ... Now, you can ask somebody else he says, "Well, I, I know how the transistor work that are inside the computer. I can write the equations for, you know, transistor, and diodes, and circuits, uh, logical circuits." And I can ask this guy, "Do you know how to operate PowerPoint?" "No idea." Right?

Lex Fridman 30:33 Yeah. So, do you think if we discovered computers walking amongst us, full of these transistors that are also, uh, operating under Windows and have PowerPoint, do you think it's ... digging in a little bit more, how useful is it to understand the transistor in order to be able to understand PowerPoint and these higher level-

Tomaso Poggio 30:58 Very good. Yes.

Lex Fridman 30:59 ... intelligent processes.

Tomaso Poggio 31:00 So, I think in the case of computers, because they were made by engineers, by us-

Lex Fridman 31:06 Mm-hmm.

Tomaso Poggio 31:06 ... this different level of understanding are rather separate on purpose.

Lex Fridman 31:11 Mm-hmm.

Tomaso Poggio 31:12 You know, you, they are separate modules so that the engineer that designed the circuit for the chips does not need to know what, uh, power, is inside PowerPoint.

Lex Fridman 31:23 Mm-hmm.

Tomaso Poggio 31:23 And somebody can write to the, the software translating from one to the end, uh, to the other end. So, um, in that case, I don't think, uh, uh, understanding the transistor help you understand PowerPoint, or very little.

Lex Fridman 31:39 Right.

Tomaso Poggio 31:40 Um, if you want to s- understand the computer, there is question, you know, I would say you have to understanding at different levels if you really-

Lex Fridman 31:47 Yeah.

Tomaso Poggio 31:47 ... want to s- to build it, one, right? (laughs) But, uh, but for the brain, I think these levels of understanding, so the algorithms, which kind of computation, you know, the equivalent of PowerPoint, and the circuits, you know, the transistors, I think they are more, much more intertwined with each other. There is not, you know, innately a level of the software separate from the hardware. And so, that's wha- why I think, in the case of the brain, the problem is more difficult and more than for computers, requires the interaction, the collaboration between different types of expertise.

Lex Fridman 32:29 So, it's a big, the brain is a big hierarchical mess-

Tomaso Poggio 32:32 Mm-hmm.

Lex Fridman 32:32 ... that you can't just un- disentangle, uh, l- levels.

Tomaso Poggio 32:35 Uh, I think you can, but is, is much more difficult, and it's not, uh, you know, it's not completely obvious. And, as I said, I think is one of the, personally I think is the greatest problem in science.

Lex Fridman 32:46 (laughs) Yeah.

Tomaso Poggio 32:47 So, you know, it, I think is, is fair that it's difficult. (laughs)

Lex Fridman 32:51 (laughs) That's a difficult one. That said, you do talk about compositionality-

Tomaso Poggio 32:56 Mm-hmm.

Lex Fridman 32:57 ... and why it might be useful, and when you discuss wha- why these neural networks, in artificial or biological sense, learn anything, you talk about c- uh, compositionality.

Tomaso Poggio 33:07 Yeah.

Lex Fridman 33:08 So, you, there's a sense that nature can be disentangled, or perpe- uh, well, all aspects of our cognition could be disentangled-

Tomaso Poggio 33:21 Mm-hmm.

Lex Fridman 33:21 ... a little, to some degree. So, why do you think, what, first of all, how do you see compositionality, and why do you think it exists at all in nature?

Tomaso Poggio 33:31 I spoke about, uh, I used the, the term compositionality when we looked at deep neural networks, multi-layers, and trying to understand when and why they are more powerful than, uh, uh, more classical one-layer networks, like s- linear classifier, uh, kernel machines, so-called. Um, and what we found is that, in terms of approximating, or learning, or representing a function, a, a mapping from an input to an output, like from an image to the label in the image, um, if this function has a particular structure-

Lex Fridman 34:20 Mm-hmm.

Tomaso Poggio 34:21 ... then deep networks are much more powerful than shallow networks to approximate the underlying f- function. And the particular structure is a structure of compositionality. If the function is made up of functions of functions, so that you need to look on, y- when you are interpreting an image, classifying an image, you don't need to look at all pixels at once, but you can compute s- something from, uh, small groups of pixels, and then you can compute something on the output of this local computation, and so on.

Lex Fridman 35:04 Yeah.

Tomaso Poggio 35:04 That is similar to what you do when you read a sentence. You don't need to read the first and the last letter, but you can read syllables, combine them in words, combine the words in sentences. So, this is this kind of structure.

Lex Fridman 35:21 So, that's as part of a discussion of why deep neural networks may be more effective than the shallow methods.

Tomaso Poggio 35:27 Yeah.

Lex Fridman 35:27 And is your sense for most things we can use neural networks for, th- those problems are going to be compositional in nature? Like, uh, like language, like vision? How far can we get in this kind of way?

Tomaso Poggio 35:47 Right. So, here is almost philosophy.

Lex Fridman 35:51 Well, let's go there. (laughs)

Tomaso Poggio 35:53 You know? (laughs) Yeah, let's go there. So, this friend of mine, Max Tegmark, who is a physicist-

Lex Fridman 35:59 Yes.

Tomaso Poggio 35:59 ... at MIT-

Lex Fridman 36:00 I've talked to him on this thing, yeah, and he disagrees with you, right?

Tomaso Poggio 36:04 Yeah. We-

Lex Fridman 36:04 A little bit.

Tomaso Poggio 36:04 ... you know, we agree on mo- most, but the conclusion is a bit different. He, he, his conclusion is that w-... for images, for instance, the compositional structure of this function that we have to learn or to solve these problems comes from physics, comes from the fact that you have local interactions in physics between m- atoms and other atoms, between particle of matter and other particles, between planets and other planets, between (laughs) stars and other ... It's all local.

Lex Fridman 36:45 Yeah.

Tomaso Poggio 36:46 Um, a- and that's true, um, but you could push this argument a bit farther. Um, not this argument actually. You could argue that, um, you know, maybe that's part of the truth, but maybe what happens is kind of the opposite, is that our brain is wired up as a deep network.

Lex Fridman 37:11 Mm-hmm.

Tomaso Poggio 37:12 So, it can learn, understand, solve problems that have this compositional structure.

Lex Fridman 37:21 Mm-hmm.

Tomaso Poggio 37:23 And it cannot do... it cannot solve problems that don't have this compositional structure. So, the problems we are accustomed to, we think about, we test our algorithms on, are this compositional structure because our brain is made up.

Lex Fridman 37:42 And that's, in a sense, an evolutionary perspective-

Tomaso Poggio 37:45 Yes.

Lex Fridman 37:45 ... that we've ... so the, the ones that didn't have, uh, th- that weren't dealing with a compositional nature of reality, uh, d- died off?

Tomaso Poggio 37:55 Yes-

Lex Fridman 37:55 Were not able

Narrator 37:56 to survive?

Tomaso Poggio 37:56 But also could be ... maybe the reason why we have this, uh, local connectivity in the brain, like, uh, simple cells in cortex looking only at a small part of the vi- image, each one of them, and then other cells looking at the small number of these simple cells, and so on. The reason for this may be purely that it was difficult to grow long-range connectivity.

Lex Fridman 38:24 Mm.

Tomaso Poggio 38:25 So, suppose it's ... you know, for biology. It's e- possible to grow short-range connectivity, but not long range also because there is a limited-

Lex Fridman 38:36 Yes.

Tomaso Poggio 38:37 ... number of long range that you c- And so, you have this, this limitation from the biology, and this means you build a deep convolutional netwo- this would be something like a deep convolutional network. And this is f- great for solving certain class of problem. These are the ones we are f- we find easy and important for our life. And yes, they were enough for us to survive. (laughs)

Lex Fridman 39:07 Yeah. And, uh, y- and you can start a successful business on solving those problems.

Tomaso Poggio 39:12 (laughs) Yeah.

Lex Fridman 39:12 Uh, right? Uh, like with Mobileye. Uh, driving is, is, is a compositional problem.

Tomaso Poggio 39:17 Right.

Lex Fridman 39:17 So, on the, on the learning task, I mean, we don't know much about how the brain learns in terms of optimization. But, uh, so the thing that's stochastic gradient descent is what, uh, artificial neural networks use for the most part to, uh, a- adjust the parameters in such a way that it's able to deal, uh, based on the label data, it- it's able to solve the problem.

Tomaso Poggio 39:42 Yeah.

Lex Fridman 39:42 So, what's your intuition about, uh, why it works at all? How hard of a problem it is to optimize a neural network, a artificial neural network? Is there other alternatives? Yeah, just in general, your intuitions behind this very simplistic algorithm that seems to do pretty good, surprisingly so.

Tomaso Poggio 40:06 Yes, s- yes. So, I find, um, neuroscience, th- the architecture of, uh, cortex is really similar to the architecture of deep networks. So, th- there is a nice correspondence there between the biology and this kind of local connectivity, hierarchical, um, architecture. The stochastic gradient descent, as you said, is, um, v- is a very simple technique. It seems pretty unlikely that biology could do that from, from what we know right now about t- n- n- you know, cortex and neurons and synapses. Um, so, uh, it's a big question open whether there are other optimization learning algorithms that can replace stochastic gradient descent. And, uh, my, my guess is yes, but nobody has found yet a real answer. I mean, people are trying, still trying, and there are some interesting ideas. The fact that, um, stochastic gradient descent is so successful, this has become clear. It's not so mysterious. And the reason is that, um, it's an interesting fact that, you know, is a change, in a sense, in, uh, how people think about statistics.

Lex Fridman 41:39 (laughs)

Tomaso Poggio 41:39 And, and this is the following, is that typically when you had, uh, data and you had, say, a model with parameters, you are trying to fit the model to the data, you know, to fit the parameter. Typically, the kind of, um, kind of, uh, crowd wisdom, uh, type idea was you should have at least, uh, you know, twice the number of data than the number of parameters, you have.

Lex Fridman 42:11 Mm-hmm.

Tomaso Poggio 42:12 Uh, maybe 10 times is better.Now, the way you train neural network these days is that they have, they have 10 or 100 times more parameters than data. Exactly the opposite. And which s- you know, i- i- it is, it has been one of the puzzles about neural networks. How can you get something that really works when you have so much freedom? In, uh, sin-

Lex Fridman 42:40 From that little data, it can generalize somehow.

Tomaso Poggio 42:43 Yeah, right, exactly.

Lex Fridman 42:44 Do you think the s- the stochastic nature of it is essential to randomness?

Tomaso Poggio 42:48 So, I think we have some initial understanding why this happens, but, um, one nice side effect of having this over-parameterization, more parameters than data, is that when you look for the minimum of a loss function, like stochastic gradi- gradient descent is doing, um, you find... I, I, I made some calculations based on some old, uh, basic theorem of algebra called Bézout's theorem, and that gives you an, a, an estimate of the number of solution of a system of polynomial equation. Anyway, the bottom line is that there are probably more minima for a typical deep networks, um, than atoms in the universe. Just to say, there are a lot. (laughs)

Lex Fridman 43:41 (laughs)

Tomaso Poggio 43:41 Because of the over-parameterization.

Lex Fridman 43:43 Yes.

Tomaso Poggio 43:44 A more global minimum, zero minimum, good minimum. So, it's not too sur-

Lex Fridman 43:50 More global minima?

Tomaso Poggio 43:51 Yeah.

Lex Fridman 43:52 Oh, okay.

Tomaso Poggio 43:52 A lot of them. So we have a lot of solutions, so it's not-

Lex Fridman 43:55 Okay.

Tomaso Poggio 43:55 ... so surprising that you can find them relatively easily. And this is-

Lex Fridman 44:01 Oh, I see.

Tomaso Poggio 44:01 This is because of the over-parameterization.

Lex Fridman 44:04 The over par- parameterization sprinkles that entire space with solutions that're pretty good.

Tomaso Poggio 44:09 Yes, yeah.

Lex Fridman 44:09 And so you-

Tomaso Poggio 44:09 It's not so surprising, right? It's like, you know, if you have a system of linear equation, and you have more unknowns than equations, then you have... We know you have an infinite number of solutions, and the question is to pick one. That's another story. But you have an infinite number of solutions, so there are a lot of, of value of your unknowns that satisfy the equations.

Lex Fridman 44:33 But it's possible that there's a lot of s- those solutions that aren't very good. Well, what's surprising is they're pretty good.

Tomaso Poggio 44:38 Right. So that's separate question. Why can you pick one that generalizes well?

Lex Fridman 44:42 Yeah, exactly.

Tomaso Poggio 44:44 That's a separate question with separate answers. Yeah.

Lex Fridman 44:47 One, one, uh, theorem that people like to talk about that kind of inspires imagination of the power of neural networks is the universality, uh, universal approximation theorem, that you can approximate any computable function with just a finite number of neurons in a single hidden layer. Do you, do you find this theorem, one, surprising? Do you find it useful, interesting, inspiring?

Tomaso Poggio 45:12 No, they, this one, uh, you know, I never found it very surprising. It's, uh, was known since the '80s, since I entered the field, because it's basically the same as Weierstrass theorem, which says that I can approximate any continuous function with a polynomial of sufficiently, with a sufficient number of terms, monomials.

Lex Fridman 45:37 Mm-hmm, yeah.

Tomaso Poggio 45:38 So, basically the same, and the proofs are very similar.

Lex Fridman 45:41 So y- your intuition was, there was never any doubt that neural-

Tomaso Poggio 45:44 Yeah.

Lex Fridman 45:44 ... networks, in theory, could-

Tomaso Poggio 45:46 Right.

Lex Fridman 45:46 ... could be very strong approximators even?

Tomaso Poggio 45:48 Right. The, the, the question, the interesting question is that if this theorem, uh, says you can approximate, fine, but when you ask how many neurons, for instance, or in the case of polynomial, how many monomials I need to get a good approximation, then it turns out that that depends on the dimensionality of your function, how many variables you have. But it depends on the dimensionality of your function in a bad way. It's, for instance, suppose you want an error which is, uh, no worse than 10% in your approximation. You come up with a network that approximate your function within 10%. Then turns out that the number of units you need are in the order of 10 to the dimensionality, D.

Lex Fridman 46:48 Mm-hmm.

Tomaso Poggio 46:48 How many variables. So if you have, you know, two variables is these two, and you have 100 units, and okay. But if you have, say, 200 by 200 pixel images, now this is, you know, tw- 40,000, whatever, and that-

Lex Fridman 47:07 We again go to the size of the universe pretty quickly.

Tomaso Poggio 47:09 Yeah, ra- exact. 10 to the 40,000 or something, and (laughs) -

Lex Fridman 47:13 Yeah. (laughs)

Tomaso Poggio 47:14 And so, uh, this is called the curse of dimensionality.

Lex Fridman 47:18 Yeah.

Tomaso Poggio 47:18 Not, you know, quite appropriately. (laughs)

Lex Fridman 47:22 (laughs) And the hope is with the extra layers, you can, uh, uh, remove the curse.

Tomaso Poggio 47:28 What we proved is that if you have deep layers or hierarchical architecture of the, with the local connectivity of the type of convolutional deep learning, and if you're dealing with a function that has this kind of, um, hierarchical architecture, then you avoid completely the curse.

Lex Fridman 47:50 (laughs) You've spoken a lot about supervised deep learning.

Tomaso Poggio 47:53 Yeah.

Lex Fridman 47:54 Uh, what are your thoughts, hopes, views on the challenges of unsupervised learning, uh, with, uh, with GANs, with, uh, generative adversarial networks? Uh, do you see those as distinct... The, the power of GANs, do you see those as distinct from supervised methods in neural networks, or are they really all in the same representation ballpark?

Tomaso Poggio 48:16 GANs is, uh, one way to get, um-... estimation of, uh, uh, probability densities, which is a somewhat new way that people have not done before. I, I don't know whether, um, this will really play an important role in, uh, you know, in intelligence or, um, it's, it's interesting. I'm, I'm less enthusiastic about it than many people in the field.

Lex Fridman 48:48 Mm-hmm.

Tomaso Poggio 48:48 I have the feeling that many people in the field are, um, really impressed by the ability to p- of producing realistic-looking images in a, in this generative way.

Lex Fridman 49:01 Which describes the popularity of the methods, but you're saying that while that's exciting and cool to look at, it may not be the tool that's useful for-

Tomaso Poggio 49:10 Yeah.

Lex Fridman 49:10 ... for... So, you described it kind of beautifully. Uh, current supervised methods go N to infinity in terms of the number of labeled points, and we really have to figure out how to go to N to one.

Tomaso Poggio 49:21 Yeah.

Lex Fridman 49:21 And you're thinking GANs might help, but they might not be the right-

Tomaso Poggio 49:25 I, I don't think in, for that problem, which I really think is important. I think they may help, uh, they certainly have applications, for instance, in computer graphics and-

Lex Fridman 49:36 Right.

Tomaso Poggio 49:36 ... you know, w- I did work long ago, which was a little bit similar in terms of, uh, saying, "Okay, I have an, a network, and, uh, I present images, and, uh, I can..." Uh, so the input is images and the output is, for instance, the pose of the image.

Lex Fridman 49:56 Mm-hmm.

Tomaso Poggio 49:57 You know, a face, how much it's smiling, if it's rotated 45 degrees or not.

Lex Fridman 50:02 Mm-hmm.

Tomaso Poggio 50:02 Uh, what about having a network that I train with the same data set, but now I invert input and output? Now, the input is the pose or the expression, a number, a set of numbers, and the output is the image, and I train it. And we did pretty good, interesting results in terms of producing very realistic-looking images. It was, um, you know-

Lex Fridman 50:29 That is-

Tomaso Poggio 50:29 ... a less sophisticated mechanism, but the, uh, output was pretty, uh, less than GANs, but the output was pretty much of the same quality. So, I think for computer graphics-type application, yeah, f- definitely GANs can be quite useful, and not only for that. For... But, um, for, you know, helping, for instance, uh, on this problem of unsupervised example of reducing the number of labeled examples, um, I think people, uh, it's like they think they can get out more than they put in. You know, it- (laughs)

Lex Fridman 51:11 There's no free lunch, as-

Tomaso Poggio 51:12 Yeah.

Lex Fridman 51:12 ... you said.

Tomaso Poggio 51:13 Right.

Lex Fridman 51:13 Uh, so what do you think... What's your intuition, um, of how can we slow the growth of N to infinity in supervise-

Tomaso Poggio 51:22 Oh-

Lex Fridman 51:22 L, uh, uh, N to infinity-

Tomaso Poggio 51:23 Oh.

Lex Fridman 51:23 ... in supervised learning? So, for example, uh, Mobileye, uh, has very successfully, I mean, essentially annotated large amounts of data to be able to drive a car. Now, one, one thought is, so we're trying to teach machines, s- s- school of AI.

Tomaso Poggio 51:40 Yeah.

Lex Fridman 51:40 And w- and we're trying to... So, how can we become better teachers maybe? That's one, one way.

Tomaso Poggio 51:47 No, you gotta, you gotta... You know, (laughs) one, I, I like that because when... Um, again, one caricature of, uh, the history of computer science, you could say, i- is, is it begins with programmers-

Lex Fridman 52:03 Mm-hmm.

Tomaso Poggio 52:03 ... expensive.

Lex Fridman 52:04 Yeah.

Tomaso Poggio 52:05 Continuous labelers, cheap.

Lex Fridman 52:08 Yeah.

Tomaso Poggio 52:09 And the future will be schools-

Lex Fridman 52:12 (laughs)

Tomaso Poggio 52:13 ... like we have for kids.

Lex Fridman 52:14 Yeah. Currently, the labeling methods, we're not selective about which examples we, we teach networks with. So, so I think the focus of making one sh- n- networks that learn much faster is often on the architecture side, but how can we pick better examples with which to learn? Uh-

Tomaso Poggio 52:38 Yeah.

Lex Fridman 52:38 Do you have intuitions about that?

Tomaso Poggio 52:39 Well, that's part of the que- uh, the part of the problem, but the other one is, um, you know, if we look at, um, biology, uh, um, a reasonable assumption, I think, is, um, in the same spirit that I said evolution is opportunistic and has weak priors. You know, the way I, I, I think, uh, the intelligence of a child, a baby may develop is, um, by bootstrapping-

Lex Fridman 53:13 Mm-hmm.

Tomaso Poggio 53:15 ... weak priors from evolution. For instance, um, in a... You can assume that you have in most organisms, including human babies, built in some basic, uh, machinery to, uh, detect motion and relative motion. S- and the fact there is, you know, we know all insects from fruit flies to other animals, they have this. Uh, even in the retinas of... In the very peripheral part of... It's very conserved across species, something that evolution discovered early. It may be the reason why babies tend to look in the first few days to moving objects and not to not-moving objects. Now, moving objects means, okay, they're attracted by motion, but motion also means that... Uh, motion gives automatic segmentation from the background.

Lex Fridman 54:20 Mm-hmm.

Tomaso Poggio 54:20 So, because of motion boundaries, you know, either the object is moving-

Lex Fridman 54:26 Mm-hmm.

Tomaso Poggio 54:26 ... or the eye of the baby is tracking the moving object, and the background is moving, right?

Lex Fridman 54:33 Yeah. So, just on, purely on the visual characteristics of the scene, that seems to be the most useful.

Tomaso Poggio 54:38 Right. So, it's like looking at an, at an, a num- an object without background.

Lex Fridman 54:43 Without background.

Tomaso Poggio 54:43 It's ideal for learning the object. Otherwise, it's really difficult, because you have so much stuff. So, suppose you do this at the beginning, first, uh, weeks. Then, after that, you can recognize object. Now they are imprinted, the number of obj-, even in the background, even without motion.

Lex Fridman 55:05 So, that's at the... By the way, I just w- wanna ask on a object recognition problem. So, there is this being responsive to movement and doing edge detection, essentially. What's the gap between being effectively, effective at visually recognizing stuff, detecting where it is, and understanding the scene? Be- is this a huge gap and many layers, or is it, are we c- is it close?

Tomaso Poggio 55:33 No, I think that's a huge gap. I think present algorithm, with all the success that we have, and the fact that are a lot of very useful... It's, I think we are, we are in a golden age for applications of, um, l- low-level vision and low-level speech recognition and so on. You know, Alexa and so on. Um, there are many more things of similar level to be done, including medical diagnosis and so on, but we are far from what we call understanding of a scene, of language, of actions, of people. Uh, that is, despite the claims, uh, that's, I think, are very far.

Lex Fridman 56:18 We're a little bit off. So, in popular culture and among many researchers, uh, some of which I've spoken with, uh, the Stuart Russell and Elon Musk, uh, uh, in and out of the AI field, uh, there's a concern about the existential threat of AI.

Tomaso Poggio 56:33 Yeah.

Lex Fridman 56:34 (laughs) And how do you think about this concern in the j- th- th- and is it valuable to think about large-scale, long-term, unintended consequences of intelligence systems we try to build?

Tomaso Poggio 56:51 I always think it's better to worry first, you know, early rather than late.

Lex Fridman 56:57 (laughs)

Tomaso Poggio 56:57 (laughs) So, uh, it's not-

Lex Fridman 56:58 So, worry is good?

Tomaso Poggio 56:59 Yeah.

Lex Fridman 57:00 Yeah.

Tomaso Poggio 57:00 I'm not against worrying at all.

Lex Fridman 57:02 Yeah.

Tomaso Poggio 57:02 Uh, personally, I think that, um, uh, uh, you know, it will take a long time before there is real reason to be worried. But as I said, I think it's, it's good to put in place and think about possible safety against, uh, uh... What I find a bit misleading are things like, um, that have been said by people I know, like Elon Musk and, uh, what is, Bostrom in particular.

Lex Fridman 57:34 Yeah, yeah.

Tomaso Poggio 57:35 U- what is his first name? I-

Lex Fridman 57:37 Uh, Nick Bostrom.

Tomaso Poggio 57:38 Nick Bostrom, right. Um, you know, and a couple of other people that, for instance, um, AI is more dangerous than nuclear weapons.

Lex Fridman 57:46 Right.

Tomaso Poggio 57:46 Yeah, I think that's really wrong, and (laughs) that can be... it's misleading, right? Because i- in terms of priority, we sh- should still be more worried about nuclear weapons and, uh, you know, what people are doing about it, and so on than AI.

Lex Fridman 58:05 And, uh, y- you've s- spoken about Demis Hassabis and yourself saying, uh, that you think it'll be about 100 years out before we have s- a general intelligence system that's on par with the human being. Do you have any updates for those predictions?

Tomaso Poggio 58:22 Well, I think he said that-

Lex Fridman 58:24 He said 20, I think you said.

Tomaso Poggio 58:25 He said 20, right. This was a couple of years ago. I have not asked him again, so should have.

Lex Fridman 58:31 Your own prediction, w- what's your prediction about when you'll be truly surprised, uh, and what's the confidence interval on that?

Tomaso Poggio 58:41 (laughs) Uh, you know, it's so difficult to predict the future and even the present sometimes. (laughs)

Lex Fridman 58:47 It's pretty hard to predict, yeah.

Tomaso Poggio 58:48 Right. But I would be... But as I said, I, this is completely-

Lex Fridman 58:53 It's-

Tomaso Poggio 58:53 ... is, I would be more like, um, uh, Rod Brooks.

Lex Fridman 58:56 Mm-hmm. (laughs)

Tomaso Poggio 58:57 I think he's about 200 years. (laughs)

Lex Fridman 58:59 200 years. (laughs) When we have this kind of AGI system, Artificial General Intelligence system, and we- you're sitting in a room with, uh, her, him, it, do you think it will be, uh, the underlying design of such a system is something we'll be able to understand, it will be simple? Do you think it will be e- explainable, uh, understandable by us? Your intuition. Again, we're in the realm of philosophy a little bit.

Tomaso Poggio 59:32 Well, probably no. But it, it, again, it depends what you really mean for understanding. So, I think, uh, you know, we don't u- understand, uh, what, how deep networks work. I think we are beginning to have a theory now.

Lex Fridman 59:55 Mm-hmm.

Tomaso Poggio 59:56 But in the case of deep networks, or even in the case of the simple, simpler kernel machines or linear classify, we really don't understand the individual units-

Lex Fridman 1:00:10 Right.

Tomaso Poggio 1:00:11 ... also. We, but we understand, you know, what the computation and the limitations and the properties of it are. Uh, it's similar to many things, you know. We-What does it mean to understand how a fusion bomb works? How many of us ... you know, many of us understand the basic principle, and some of us may understand deeper details.

Lex Fridman 1:00:40 In that sense, understanding is, as a community, as a civilization, can we build another copy of it?

Tomaso Poggio 1:00:46 Okay.

Lex Fridman 1:00:47 And in that sense, do you think there'll be, there'll need to be some evolutionary component where it runs away from our understanding? Or, do you think it could be engineered from the ground up, the same way you go from the transistor to PowerPoint?

Tomaso Poggio 1:01:02 Right. All right. So, many years ago, this was actually, let me see, 40, 41 years ago, (laughs) I wrote a paper with, uh, David Marr, who was, um, one of the founding father of computer vision, computational vision. I wrote a paper about levels of understanding, which is related to the question we discussed earlier about, uh, understanding PowerPoint, understanding transistors, and so on. And, uh, uh, you know, in that kind of fr- framework, we are at the level of the hardware-

Lex Fridman 1:01:38 Mm-hmm.

Tomaso Poggio 1:01:38 ... and the top level of the algorithms. We did not have learning. Recently, I updated, adding levels, and one level I added to those three was learning. So, and you can imagine, you could have a good understanding of how you construct learning machine, like we do, but being unable to describe in detail what the learning machines will discover.

Lex Fridman 1:02:11 Mm-hmm.

Tomaso Poggio 1:02:12 Right? Now, that would be still a powerful understanding, if I can build a learning machine, even if I don't understand in detail every time it, it learns something.

Lex Fridman 1:02:26 Just like our children, if they, if they start listening to a certain type of music, I don't know, Miley Cyrus or something, you don't understand why they came-

Tomaso Poggio 1:02:36 Yep.

Lex Fridman 1:02:36 ... to that particular preference, but you understand the learning process.

Tomaso Poggio 1:02:39 Right.

Lex Fridman 1:02:39 That's very interesting.

Tomaso Poggio 1:02:40 Yep, yep.

Lex Fridman 1:02:41 So, uh, unlearning, for systems to be part of our world, it has a certain ... One of the challenging things that you've spoken about is learning ethics, learning-

Tomaso Poggio 1:02:57 Yeah.

Lex Fridman 1:02:58 ... morals. And what, w- how hard do you think is the problem of, first of all, humans understanding our ethics? What is the origin on a neural and low level of ethics? What is it at the higher level? Is it something that's learnable for machines, in your intuition?

Tomaso Poggio 1:03:18 I think, uh, yeah, ethics is learnable, very likely. Um, I, I think I, it's one of these problems where ... I think understanding the neuroscience of ethics. You know, people discuss, there is an ethics of neuroscience.

Lex Fridman 1:03:40 (laughs)

Tomaso Poggio 1:03:40 (laughs)

Lex Fridman 1:03:41 Yeah, yes.

Tomaso Poggio 1:03:42 You know, how a neuroscientist should or should not behave. Uh-

Lex Fridman 1:03:46 Yeah.

Tomaso Poggio 1:03:46 Can you think of a neurosurgeon and the ethics that he r- he really has to obey, or she has to obey. But I'm more interested on the, on the-

Lex Fridman 1:03:56 (laughs)

Tomaso Poggio 1:03:56 ... neuroscience of ethics.

Lex Fridman 1:03:57 You're blowing my mind right now. The neuroscience of ethics is very meta.

Tomaso Poggio 1:04:01 Yeah. And, uh, you know, I think that will be important to understand also for being able to, to design machines that have, that are ethical machines, in our sense of ethics.

Lex Fridman 1:04:15 And you think there is something in neuroscience, there's patterns, tools in neuroscience that could help us, uh, shed some light on ethics? Or-

Tomaso Poggio 1:04:25 Yeah.

Lex Fridman 1:04:25 ... is it mostly on the psychologist or sociology, a much higher level?

Tomaso Poggio 1:04:29 No, there is psychology, but there is also, in the meantime, there are, um, there is evidence, fMRI, of, eh, specific areas of the brain that are involved in certain ethical judgment. And not only this, you can stimulate those area with magnetic fields and change the ethical decisions.

Lex Fridman 1:04:54 Yeah.

Tomaso Poggio 1:04:54 Okay.

Lex Fridman 1:04:55 Wow.

Tomaso Poggio 1:04:56 So, that's the work by a colleague of mine, Rebecca Sax.

Lex Fridman 1:05:00 Yeah.

Tomaso Poggio 1:05:00 And there is, uh, other s- researchers doing similar work. And I think, you know, this is the beginning that, um, ideally, at some point, we'll have an understanding of how this works and why it evolved, right? Um ...

Lex Fridman 1:05:18 The big why question. Yeah, it must have some, some purpose.

Tomaso Poggio 1:05:22 Yeah. Obviously, it has, you know, some social purposes, is, uh ... probably.

Lex Fridman 1:05:30 If neuroscience holds the key to at least illuminate some aspect of ethics, that means it could be a learnable problem.

Tomaso Poggio 1:05:37 Yep, exactly.

Lex Fridman 1:05:39 And as we're getting into harder and harder questions, let's go, uh-

Tomaso Poggio 1:05:42 (laughs)

Lex Fridman 1:05:42 ... to the hard problem of consciousness. (laughs)

Tomaso Poggio 1:05:45 Yeah.

Lex Fridman 1:05:45 Uh, is this an important problem for us to think about and solve on the engineering of intelligence side of your work, of our dream?

Tomaso Poggio 1:05:55 Uh, you know, it's unclear. So, you know, again, this is, uh, a, a deep problem, partly because it's very difficult to define consciousness. And, uh, and there is a debate, uh, among, uh, neuroscientists and, uh, about whether consciousness, and philosophers, of course, whether consciousness is something that requires bl- flesh and blood, so to speak. (laughs)

Lex Fridman 1:06:30 Yeah.

Tomaso Poggio 1:06:31 Or-... could be, you know, that we could have silicon devices that are conscious.

Lex Fridman 1:06:39 Yes.

Tomaso Poggio 1:06:41 Or up to statement like everything has some degree of consciousness and some more than others.

Lex Fridman 1:06:48 Yeah.

Tomaso Poggio 1:06:48 This is like, uh, Giulio Tononi and, uh, Phi-

Lex Fridman 1:06:53 Yeah. We just recently talked to Christof Koch.

Tomaso Poggio 1:06:56 Okay.

Lex Fridman 1:06:57 So, he- he-

Tomaso Poggio 1:06:57 Yeah, Christof was my first graduate student.

Lex Fridman 1:06:59 Yeah, yeah.

Tomaso Poggio 1:07:00 And, yeah.

Lex Fridman 1:07:00 Do you think it's important to illuminate aspects of consciousness in order to engineer intelligence systems? Do you think an intelligent system would ultimately have consciousness? Are they two, are they interlinked?

Tomaso Poggio 1:07:18 You know, most of the people working in artificial intelligence, I think, would answer, we don't strictly need consciousness to have an intelligent system.

Lex Fridman 1:07:30 That's sort of the easier question because-

Tomaso Poggio 1:07:32 Yeah.

Lex Fridman 1:07:32 ... because it's, it's a very engineering answer to the question.

Tomaso Poggio 1:07:35 Yes.

Lex Fridman 1:07:36 Pass the Turing test, we don't need consciousness. But if you were to go, do you think it's possible that we need to have so- that kind of self-awareness?

Tomaso Poggio 1:07:47 Uh, we may, yes. So for instance, I, I personally think that when test a machine or a person in a Turing test, in an extended Turing test, you know, I think consciousness is part of what we reco- wire in that test, you know, implicitly to say that this is intelligent. Christof disagrees. So-

Lex Fridman 1:08:17 Yes, he does.

Tomaso Poggio 1:08:18 Yeah.

Lex Fridman 1:08:18 He does. (laughs) It, it, despite many other romantic notions he holds-

Tomaso Poggio 1:08:23 (laughs)

Lex Fridman 1:08:23 ... he disagrees with that one.

Tomaso Poggio 1:08:24 Yes. Uh, that's right. So, you know, we'll see.

Lex Fridman 1:08:30 Do you think, as a quick question, Ernest Becker's Fear of Death, do you think mortality and those kinds of things are important for, well, for consciousness and for intelligence?

Tomaso Poggio 1:08:49 Hmm.

Lex Fridman 1:08:49 The finiteness of life, finiteness of existence? Or is that just a side effect of evol- ev- evolutionary side effect that's useful to, uh, for natural selection? Do you think this kind of thing that we're going, this interview is going to run out of time soon, our life will run out of time soon.

Tomaso Poggio 1:09:08 (laughs)

Lex Fridman 1:09:08 Do you think that's needed to make this conversation good, and (laughs) , and life good?

Tomaso Poggio 1:09:12 You know, I never thought about it. It's an, an, a very interesting question. I think, uh, uh, Steve Jobs, in his, uh, commencement speech at Stanford argued that, you know, having a finite life was important for, for stimulating achievements. And so it was a different-

Lex Fridman 1:09:31 Yeah, live every day like it's your last, right? With him.

Tomaso Poggio 1:09:34 Yeah, yeah. So, uh, rationally, I don't think strictly you need mortality for consciousness, but...

Lex Fridman 1:09:45 Who knows? They seem to go together in our biological systems, right?

Tomaso Poggio 1:09:48 Yep, yep.

Lex Fridman 1:09:51 You've mentioned, uh, before, and students are associated with, uh, you, you, the AlphaGo and Mobileye is the big recent success stories in AI, and I think it's captivated the entire world of what AI can do. So, what do you think will be the next breakthrough?

Tomaso Poggio 1:10:10 (laughs)

Lex Fridman 1:10:10 And you, what's your intuition about the next breakthrough?

Tomaso Poggio 1:10:13 Of course, I don't know where the next breakthroughs is. I, I think that, um, there is a good chance, as I said before, that the next breakthrough would also be inspired by, you know, neuroscience. But, uh, which one, I don't know.

Lex Fridman 1:10:32 And there's, so MIT has this quest for intelligence.

Tomaso Poggio 1:10:35 Yeah.

Lex Fridman 1:10:35 And there's a few moonshots which, in that spirit, which ones are you excited about? What, uh, which projects kind of, uh-

Tomaso Poggio 1:10:44 Well, of course, I'm excited about one of the moonshots with, which is our Center for Brains, Minds and Machines-

Lex Fridman 1:10:51 Yes.

Tomaso Poggio 1:10:51 ... is very, the one which is fully, fully funded, uh, by NSF. Um, and it's a, it, it is about visual intelligence. It's very-

Lex Fridman 1:11:02 And that, that one is about, uh, particularly about understanding.

Tomaso Poggio 1:11:06 Visual intelligence.

Lex Fridman 1:11:07 Uh, vis- visual intelligence.

Tomaso Poggio 1:11:07 So, the visual cortex, and, uh, and visual, uh, intelligence in the sense of how we look around ourselves and understand the world around ourselves. You know, meaning, what, what is going on, how we could, um, go from here to there without hitting obstacles. Um, you know, whether there are other agents, people in the environment. Th- these are all things that we perceive very quickly.

Lex Fridman 1:11:40 Mm-hmm.

Tomaso Poggio 1:11:41 And, uh, and it's something actually quite close to being conscious. Not quite, but, you know, there is this interesting experiment that was run at Google X.

Lex Fridman 1:11:52 Mm-hmm.

Tomaso Poggio 1:11:53 Um, which is, in a sense, is just a, a virtual reality experiment, but in which they had a subject sitting, say, in a chair with goggles like Oculus and so on.

Lex Fridman 1:12:07 Mm-hmm.

Tomaso Poggio 1:12:09 Earphones, and they were seeing through the eyes of a robot nearby, two cameras, microphones for receiving. So, their sensory system was there, right? And the impression of all the subject, very strong, they could not shake it off, was that they were where the robot was.They could look at themselves from the robot and still feel they were, they were where the robot is. They were looking at their body. Their self wa- had moved.

Lex Fridman 1:12:48 So, some aspect of scene understanding has to have ability to place yourself, uh, have a self-awareness about your position in the world, and-

Tomaso Poggio 1:12:57 Yeah.

Lex Fridman 1:12:58 ... what the world is.

Tomaso Poggio 1:12:59 Right.

Lex Fridman 1:12:59 So-

Tomaso Poggio 1:13:00 Yeah.

Lex Fridman 1:13:01 So, we may have to solve the hard problem of consciousness (laughs) to solve the-

Tomaso Poggio 1:13:04 On their way, yes. (laughs)

Lex Fridman 1:13:05 It's quite, it's quite, qui- quite a moonshot.

Tomaso Poggio 1:13:07 Right, right.

Lex Fridman 1:13:07 So, you've been an, an advisor to some incredible minds, uh, including Demis Hassabis, Christof Koch, Amna Shashua, like you said. All went o- on to become seminal figures in their respective fields. From your own success as a researcher, and f- from perspective as a mentor of these researchers, having guided them, m- h- in the way of advice, what does it take to be successful in science and engineering careers? Whether you're talking to somebody in their teens, 20s, and 30s, what does that path look like?

Tomaso Poggio 1:13:48 It's curiosity and having fun. And, uh, I think is important also, having fun w- with other curious minds.

Lex Fridman 1:14:02 It's the, uh, the people you surround with too, so-

Tomaso Poggio 1:14:04 Yeah.

Lex Fridman 1:14:05 ... fun and curiosity. Is there, f- you mentioned Steve Jobs, is there also an underlying ambition that's unique that you saw? Or is it really does boil down to insatiable curiosity and fun?

Tomaso Poggio 1:14:18 Well, of course, eh, you know, it's being c- curious in a active and ambitious way, yes. And, uh, um, definitely. But I think, uh, sometime in, in, uh, science, there are friends of mine who are like this. Um, you know, there are some of the scientists like to work by themselves and kind of communicate, um, only when they completed their work or discover something. Um, I think I always found the, the actual process of, uh, you know, discovering something is more fun if it's together with other intelligent and curious and fun people.

Lex Fridman 1:15:09 So, if you see the fun in that process, ult- the side effect of that process will be that you'll actually end up discovering some interesting things.

Tomaso Poggio 1:15:15 Yeah, yes.

Lex Fridman 1:15:16 So, as, uh, you've led, uh, uh, uh, many incredible efforts here, what's the secret to being a good advisor, mentor, leader in a research setting?

Tomaso Poggio 1:15:28 Oh, th-

Lex Fridman 1:15:28 Is it a similar spirit? Or, yeah, what's, what, what advice could you give to people, young faculty and so on?

Tomaso Poggio 1:15:36 It's partly repeating what I said about-

Lex Fridman 1:15:39 Mm-hmm.

Tomaso Poggio 1:15:39 ... an environment that should be friendly and fun and, uh, ambitious. And, uh, you know, I, I think I learned a lot from some of my advisors and friends, and some were physicists, and, uh, there was, for instance, this, um, um, behavior that was encourage of when somebody comes with a new idea in the group, you are, unless it's really stupid, but you are always enthusiastic. And then, and you're enthusiastic for a few minutes, for a few hours. Then you start, you know, asking critically a few questions-

Lex Fridman 1:16:20 Yeah.

Tomaso Poggio 1:16:20 ... to in- te- like, testing this. But, you know, this is a process that is, I think is very, very good, this. You have to be enthusiast- Sometime people are very critical from the beginning. That's, that's, that's not-

Lex Fridman 1:16:36 You have to, you have to give it a chance for that-

Tomaso Poggio 1:16:37 Yes.

Lex Fridman 1:16:38 ... for that seed to grow.

Tomaso Poggio 1:16:39 Yeah.

Lex Fridman 1:16:39 That said, s- with some of your ideas, which are quite revolutionary, so there's... I've, I witnessed, especially in the human vision side and neuroscience side, there could be some pretty heated arguments. Um, do you enjoy these? Is that a part of science and acad-

Tomaso Poggio 1:16:53 Yeah.

Lex Fridman 1:16:53 ... eca- academic pursuits that you enjoy?

Tomaso Poggio 1:16:55 Yeah.

Lex Fridman 1:16:56 Is it... (laughs) Is that something that happens in your group as well? Uh-

Tomaso Poggio 1:17:01 Yeah, absolutely. I also spent some time in Germany. Again, there is this tradition in which people are more, uh, forthright, less kind than here.

Lex Fridman 1:17:13 Yeah.

Tomaso Poggio 1:17:14 So, you know, in the US, y- you, when you write a bad letter, you still say, "This guy is nice." You know? (laughs)

Lex Fridman 1:17:23 Yes, yes.

Tomaso Poggio 1:17:24 (laughs) And so, I th-

Lex Fridman 1:17:26 Yeah, here in America, it's degrees of nice. Uh-

Tomaso Poggio 1:17:28 Yes.

Lex Fridman 1:17:29 ... (laughs) it's, it's all just degrees of nice, yeah.

Tomaso Poggio 1:17:31 Right, right. So, as long as this does not become personal, and it's really like, you know, a football game with its rules, that's great. (laughs)

Lex Fridman 1:17:44 (laughs) And it's fun. So, so if you somehow found yourself in a position to ask one question of an oracle, like a genie maybe, a god, wh- and you're guaranteed to get a clear answer, what kinda question would you ask? What, what would be the question you would ask?

Tomaso Poggio 1:18:04 In the spirit of our discussion, it could be how could be, how could I become 10 times more intelligent? (laughs)

Lex Fridman 1:18:10 (laughs) And so, but see, you only get a clear, short answer. So, do you think there's a clear, short answer to that?

Tomaso Poggio 1:18:18 No.

Lex Fridman 1:18:19 (laughs)

Tomaso Poggio 1:18:20 (laughs)

Lex Fridman 1:18:20 And that's the answer you'll get. (laughs) okay. So, you've mentioned, uh, Flowers of, uh, Algernon.

Tomaso Poggio 1:18:26 Oh, yeah.

Lex Fridman 1:18:28 As a s- a story that inspired you in your, in your childhood. As this, uh, story of a mouse and human achieving genius-level intelligence, and then understanding what was happening while slowly becoming not intelligent again, and this tragedy of gaining intelligence and losing intelligence. Do you think, in that spirit, in that story, do you think intelligence is a gift or a curse from the perspective of happiness and meaning of life? So, you, you try to create an intelligent system that understands the universe, but on an individual level, the meaning of life, do you think intelligence is a gift?

Tomaso Poggio 1:19:10 That's a good question. I don't know.

Lex Fridman 1:19:22 As one of the s- as one w- people consider the smartest people in the world, (laughs) in some, in some dimension, at the very least, uh, what do you think?

Tomaso Poggio 1:19:34 I don't know. It may be invariant to intelligence, the degree of happiness. Would be nice if it were.

Lex Fridman 1:19:43 That's the hope.

Tomaso Poggio 1:19:44 Yeah.

Lex Fridman 1:19:46 You could be smart and happy, and clueless and happy.

Tomaso Poggio 1:19:50 Yeah. I-

Lex Fridman 1:19:51 As always, on the discussion of the meaning of life, it's probably a good place to end. Tomaso, thank you so much for talking today.

Tomaso Poggio 1:19:59 Thank you. This was great.