Vladimir Vapnik: Statistical Learning | Lex Fridman Podcast #5

1. 0:00 – 4:08

   Instrumentalism vs. realism: prediction rules or “God’s laws”

   1. LFLex Fridman0:00

      The following is a conversation with Vladimir Vapnik. He's the co-inventor of support vector machines, support vector clustering, VC theory, and many foundational ideas in statistical learning. He was born in the Soviet Union and worked at the Institute of Control Sciences in Moscow. Then, in the United States, he worked at AT&T, NEC labs, Facebook Research, and now is a professor at Columbia University. His work has been cited over 170,000 times. He has some very interesting ideas about artificial intelligence and the nature of learning, especially on limits of our current approaches and the open problems in the field. This conversation is part of MIT course on Artificial General Intelligence and the Artificial Intelligence podcast. If you enjoy it, please subscribe on YouTube or rate it on iTunes or your podcast provider of choice, or simply connect with me on Twitter or other social networks @lexfridman, spelled F-R-I-D. And now, here's my conversation with Vladimir Vapnik. Einstein famously said that God doesn't play dice.

   2. VVVladimir Vapnik1:08

      Yeah.

   3. LFLex Fridman1:10

      You have studied the world through the eyes of statistics, so let me ask you in terms of the nature of reality, fundamental nature of reality, does God play dice?

   4. VVVladimir Vapnik1:21

      We don't know some factors, and because we don't know some factors which could be important, it looks like God play dice, but we only should describe. In philosophy, they distinguish between two positions, positions of instrumentalism, where you're creating theory for prediction, and position of realism, where you're trying to understand what God did.

   5. LFLex Fridman1:54

      Can you describe instrumentalism and realism a little bit?

   6. VVVladimir Vapnik1:58

      For example, if you have some mechanical laws, what is that? Is it law which true always and everywhere or it is law which allow you to predict position of moving element? The, uh, what, what you believe, you believe that it is God's law, that God created the world which adhere to this physical law-

   7. LFLex Fridman2:33

      Yeah.

   8. VVVladimir Vapnik2:33

      ... or it is just law for predictions?

   9. LFLex Fridman2:36

      And which one is instrumentalism?

   10. VVVladimir Vapnik2:38

       For predictions.

   11. LFLex Fridman2:39

       Just predict.

   12. VVVladimir Vapnik2:39

       If you believe that this is law of God-

   13. LFLex Fridman2:43

       Mm-hmm.

   14. VVVladimir Vapnik2:43

       ... and it's always true everywhere, that means that you're realist.

   15. LFLex Fridman2:50

       So you-

   16. VVVladimir Vapnik2:50

       You're trying to re- to really, uh, understood, understand the God's thought.

   17. LFLex Fridman2:55

       So the way you see the world is, is an instrumentalist?

   18. VVVladimir Vapnik3:00

       You know-

   19. LFLex Fridman3:00

       Absolutely.

   20. VVVladimir Vapnik3:00

       ... I'm working for some models, model of, uh, machine learning. So in this model, we can see, um, setting and we try to solve, resolve the setting to solve the problem. And you can do it in two different way. From the point of view of instrumentalist, and that's what everybody does now, because, uh, they say the goal of machine learning is to, uh, find the rule for classification.

   21. LFLex Fridman3:36

       Mm-hmm.

   22. VVVladimir Vapnik3:37

       That is true, but it is instrument for prediction. But I can say the goal of, uh, machine learning is to, to learn about conditional probability, so how God play deuce, and he... if he play, what is probability for one, what is probability for another given situation. But for prediction, I don't need this. I need the rule.

   23. LFLex Fridman4:04

       The rule.

   24. VVVladimir Vapnik4:04

       But for understanding, I need conditional probability.

2. 4:08 – 7:48

   Mathematics as the language of reality and the discipline of equations

   1. LFLex Fridman4:08

      So let me just step back a little bit first to talk about, you mentioned, uh, which I read last night, the, the parts of the 1960 paper by Eugene, uh, Wigner-

   2. VVVladimir Vapnik4:20

      Yeah.

   3. LFLex Fridman4:20

      ... uh, unreasonable effectiveness of mathematics in natural sciences. Such a, such a beautiful paper-

   4. VVVladimir Vapnik4:28

      Yeah. Absolutely.

   5. LFLex Fridman4:28

      ... by the way. It made me feel, uh, to be honest, to confess my own work in the past few years on deep learning heavily applied, made me f- feel that I was missing out on some of the beauty of nature i- in a way that math can uncover. So let me just s- step away from the, the poetry of that for a second. How do you see the role of math in your life? Is it, is it a tool? Is it poetry? Where, where does it sit and does math for you have limits of what it can describe?

   6. VVVladimir Vapnik5:01

      Some people saying that math is language which use God. So I believe exactly-

   7. LFLex Fridman5:10

      And speak to God or use God or-

   8. VVVladimir Vapnik5:12

      Use God.

   9. LFLex Fridman5:12

      ... use God.

   10. VVVladimir Vapnik5:14

       Yeah. So I believe that this article about effectiveness, unreasonable effectiveness of maths is that if you're looking at mathematical structures, they know something about reality. And the most scientists from natural science, they're looking on equation and trying to understand reality, so the same with machine learning. If you're trying very carefully look on all equations which define conditional probability, you can understand something...... about reality more than from your fantasy.

   11. LFLex Fridman6:08

       So math can reveal the simple underlying principles of reality, perhaps.

   12. VVVladimir Vapnik6:13

       You know, what means simple? It is very hard to discover them. But then, when you discover them and look at them, you see how beautiful they are and, and it is surprising why people did not see that before. You're looking an equation and derive it from equations. For example, I talk yesterday-

   13. LFLex Fridman6:40

       Mm-hmm.

   14. VVVladimir Vapnik6:41

       ... about least squares method and people had a lot of fun as you have to improve least squares method. But if you look going step by step by solving some equations, you'll suddenly, you get some term which after thinking, you understands it, it describe position of observation point. In least squares method, we throw out a lot of information, we don't look in composition of points of observations, we're looking only on residuals.

   15. LFLex Fridman7:14

       Mm-hmm.

   16. VVVladimir Vapnik7:14

       But when you understood that, that's very simple idea but which not too simple to understand and you can derive this just from equations.

   17. LFLex Fridman7:25

       So some simple algebras, a few steps will take you to something surprising that when you think about-

   18. VVVladimir Vapnik7:32

       Absolutely, yes.

   19. LFLex Fridman7:34

       ... you understand.

   20. VVVladimir Vapnik7:34

       So here, and that is proof that human intuition not to reach and very primitive and it does not see very s- simple situations.

3. 7:48 – 9:59

   Why intuition misleads—and what “brilliance” really is

   1. LFLex Fridman7:48

      So, uh, let me just take a step back. Eh, in general, y- yes, right? Uh, but what about human, as opposed to intuition, ingenuity, um, m- moments of brilliance? So y- uh, are you so, uh... Do you have to be so hard on human intuition? Are there moments of brilliance in human intuition that can leap ahead of math and then the math will catch up?

   2. VVVladimir Vapnik8:17

      I don't think so. I think that the, z- the best human intuition, it is putting in axioms and then it is technical where ƒ-

   3. LFLex Fridman8:28

      See where the axioms take you.

   4. VVVladimir Vapnik8:30

      Yeah.

   5. LFLex Fridman8:31

      S-

   6. VVVladimir Vapnik8:31

      But if they correctly take axioms, but it, axiom polished during generations of scientists and this is integral wisdom.

   7. LFLex Fridman8:45

      So, (laughs) that's beautifully put. But if you, uh, maybe look at... When you, when you think of Einstein and, uh, special relativity, uh, what is the role of imagination coming first there in the moment of discovery of an idea? So there's obviously a mix of math and out of the box imagination there.

   8. VVVladimir Vapnik9:10

      That, I don't know. Whatever I did, I exclude any imagination because whatever I saw in machine learning that come from imagination, like features-

   9. LFLex Fridman9:24

      Mm-hmm.

   10. VVVladimir Vapnik9:24

       ... like deep learning-

   11. LFLex Fridman9:25

       Mm-hmm.

   12. VVVladimir Vapnik9:26

       ... they are not really want of the problem. When you're looking very carefully for mathematical equations, you deriving very simple theory which goes far beyond theoretically than whatever people can imagine because it is not good fantasy.

   13. LFLex Fridman9:44

       Yeah.

   14. VVVladimir Vapnik9:44

       It is just interpretation. It is just fantasy but it is not what you need. You don't need any imagination to derive, uh, say, main principle of machine learning.

4. 9:59 – 11:33

   Interpretation pitfalls: the microscope analogy and the risk with brains

   1. LFLex Fridman9:59

      Yeah. When you think about learning and intelligence, maybe thinking about the human brain and trying to describe mathematically the process of learning, uh, that is something like what happens in the human brain, do you think we have the tools currently? Do you think we will ever have the tools to try to describe that process of learning?

   2. VVVladimir Vapnik10:21

      You... It is not description what's going on, it is interpretation. It is your interpretation. Your vision can be wrong. You know when guy invent microscope-

   3. LFLex Fridman10:35

      Mm-hmm.

   4. VVVladimir Vapnik10:36

      ... Leeuwenhoek, for the first time, only he got this instrument and nobody... when he kept secret about microscope. But he wrote report in London Academy of Science. In his report, when he looking at the blood, he look everywhere, on the water, on the blood, on the steam, but he described blood like fight-

   5. LFLex Fridman10:59

      Mm-hmm.

   6. VVVladimir Vapnik11:00

      ... between queen and king.

   7. LFLex Fridman11:03

      Mm-hmm.

   8. VVVladimir Vapnik11:04

      So he saw blood cells, red cells, and he imagines that it is army fighting each other and it was his interpretation of situation. And he said that, this report in Academy of Science, they, uh, very carefully looked because they believed that he's wro- he's right, he saw something.

   9. LFLex Fridman11:25

      Yes.

   10. VVVladimir Vapnik11:25

       But he gave wrong interpretation and I believe the same can happen with brain.

   11. LFLex Fridman11:32

       With brain, yeah.

5. 11:33 – 12:49

   The ‘great teacher’ problem: predicates, invariants, and learning faster

   1. VVVladimir Vapnik11:33

      Because the most important part... You know, I believe in human, uh, language. In some proverb is so much wisdom. For example, people say that it is better than a thousand days of diligent studies one day with great teacher.

   2. LFLex Fridman11:54

      Mm-hmm.

   3. VVVladimir Vapnik11:54

      But if it i- I will ask you what teacher does, nobody knows and that is intelligence. And, but we know from history and, uh, now from, from mass in machine learning that-... teacher can do a lot.

   4. LFLex Fridman12:12

      So what, from a mathematical point of view, is the great teacher?

   5. VVVladimir Vapnik12:16

      I don't know.

   6. LFLex Fridman12:17

      That's an open question.

   7. VVVladimir Vapnik12:18

      The, the, no, uh, but we can, uh, say what teacher can do. He can introduce some, uh, invariants, some predicate for creating invariants. How he doing it, I don't know, because teacher knows reality and can describe from this reality a predicate invariant. But he knows that when you're using invariant, you can decrease number of observations 100 times. That's-

6. 12:49 – 18:17

   ‘Play like a butterfly’ and the duck test: what makes a predicate useful

   1. LFLex Fridman12:49

      So, but (laughs) uh, maybe try to pull that apart a little bit. I think you mentioned, uh, like a piano teacher saying, uh, uh, to the student, "Play like a butterfly."

   2. VVVladimir Vapnik12:59

      Yeah.

   3. LFLex Fridman12:59

      Right? I played piano, I played guitar for a long time and, and, and, yeah, that's... there's, um... maybe it's romantic, poetic, but it feels like there's a lot of truth in that statement. Like there's, there is a lot of instruction in that statement. And so c- c- can you pull that apart? What i- what, what is that? I- the language itself may not contain this information.

   4. VVVladimir Vapnik13:22

      It is, it is not blah, blah, blah, because it affect, affect you.

   5. LFLex Fridman13:24

      It is not blah, blah, blah, yeah.

   6. VVVladimir Vapnik13:25

      Affect you.

   7. LFLex Fridman13:27

      It's what?

   8. VVVladimir Vapnik13:27

      Affect you.

   9. LFLex Fridman13:28

      Yeah.

   10. VVVladimir Vapnik13:28

       Affect your playing.

   11. LFLex Fridman13:29

       Yes, it does, but wha- it's not the lang... It's- I- if- it feels like a, um... What is the information being exchanged there? Wha- what is the nature of information? What is the representation of that information?

   12. VVVladimir Vapnik13:42

       I believe that it is sort of predicate, but I don't know.

   13. LFLex Fridman13:44

       Some kind of predicate.

   14. VVVladimir Vapnik13:45

       That is exactly what, what intelligence in machine learning should be.

   15. LFLex Fridman13:49

       Yes.

   16. VVVladimir Vapnik13:50

       Because the rest is just mathematical technique. I think that, uh, what was discovered recently is that there is two type, two mechanism of learning. Uh, one called strong convergence mechanism.

   17. LFLex Fridman14:05

       Mm-hmm.

   18. VVVladimir Vapnik14:06

       And weak convergence mechanism. Before, people use only one convergence. In weak convergence mechanism, you can use predicate, that's what play like butterfly-

   19. LFLex Fridman14:18

       Mm-hmm.

   20. VVVladimir Vapnik14:19

       ... and, uh, it will immediately affect your playing. You know there's, there's English proverb great?

   21. LFLex Fridman14:27

       Mm-hmm.

   22. VVVladimir Vapnik14:27

       If it looks like a duck, swims like a duck, and quack like a duck, then it is probably duck.

   23. LFLex Fridman14:35

       Yes.

   24. VVVladimir Vapnik14:36

       But this is exact a- a- about predicates. Looks like a duck, what it means. So you saw many ducks that you're training data, so you, you have description of how, how looks, integral looks ducks.

   25. LFLex Fridman14:56

       Yeah, the visual characteristics of a duck. Yeah.

   26. VVVladimir Vapnik14:59

       Uh, uh, yeah. But you want... and you have model for recognition now, so you would like so that theoretical description from model coincide this empirical description which you saw on

   27. NANarrator15:13

       Okay.

   28. VVVladimir Vapnik15:13

       ... text there.

   29. LFLex Fridman15:13

       Mm-hmm.

   30. VVVladimir Vapnik15:14

       So about looks like a duck, it is general. But what about swims like a duck? You should know that duck swims. You can't say, "It play chess like a duck." Okay, duck doesn't play chess. And it is completely legal predicate, but it is useless. So how teacher can recognize not useless predicate? So up to now, we don't use this predicate in existing machine learning.
7. 18:17 – 20:02

   Admissible function sets, capacity, and VC dimension (explaining ‘VC’)

   1. LFLex Fridman18:17

      So you talk about admissible set of functions and you talk about good functions. So what makes a good function?

   2. VVVladimir Vapnik18:24

      So admissible set of function is set of function which has small capacity or small diversity, small VC dimension exactly-

   3. LFLex Fridman18:34

      Mm-hmm.

   4. VVVladimir Vapnik18:35

      ... which contain good function inside.

   5. LFLex Fridman18:37

      So, by the way, for people who don't know VC, um, you're the V in the VC. (laughs) Uh-

   6. VVVladimir Vapnik18:44

      Yeah.

   7. LFLex Fridman18:44

      ... uh, so, uh, w- how would you describe to a lay person what VC Theory is? Um, how would you describe VC?

   8. VVVladimir Vapnik18:51

      So when you have a machine... (clears throat) So machine capable to pick up one function from the admissible set of function.

   9. LFLex Fridman19:01

      Mm-hmm.

   10. VVVladimir Vapnik19:02

       But set of admissibles function can be big, say contain all continuous functions then it's useless. You don't have so many examples to pick up function. But it can be small, small, uh, we call it capacity but maybe better call diversity. So not very different function in the set, it's, uh, infinite set of function but not very diverse. So if it's small VC dimension, when VC dimension is small, you need not... you, you need small amount of training data. So the goal is to create admissible set of functions which is have small VC dimension and contain good function. Then you should, then you will be able to pick up the function using small amount of observations.

8. 20:02 – 23:00

   What learning theory omits: creating the admissible set via invariants

   1. LFLex Fridman20:02

      So that is the task of, uh, learning?

   2. VVVladimir Vapnik20:06

      Yeah.

   3. LFLex Fridman20:06

      Is creating a set of admissible functions that has a small VC dimension and then you've figured out a clever way of picking up-

   4. VVVladimir Vapnik20:17

      No. That-

   5. LFLex Fridman20:18

      ... the good-

   6. VVVladimir Vapnik20:18

      That is goal of learning which I fo- formulated yesterday.

   7. LFLex Fridman20:21

      Yeah.

   8. VVVladimir Vapnik20:22

      Statistical learning theory does not involve in creating admissible set of function. In classical learning theory, everywhere, 100% in textbook, the set of function, admissible set of function is given.

   9. LFLex Fridman20:38

      Mm-hmm.

   10. VVVladimir Vapnik20:39

       But this is science about nothing, because the most difficult problem to create admissible set of functions given, say, a lot of functions, continuum set of function, create admissible set of functions that means that it has finite VC dimensions, small VC dimension, and contain good function. So this was out of consideration. So-

   11. LFLex Fridman21:05

       So how, what's the process of doing that? I mean, it's fascinating. What is the process of creating this, um, admissible set of functions?

   12. VVVladimir Vapnik21:13

       That is invariants.

   13. LFLex Fridman21:14

       That's invariants?

   14. VVVladimir Vapnik21:15

       Yeah.

   15. LFLex Fridman21:15

       Can you describe invariants-

   16. VVVladimir Vapnik21:17

       Yeah, you-

   17. LFLex Fridman21:17

       ... you spoke about yesterday?

   18. VVVladimir Vapnik21:18

       ... you're looking of properties of training data and, uh, properties means that you, uh, say have some function and you ne- you, you just count what is value, average value of function on training data. (clears throat) You have model and what is expectation of this function on the model-

   19. LFLex Fridman21:44

       Mm-hmm.

   20. VVVladimir Vapnik21:44

       ... and they should coincide. So the, the problem is about have to pick up functions. It can be any function. It... in, in, in fact, it, it is true for all functions but because when we're talking set, uh, say, um, duck does not jumping-

   21. LFLex Fridman22:07

       Mm-hmm.

   22. VVVladimir Vapnik22:08

       ... so you don't ask question, jump like a duck, because it is trivial. It doesn't jumping and doesn't help you to recognize jump. But you know something, which question to ask and you're asking it swims like a j- like a duck. But looks like a duck, it is general situation.

   23. LFLex Fridman22:27

       Mm-hmm.

   24. VVVladimir Vapnik22:28

       Looks like, say, guy who have this illness, this, this disease, it-

   25. LFLex Fridman22:36

       Yeah.

   26. VVVladimir Vapnik22:36

       It is legal.

   27. LFLex Fridman22:38

       Yeah.

   28. VVVladimir Vapnik22:39

       So there is a, a general type of predicate looks like-

   29. LFLex Fridman22:44

       Yeah.

   30. VVVladimir Vapnik22:44

       ... and speci- special type of predicate which related to this specific problem. And that is intelligence part of all this business and that's where teacher is involved.
9. 23:00 – 27:57

   Deep learning critique: ‘interpretation’ over math, and why shallow can suffice

   1. LFLex Fridman23:00

      Okay. What do you think about deep learning as, as, um, neural networks, these arbitrary architectures as helping accomplish some of the tasks you're thinking about? Their effectiveness or lack thereof, what are, what are the weaknesses and what are the possible strengths?

   2. VVVladimir Vapnik23:20

      You know, I think that this is fantasy. Everything which... like deep learning, like features. Let me give you this example.

   3. LFLex Fridman23:32

      Mm-hmm.

   4. VVVladimir Vapnik23:33

      Uh, one of the greatest book is Churchill book about history of Second World War.

   5. LFLex Fridman23:38

      Mm-hmm.

   6. VVVladimir Vapnik23:39

      And he's starting this book describing that in old time when war is over, so the great kings, they gather together, almost all of them were relatives, and they discussed what should be done, how to create peace. And they came to agreement. And what h- when happens First World War, the general public came in power and they were so greedy that robbed Germany.

   7. LFLex Fridman24:17

      Mm-hmm.

   8. VVVladimir Vapnik24:18

      And it was clear for everybody that it is not peace, that peace will last only 20 years.... because they was not professionals, and the same I see in machine learning.

   9. LFLex Fridman24:32

      Right.

   10. VVVladimir Vapnik24:32

       There are mathematicians who are looking for the problem from a very deep point of view, a mathematical point of view, and there are, uh, computer scientists-

   11. LFLex Fridman24:43

       Mm-hmm.

   12. VVVladimir Vapnik24:43

       ... which mostly does not know mathematics. They just have interpretation of that, and they invented a lot of blah, blah, blah interpretations, like deep learning. Why you did deep learning? Mathemat- does not know deep learning. Mathematic does not know, uh, neurons. It is just function.

   13. LFLex Fridman25:02

       It's just-

   14. VVVladimir Vapnik25:02

       If you like to say-

   15. LFLex Fridman25:03

       Yeah.

   16. VVVladimir Vapnik25:03

       ... piecewise linear function, say that, and do in k- in class of piecewise linear function. But they invent something and then they try to- to- to- to prove advantage of that through interpretations, which mostly wrong, and when it b-

   17. LFLex Fridman25:22

       With the kings and-

   18. VVVladimir Vapnik25:23

       ... not enough, they- they appeal to brain which they know nothing about that. Nobody knows what's going on in the brain. So I think that more reliable, work on maths. This is mathematical problem.

   19. LFLex Fridman25:36

       Right.

   20. VVVladimir Vapnik25:36

       Do your best to solve this problem. Try to understand that there is no only one way of convergence-

   21. LFLex Fridman25:43

       Mm-hmm.

   22. VVVladimir Vapnik25:44

       ... which is strong way of convergence. There is a weak way of convergence which requires predicate. And if you will go through all this stuff, you will see that you don't need deep learning. Even more, I would say one of the theorem which is called representer theory-

   23. LFLex Fridman26:03

       Mm-hmm.

   24. VVVladimir Vapnik26:03

       ... it says that optimal solution of mathematical problem which is- which describe learning-

   25. LFLex Fridman26:14

       Mm-hmm.

   26. VVVladimir Vapnik26:14

       ... is on shadow network-

   27. LFLex Fridman26:19

       And-

   28. VVVladimir Vapnik26:19

       ... not on deep learning.

   29. LFLex Fridman26:21

       And a shallow network, yeah.

   30. VVVladimir Vapnik26:22

       Yeah, shallow network.
10. 27:57 – 31:09

    AlphaGo and problem difficulty: success doesn’t imply understanding

    1. LFLex Fridman27:57

       about the success of a system like AlphaGo at beating the game of Go, using, uh, neural networks to estimate th- the quality of a bo- of a board and- and- and- and the quality of the positions?

    2. VVVladimir Vapnik28:11

       That is your interpretation, quality of the board.

    3. LFLex Fridman28:14

       Yeah. Yes. (laughs)

    4. VVVladimir Vapnik28:15

       Yeah. (laughs)

    5. LFLex Fridman28:17

       But it wo- so it's not our interpretation. The fact is, a neural network system, doesn't matter, a learning system that we don't, I think, mathematically understand that well beats the best human player. It does something that was thought impossible.

    6. VVVladimir Vapnik28:31

       That means that it's not very difficult problem. That's it.

    7. LFLex Fridman28:34

       That's- that's- so you empiric- we've empirically have discovered that this is not a very difficult problem, yeah. (laughs)

    8. VVVladimir Vapnik28:42

       Yeah.

    9. LFLex Fridman28:42

       It's true. Uh, so maybe it, uh, (laughs) can't argue. Uh, so w- what-

    10. VVVladimir Vapnik28:50

        Even more, I would say that if they use deep learning, it is not the most effective way of learning theory. And usually, when people use deep learning, they're using zillions of training data.

    11. LFLex Fridman29:09

        Yeah.

    12. VVVladimir Vapnik29:10

        Yeah, but you don't need this. So I describe challenge. Can we do some problems which do it well, deep learning method, this deep net, uh, using hundred times less training data? Even more, some problems deep learning cannot solve, because it's not necessary they create admissible set of function when they... To create deep architecture means to create admissible set of functions. You cannot say that you're creating good admissible set of functions. You're just- it's your fantasy. It does not comes from maths. But it is possible to create admissible set of functions because you have your training data. Uh, actually, for mathematicians, uh, when you're considering variant, you need to use law of large numbers. When you're making training in existing algorithm, you need uniform law of large numbers-

    13. LFLex Fridman30:20

        Mm-hmm.

    14. VVVladimir Vapnik30:20

        ... which is much more difficult (‎;…) ‎and requires (‎;...) dimension and all this stuff. But nevertheless-... if you use both weak and strong way of convergence, you can decrease a lot of training data.

    15. LFLex Fridman30:35

        Yeah. You could do the- the three, the swims like a duck, uh, and quacks like a duck.

    16. VVVladimir Vapnik30:39

        Yeah, yeah.

    17. LFLex Fridman30:39

        But our... So let- let's- let's step back and, um, think about intel- human intelligence in general. And clearly, that has evolved in a non-mathematical way. (laughs) It wa- it wasn't, uh... As far as we know, uh, God, uh, or- or- or whoever, didn't, uh, come up with a model, um, and place it in our brain of admissible functions. It kind of evolved. I don't know. Maybe you have a view on this. But,

11. 31:09 – 33:27

    Can machines think? Imitation vs intelligence, and intelligence ‘outside us’

    1. LFLex Fridman31:09

       so Alan Turing in the '50s, in- in his paper, um, asked and rejected the question, "Can machines think?" It's not a very useful question, but can you briefly entertain this useful qu- useless question? Can machines think? So talk about intelligence and your view of it.

    2. VVVladimir Vapnik31:28

       I don't know that. I know that Turing described imitation. If computer can imitate human being, let's call it intelligent. And he understands that it is not thinking, computer.

    3. LFLex Fridman31:45

       Yes.

    4. VVVladimir Vapnik31:46

       He- he completely understand what he doing, but he set up problem of le- im- imitation. So now, we understand that the problem not in imitation. I'm not sure that intelligence just inside of us. It might be also outside of us. I have several observations. So when I prove some theorem-

    5. LFLex Fridman32:12

       Mm-hmm.

    6. VVVladimir Vapnik32:12

       ... it's very difficult theorem. Uh, in couple of years, in several places, uh, people proved the same theorems, say, Sawyer Lemma after us was done then- and other guys proved the same theorem.

    7. LFLex Fridman32:28

       Yeah.

    8. VVVladimir Vapnik32:28

       In the history of science, it's happened all- all the time. For example, geometry.

    9. LFLex Fridman32:34

       Mm-hmm.

    10. VVVladimir Vapnik32:35

        It's happened simultaneously. First did, Lobachevsky and then Gauss and Bolyai and- and other guys in the... approximately in 10 times period, 10-10 years period of time.

    11. LFLex Fridman32:48

        Mm-hmm.

    12. VVVladimir Vapnik32:48

        And I saw a lot of examples like that. And many mathematicians thinks that when they develop something-

    13. LFLex Fridman32:55

        Mm-hmm.

    14. VVVladimir Vapnik32:56

        ... they developing something in general which affect everybody.

    15. LFLex Fridman33:00

        Mm-hmm.

    16. VVVladimir Vapnik33:01

        So maybe our models that intelligence is only inside of us is incorrect.

    17. LFLex Fridman33:07

        It's our interpretation, yeah.

    18. VVVladimir Vapnik33:09

        It might be there exist some connection-

    19. LFLex Fridman33:14

        Yeah.

    20. VVVladimir Vapnik33:14

        ... this world intelligence. I don't know that.

    21. LFLex Fridman33:16

        You're almost like plugging in into w- uh...

    22. VVVladimir Vapnik33:19

        Yeah, exactly.

    23. LFLex Fridman33:21

        (laughs) And contributing to this, uh...

    24. VVVladimir Vapnik33:23

        Into big network.

    25. LFLex Fridman33:23

        (laughs) Into- into a big, uh, maybe neural network.

    26. VVVladimir Vapnik33:26

        No, no, no, no.

12. 33:27 – 36:53

    Complexity, worst-case thinking, and why edges matter in theory

    1. LFLex Fridman33:27

       (laughs) On the flip side of that, maybe you can comment on, uh, big O-complexity and how you see classifying algorithms by worst case running time in relation to their input, so that way of thinking about functions. Do you think P equals NP? Do you think that's an interesting question?

    2. VVVladimir Vapnik33:49

       Yeah, it is interesting question. But let me talk about complexity and about worst case scenario. There is a mathematical setting. When I came to United State in 1990 first, people did not know this is theories. They did not know statistical learning theory. So in Russia, it was published to monographs, our monographs, but in America, they did not know. Then they learned, and somebody told me that it is worst case theory and they will create real case theory, but till now, I did not... Because it is mathematical tool. You can do only what you can do using mathematics, and which has a clear understanding-

    3. LFLex Fridman34:41

       Mm-hmm.

    4. VVVladimir Vapnik34:42

       ... and clear description. And for this reason, we introduce complexity, and we need this because using... actually, it is diversity. Like this one more, we should have mentioned you can prove some theorems. But we also create theory for case when you know probability measure, and that is the best case which can happen. It is entropy theory. So from mathematical point of view, you know the best possible case and the worst cos- possible case. You can derive different model in medium- medium, but it's not so interesting.

    5. LFLex Fridman35:30

       You think the edge ca- the edges are interesting?

    6. VVVladimir Vapnik35:33

       The edges is interesting because it is not so easy to get good bound, exact bound. It's not many cases where you have... the bound is not exact, but interesting principles which discover-

    7. LFLex Fridman35:53

       Mm-hmm.

    8. VVVladimir Vapnik35:53

       ... the mass.

    9. LFLex Fridman35:54

       Do you think it's interesting because it's challenging and reveals interesting principles that allow you to get those bounds? Or do you think it's interesting because it's actually very useful for understanding the essence of a function of a- of an algorithm? Uh, so (laughs) it's like me judging your life as a human being by the worst thing you did and the best thing you did, versus all the stuff in the middle. It seems, uh, not productive.

    10. VVVladimir Vapnik36:25

        I don't think so, because you cannot describe situation in the middle.... or it will be not general. So you can describe edge of cases and it is clear it has some model, but you cannot describe model for every new case. So you, you'll be never accurate when you're using model.

13. 36:53 – 39:57

    Uniform law of large numbers vs. ‘invariants’ learning: data efficiency

    1. LFLex Fridman36:53

       But from a statistical point of view, the way you've studied, uh, functions and, and the nature of learning and the world, don't you think that the real world has a very long tail that the edge cases are very far away from the mean? (laughs) The- the stuff in the middle or no?

    2. VVVladimir Vapnik37:19

       I don't know that.

    3. LFLex Fridman37:21

       Because-

    4. VVVladimir Vapnik37:21

       I think that... But from my point of view, if you will use formal statistic-

    5. LFLex Fridman37:33

       Mm-hmm.

    6. VVVladimir Vapnik37:34

       ... you need (clears throat) uniform law of large numbers. If you will use this invariants business, you will need just law of large numbers. You don't... And, and there's this huge difference between uniform law of large numbers and large numbers.

    7. LFLex Fridman37:56

       Is it useful to describe that a little more or should we just take it to-

    8. VVVladimir Vapnik38:01

       No. For example, when, when I'm talking about duck, I gave three predicates and that was enough. But if you will try to, to do formal distinguish, you will need a lot of observation.

    9. LFLex Fridman38:16

       I gotcha.

    10. VVVladimir Vapnik38:18

        Uh, and so that means that information about looks like a duck contain a lot of bit of information, formal bits of informations. So we don't know that how much bit of information contain things from artificial in- from intelligence.

    11. LFLex Fridman38:39

        Mm-hmm.

    12. VVVladimir Vapnik38:39

        And that is the subject of analysis. Till now, old business, I- I don't like how, uh, people consider artificial intelligence. They consider as some codes which imitate activity of human being.

    13. LFLex Fridman39:00

        Mm-hmm.

    14. VVVladimir Vapnik39:01

        It is not science. It is applications. You would like to imitate, go ahead, it is very useful and-

    15. LFLex Fridman39:07

        Mm-hmm.

    16. VVVladimir Vapnik39:07

        ... and, and yeah, good problem. But you need to, to, to, to learn something more, how people try to do... how people can, to develop say, uh, predicate swims like a duck or play like butterfly or something like that. They're not, not the teacher tells you how it came in his mind, how he choose this image.

    17. LFLex Fridman39:37

        So that process-

    18. VVVladimir Vapnik39:37

        That is problem of intelligence.

    19. LFLex Fridman39:39

        That is the problem of intelli- And you see that connected to the problem of learning?

    20. VVVladimir Vapnik39:44

        Absolutely.

    21. LFLex Fridman39:45

        Are they-

    22. VVVladimir Vapnik39:46

        Because you immediately give this predicate like, uh, specific predicates swims like a duck or quack like a duck. It was chosen somehow.

14. 39:57 – 48:47

    Open problems and the digit-recognition challenge: inventing the right invariants

    1. LFLex Fridman39:57

       So what is the line of work would you say whe- uh, if you were to formulate as a set of open problems that will take us there? Will... to, to play like a butterfly will get a system to be able to-

    2. VVVladimir Vapnik40:12

       Let's separate two stories. One mathematical story that if you have predicate you can do something, and another story how to get predicate. It is intelligence problem and people even did not start understand intelligence. Because to understand intelligence first of all, try to understand what doing teachers. How teacher teach. Why one, one teacher better than another one?

    3. LFLex Fridman40:44

       Yeah. And so you, you think we really even haven't started on the journey of-

    4. VVVladimir Vapnik40:48

       No.

    5. LFLex Fridman40:49

       ... generating the predicates?

    6. VVVladimir Vapnik40:50

       No. We don't understand. We even don't understand that this problem exist because did you hear-

    7. LFLex Fridman40:57

       You do. (laughs) Yeah, yeah.

    8. VVVladimir Vapnik40:59

       No. I, I, I just know name.

    9. LFLex Fridman41:02

       Yeah.

    10. VVVladimir Vapnik41:02

        I, I want to understand, uh, why one teacher better than another, and how effect teacher student. It is not because he repeating the problem which is in textbook.

    11. LFLex Fridman41:17

        Yes.

    12. VVVladimir Vapnik41:18

        He make some remarks. He make some philosophy of reasoning.

    13. LFLex Fridman41:23

        Yeah, that's a beautiful... So it is a formulation of a question that is the open problem, why is one teacher better than another?

    14. VVVladimir Vapnik41:33

        Right. What he does better?

    15. LFLex Fridman41:37

        Yeah. What, what, what... (laughs) Why at, at every level?

    16. VVVladimir Vapnik41:42

        What people-

    17. LFLex Fridman41:43

        Uh, ho- how do they get better? What does it mean to be better? Uh, I've... The, the whole...

    18. VVVladimir Vapnik41:49

        Yeah. Uh, yeah, from, from whatever model I have-

    19. LFLex Fridman41:53

        Yeah.

    20. VVVladimir Vapnik41:54

        ... one teacher can give a very good predicate. One teacher can say, uh, swims like a duck, and another can say jump like a duck. And jump like a duck carries zero information.

    21. LFLex Fridman42:07

        Yeah. (laughs) So what is the most exciting problem in statistical learning you've ever worked on or are working on now?

    22. VVVladimir Vapnik42:16

        Um, I just finished this invariant story.

    23. LFLex Fridman42:21

        Mm-hmm.

    24. VVVladimir Vapnik42:23

        And I'm happy that... I believe that ee- it is ultimate learning story. At least I can show that there are no another mechanism, only two mechanisms.... but they separate statistical part from intelligent part, and I know nothing about intelligent part. And if we will know this intelligent part, so it will help us a lot in teaching, in, in, in, in learning.

    25. LFLex Fridman42:59

        In learning.

    26. VVVladimir Vapnik43:00

        Yeah. So-

    27. LFLex Fridman43:00

        Do you think we'll know it when we see it?

    28. VVVladimir Vapnik43:02

        So for example, in, in my talk, the last slide was a challenge.

    29. LFLex Fridman43:06

        Mm-hmm.

    30. VVVladimir Vapnik43:07

        So you have, say, at least digit recognition problem, and deep learning claims that they did it, uh, very well. Say, 99.5% of correct answers. But they used 60,000 observations.
15. 48:47 – 54:02

    Ground truth, structure in Bach, and the researcher’s ethic of self-skepticism

    1. LFLex Fridman48:47

       Beautifully put. Maybe just me, but in all the math you show, in your work, uh, which is some of the most profound mathematical work in, in, in the field of learning AI and just math in general, I hear a lot of poetry and philosophy. Y- you really kinda, um, talk about philosophy of science. You, there's a po- there's a poetry and music to a lot of the work you're doing and the way you're thinking about it. So do you... Where does that come from? Do you es- do you escape to poetry? Do you escape to music? Or not? (laughs)

    2. VVVladimir Vapnik49:21

       I think that, I think that there exist ground truths.

    3. LFLex Fridman49:24

       There exists ground truth? (laughs)

    4. VVVladimir Vapnik49:25

       Yeah. And that ex- can be seen everywhere.

    5. LFLex Fridman49:29

       Yeah.

    6. VVVladimir Vapnik49:30

       The smart guy philosophers, sometime I surprise how they deep see, sometime I see that some of them are completely out of subject.

    7. LFLex Fridman49:43

       Mm-hmm.

    8. VVVladimir Vapnik49:45

       But the ground truth, I see music.

    9. LFLex Fridman49:50

       Music is a ground truth?

    10. VVVladimir Vapnik49:52

        Yeah. And in poetry, many poet, they, they believe the... They take dictation.

    11. LFLex Fridman50:00

        (laughs) So what, uh, what piece of music as a piece of em- empirical evidence gave you a sense that they are, um, they're touching something in the ground truth?

    12. VVVladimir Vapnik50:14

        It is structure.

    13. LFLex Fridman50:16

        The structure (laughs) within-

    14. VVVladimir Vapnik50:17

        Yeah.

    15. LFLex Fridman50:17

        ... the math of music.

    16. VVVladimir Vapnik50:18

        Because when you're listening to Bach-

    17. LFLex Fridman50:20

        Yeah, Bach.

    18. VVVladimir Vapnik50:20

        ... you see this structure.

    19. LFLex Fridman50:21

        Yeah.

    20. VVVladimir Vapnik50:22

        Very clear, very classic, very simple, and the same in mass when you have a- axioms in geometry, you have the same feeling.

    21. LFLex Fridman50:32

        Yeah. (laughs)

    22. VVVladimir Vapnik50:33

        And in poetry, sometimes you see the same.

    23. LFLex Fridman50:36

        Yeah. Um, and if you look back at your childhood, you grew up in Russia, you maybe were born as a researcher in Russia, you've developed as a researcher in Russia, you've came to United States and a few places. If you look back, what were, what was some of your happiest moments as a researcher? Uh, some of the most profound moments? Not in terms of their impact on society, but in terms of their impact on how damn good you feel that day, and you remember that moment.

    24. VVVladimir Vapnik51:15

        You know, every time, when you found something, it is great-

    25. LFLex Fridman51:21

        Yeah.

    26. VVVladimir Vapnik51:22

        ... ............................ in the life. Every simple things.

    27. LFLex Fridman51:26

        Just even number-

    28. VVVladimir Vapnik51:26

        But my general feeling is that I mostly, most of my time was wrong. You should go again and again and again, and try to be honest in front of yourself. Not to make interpretation, but try to u- uh, understand that it related to ground truths.

    29. LFLex Fridman51:46

        Mm-hmm.

    30. VVVladimir Vapnik51:46

        It is not my blah, blah, blah, interpretation and something like that.

Episode duration: 54:02