Lex Fridman PodcastVladimir Vapnik: Predicates, Invariants, and the Essence of Intelligence | Lex Fridman Podcast #71
EVERY SPOKEN WORD
135 min read · 26,659 words- 0:00 – 2:55
Introduction
- LFLex Fridman
The following is a conversation with Vladimir Vapnik, part two, the second time we spoke on the podcast. He's the co-inventor of support vector machines, support vector clustering, VC theory, and many foundational ideas in statistical learning. He was born in the Soviet Union, worked at the Institute of Control Sciences in Moscow, then in the US, worked at AT&T, NEC labs, Facebook AI Research, and now is a professor at Columbia University. His work has been cited over 200,000 times. The first time we spoke on the podcast was just over a year ago on one of the early episodes. This time, we spoke after a lecture he gave titled Complete Statistical Theory of Learning as part of the MIT series of lectures on Deep Learning and AI that I organized. I'll release the video of the lecture in the next few days. This podcast and lecture are independent from each other, so you don't need one to understand the other. The lecture is quite technical and math heavy. So if you do watch both, I recommend listening to this podcast first since the podcast is probably a bit more accessible. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, give it five stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter @lexfridman, spelled F-R-I-D-M-A-N. As usual, I'll do one or two minutes of ads now, and never any ads in the middle that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. When you get it, use code LEXPODCAST. Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1. Broker services are provided by Cash App Investing, a subsidiary of Square and member SIPC. Since Cash App allows you to send and receive money digitally peer-to-peer, and security in all digital transaction is very important, let me mention the PCI data security standard, PCI DSS level 1 that Cash App is compliant with. I'm a big fan of standards for safety and security, and PCI DSS is a good example of that, where a bunch of competitors got together and agreed that there needs to be a global standard around the security of transactions. Now, we just need to do the same for autonomous vehicles and AI systems in general. So again, if you get Cash App from the App Store or Google Play and use the code LEXPODCAST, you get $10 and Cash App will also donate $10 to FIRST, one of my favorite organizations that's helping to advance robotics and STEM education for young people around the world. And now, here's my conversation with Vladimir Vapnik.
- 2:55 – 9:09
Alan Turing: science and engineering of intelligence
- LFLex Fridman
You and I talked about Alan Turing yesterday a little bit.
- VVVladimir Vapnik
Yes.
- LFLex Fridman
And that he, as the father of artificial intelligence, may have instilled in our field an ethic of engineering and not science, seeking more to build intelligence rather than to understand it. What do you think is the difference between these two paths of engineering intelligence and the science of intelligence?
- VVVladimir Vapnik
It's completely different story. Engineering is imitation of human activity. You have to make a device which behave as human behave, have all the functions of human. It does not matter how you do it. But to understand what is intelligence about is quite different problem. So I think, I believe that it's somehow related to predicate we talked yesterday about, uh, because look at Vladimir Propp's idea. He just found 31 here predicates, he called it units, which can explain human behavior, at least in Russian tales here. Look at the Russian tales and derive from that, and then people realize that it's more wider than Russian tales. It is in TV, in movie serials, and so on and so on.
- LFLex Fridman
So you're talking about Vladimir Propp...
- VVVladimir Vapnik
Right.
- LFLex Fridman
... who in 1928 published a book, Morphology of the Folktale...
- VVVladimir Vapnik
Exactly.
- LFLex Fridman
... describing 31 predicates that have this kind of sequential structure that a lot of the stories, narratives follow in Russian folklore and in other content. We'll talk about it. I'd like to talk about predicates in a focused way, but let me, if you allow me to stay zoomed out on our friend Alan Turing, and, you know, he inspired a generation with the, The Imitation Game.
- VVVladimir Vapnik
Yes.
- LFLex Fridman
Do you think... If we can linger on that a little bit longer, do you think we can learn... Do you think learning to imitate intelligence can get us closer to the science, to understanding intelligence? So, why do you think imitation is so far from understanding?
- VVVladimir Vapnik
I think that it is different, between you have different goals. So, your goal is to, to, to create something, something useful.
- LFLex Fridman
Yeah.
- VVVladimir Vapnik
And that is great, and you can see how much things was done, and I believe that it will be done even more, self-driving cars and also this business. It is great, and it was inspired by Turing's vision.But understanding is very difficult. It's more or less a philosophical category. What means, understand the world? I believe in scheme which start from Plato, that there exists world of ideas. I believe that intelligence, it is world of ideas. But it is world of pure ideas. And when you combine them with reality things, it create, as in my case, invariants, which is very specific. And that's, I believe, the combination of ideas in way to constructing invariant is intelligence. But first of all, uh, uh, predicate, if you know predicate and hopefully then not, not too much predicate exist. For example, 31 predicate for human behavior, it is not a lot.
- LFLex Fridman
Vladimir Propp used 31, (sighs) you can even call them predicates, 31 predicates to describe stories, narratives.
- VVVladimir Vapnik
Right.
- LFLex Fridman
So you think human behavior... how much of human behavior, how much of our world, our universe, all the things that matter in our existence can be summarized in predicates of the kind that Propp was working with?
- VVVladimir Vapnik
I think that we have a lot of form of behavior, but I think the predicate is much less because even in this examples which I gave you yesterday, you saw that predicate can be... can const- one predicate can construct many different invariants depending on, on, on your data. They're applying to different data and they give different invariants. So, but pure ideas, maybe not so much.
- LFLex Fridman
Not so many. Let's-
- VVVladimir Vapnik
I don't know about that, but my guess, I hope, that's why challenge about digit recognition, how much you need.
- LFLex Fridman
I think we'll talk about computer vision and 2D images a little bit in your challenge.
- VVVladimir Vapnik
That's exact about intelligence.
- LFLex Fridman
That's exactly. That's exactly about... no, that hopes to be exactly about the spirit of intelligence in the simplest possible way.
- VVVladimir Vapnik
Yeah, absolutely. You should start the simplest way, otherwise you will not be able to do it.
- LFLex Fridman
Well, there's an open question whether starting at the MNIST digit recognition is a step towards intelligence or it's an entirely different thing.
- VVVladimir Vapnik
I think that to beat records using say, 100, 200 times less examples, you need intelligence.
- LFLex Fridman
You need intelligence. So let's, because you use this term, and it would be nice... I'd, I'd like to ask simple, maybe even dumb questions.
- 9:09 – 14:22
What is a predicate?
- LFLex Fridman
Let's start with a predicate. In terms of terms and how you think about it, what is a predicate?
- VVVladimir Vapnik
I don't know.
- LFLex Fridman
(laughs)
- VVVladimir Vapnik
I, I have a feeling formally they exist, but I believe that predicate for 2D images, uh, one of them is symmetry.
- LFLex Fridman
Hold on a second. Sorry. Sorry to interrupt and pull you back. At the simplest level, we're not even s- we're not being profound currently, a predicate is a statement of something that is true.
- VVVladimir Vapnik
Yes.
- LFLex Fridman
Do you think of predicates as somehow probabilistic in nature, or is this binary, this is truly constraints of logical statements about the world?
- VVVladimir Vapnik
In my definition, the simplest predicate is function. Function, and you can use this function to make inner product, that is predicate.
- LFLex Fridman
What's the input and what's the output of the function?
- VVVladimir Vapnik
Input is x, something which is input in reality. Say, if you consider digit recognition, it pixel space-
- LFLex Fridman
Yes.
- VVVladimir Vapnik
... input, but it is function which in pixel space, but it can be any function from, uh, pixel space, and you choose the... and, and, and I believe that there are several functions which is important to f- understanding of images. One of them was symmetry. It's not so simple construction as I described with linearity, with all the stuff, but another I believe, I don't know how many, is how well structurized is picture.
- LFLex Fridman
Structurized?
- VVVladimir Vapnik
Yeah.
- LFLex Fridman
What do you mean by structurized?
- VVVladimir Vapnik
It is formal definition. Say something happ- heavy on the left corner, not so heavy in the middle, and so on. You describe in general concept of, of what, what you are seeing.
- LFLex Fridman
Concepts, some kind of universal concepts.
- VVVladimir Vapnik
Yeah. But I don't know how to formalize this.
- LFLex Fridman
Do you... so this is the thing. There's a million ways we can talk about this. I'll keep bringing it up, but, uh, we humans have such concepts, uh, when we look at digits, but it's hard to put them, just like you're saying now, it's hard to put them into words.
- VVVladimir Vapnik
You know, that is example. When critics in music trying to describe music, they use predicate.
- LFLex Fridman
(sighs) Right.
- VVVladimir Vapnik
And not too many predicate, but in different combination, but they have some special words for describing music and-... the same should be for images. But my bizarre critics who understand essence of what is images about.
- LFLex Fridman
Do you think there exists critics who can summarize the essence of images, human beings? The-
- VVVladimir Vapnik
I hope that, yes. But-
- LFLex Fridman
Like, explicitly state them on paper?
- VVVladimir Vapnik
Hmm.
- LFLex Fridman
I, this, the fundamental question I'm asking is, do you, (laughs) do you think there exists a f- a small set of predicates that will summarize images? It feels to our mind like it does, that the concept of what makes a two and a three and a four-
- VVVladimir Vapnik
No, no, no. It's not on, on, on this level. Uh, what, it, it, it, it should not describe two, three, four.
- LFLex Fridman
(laughs) .
- VVVladimir Vapnik
It, it, it describes some construction which allow you to create invariants.
- 14:22 – 21:06
Plato's world of ideas and world of things
- VVVladimir Vapnik
that.
- LFLex Fridman
Let's talk about Plato a little bit. So, you draw line from Plato to Hegel to Wigner to today.
- VVVladimir Vapnik
Yes.
- LFLex Fridman
So, Plato has forms, the Theory of Forms.
- VVVladimir Vapnik
Yeah.
- LFLex Fridman
There's a world of ideas-
- VVVladimir Vapnik
Yes.
- LFLex Fridman
... and a world of things, as you talk about.
- VVVladimir Vapnik
Right, right.
- LFLex Fridman
And there's a connection. And presumably, the world of ideas is very small-
- VVVladimir Vapnik
Yes.
- LFLex Fridman
... and the world of things is arbitrarily big.
- VVVladimir Vapnik
Sure.
- LFLex Fridman
But they're all, what Plato calls them, like, the, it's a shadow.
- VVVladimir Vapnik
Yeah.
- LFLex Fridman
The real world is a shadow from the world of form.
- VVVladimir Vapnik
Yeah, you have projection-
- LFLex Fridman
Projection.
- VVVladimir Vapnik
... of world of idea.
- LFLex Fridman
Yeah. Very poetic
- NANarrator
(laughs) .
- VVVladimir Vapnik
And in, in reality, you can realize this projection using, using inva- these invariants, because it is projection for, on specific examples, which creates specific features of specific objects. So, and, uh-
- LFLex Fridman
So, the essence of intelligence is while only being able to observe the world of things, try to come up with a world of ideas?
- VVVladimir Vapnik
Exactly. Like in this music story.
- LFLex Fridman
(laughs) .
- VVVladimir Vapnik
Intelligent musical critics knows the soul of this world and have a feeling about what and-
- LFLex Fridman
I feel like that's a contradiction, intelligent music critics, but I think y- I think music is to be enjoyed in all its forms. The notion of critic, like a food critic-
- VVVladimir Vapnik
No, I don't want to touch emotion.
- LFLex Fridman
That's an interesting question. Does emotion... There's a certain elements of the human psychology, of the human experience, which seem to almost contradict intelligence and reason, like emotion, like fear, like, like love. All of those things, are those not connected in any way to the space of, uh, ideas?
- VVVladimir Vapnik
That, I don't know. I, I, I just want to be concentrate on very simple story-
- 21:06 – 28:37
Strong and weak convergence
- VVVladimir Vapnik
- LFLex Fridman
What is convergence? What is weak convergence? What is strong convergence?
- VVVladimir Vapnik
So-
- LFLex Fridman
I'm sorry, I'm gonna do this to you. What are we converging from and to?
- VVVladimir Vapnik
You're converging... You would like to have a function, the function which, uh, say indicate a function which indicate your digit 5, for example.
- LFLex Fridman
A classification task is-
- VVVladimir Vapnik
Let's talk only about classification task.
- LFLex Fridman
So classification means-
- VVVladimir Vapnik
Yeah.
- LFLex Fridman
... you'll say whether this is a five or not, or say which of the 10 digits it is.
- VVVladimir Vapnik
Right, right. I would like to, to, to have these functions. Then I have some examples. I can consider property of those examples, say symmetry, and I can measure level of symmetry for every digit. And then I can take average, and I... uh, from, from my training data, and I will consider only functions of conditional probability which I am looking for my decision rule, which applying to, to digits, will give me the same average as I observe on training data. So actually, this is different level of description of what you want. You want, uh, not just your, your show, not one digit. You show this, this predicate show general property of all digits which you have in mind. If you have in mind digit three, it gives you property of digit three, and you select as admissible set of function only function which keeps this property. You will not consider other functions. So you're immediately looking for smaller subset of function from-
- LFLex Fridman
That's what you mean by admissible functions, you're-
- VVVladimir Vapnik
Admissible function, exactly.
- LFLex Fridman
Which is still a pretty large, for the number three, it's a large-
- VVVladimir Vapnik
Uh, it is pretty large, but if you have one predicate.
- LFLex Fridman
Right.
- VVVladimir Vapnik
But according to... Uh, we... there is a strong and weak convergence.
- LFLex Fridman
Okay.
- VVVladimir Vapnik
Strong convergence is convergence in function. You're looking at the function, on one function, and you're looking at another function, and, uh, uh, square difference from them should be small. If you take difference in, in your points, make a square, make an integral, and it should be small. That is convergence in function. Suppose you have some function, any function. So I would say, uh, I say that some function converge to this function.... if integral from square difference between them is small.
- LFLex Fridman
That's the definition of strong convergence?
- VVVladimir Vapnik
That's definition of strong-
- LFLex Fridman
Two functions, the integral of the difference-
- VVVladimir Vapnik
Yeah.
- LFLex Fridman
... is small.
- VVVladimir Vapnik
It is convergence in functions.
- LFLex Fridman
Yeah.
- VVVladimir Vapnik
But you have different convergence in functionals. You take any function, you take some function fi, and take inner product, this function, this F function, F0 function which you want, uh, to find, and that gives you some value. So you say that set of functions converge in, in a product to this function if this value of inner product converge to value F0. That is for one fi. But weak convergence requires that it converge for any function of Hilbert space. If it converge for any function of Hilbert space, then you will say that this is weak convergence. You can think that when you take integral, that is property, integral property of function. For example, if you will take sign or cosine, it is coefficient of, uh, say, free expansion. So i- if it converge for all coefficients of free expansion, so under some condition, it converge to, to, to function you're looking for. But weak convergence means any property. Convergence not pointwise but integral property of function. So weak convergence means integral property of functions. When I talking about predicate, I would like to, um, formulate which integral properties I would like to have for convergence. So, and if I will take one predi- predicate its function which I measure property, uh, if I will use one predicate and say I will consider only function which give me the same value as does this predicate, I selecting set of functions from functions which is admissible in the sense that function which I looking for is this set of functions, because I checking in, in, in training data. It is, it gives the same, uh-
- LFLex Fridman
Yeah, so it always has to be connected to the training data in terms of...
- VVVladimir Vapnik
Yeah. But, but property you can know independent on training data. And this guy, Prop-
- LFLex Fridman
Yeah.
- 28:37 – 50:36
Deep learning and the essence of intelligence
- LFLex Fridman
Do you think it's possible to automatically discover the predicates? This... so you basically said that the essence of intelligence is the discovery of good predicates.
- VVVladimir Vapnik
Yeah.
- LFLex Fridman
Now, the natural question is, you know, that's what Einstein was good at doing in physics. Can we make machines do these kinds of discovery of good predicates, or is this ultimately a human endeavor?
- VVVladimir Vapnik
That I don't know. I don't think that machine can do because, uh, according to theory about weak convergence, uh, any function from Hilbert space can be predicate. So you, you have infinite number of predicate in upper, in, in, before, you don't know which predicate is good than which. But whatever Prop show and why people call it breakthrough, that there is not too many predicate which cover most of situation happened in the world.
- LFLex Fridman
(inhales deeply) So there's a sea of predicates, and most of the... o- only a small amount are useful for the kinds of things that happen in the world?
- VVVladimir Vapnik
I think that I would say only small part of predicate very useful, useful all of... uh, all of them.
- LFLex Fridman
Right. Only very few are what we should, sh- let's call them good predicates.
- VVVladimir Vapnik
Very good predicate.
- LFLex Fridman
Right, very good predicates.
- VVVladimir Vapnik
Yes.
- LFLex Fridman
So, can we linger on it? What's your intuition? Why is it-... hard for a machine to discover good predicates.
- VVVladimir Vapnik
I, even in my talk, described how to do predicate, how to find new predicate. I, I'm not sure that it is very good-
- LFLex Fridman
What did you propose in your talk?
- VVVladimir Vapnik
No, in my talk, I, I, I gave example for diabetes.
- LFLex Fridman
Diabetes, yeah.
- VVVladimir Vapnik
When, when, when we achieve some percent, so then they're looking from area where some sort of predicate, which I formulate, uh, does not, uh, keeps invariant. So, if it doesn't keep, I, I retrain my data, I, uh, select only function which keeps this invariant. And the way I did it, I improved my performance. I came looking for this predicate. I know technically how to do that, and you can, uh, of course, uh, uh, do it using machine, but I am not sure that we will construct the smartest predicate.
- LFLex Fridman
But this is the... allow me to linger on it because that's the essence, that's the challenge. That is artificial int- that's, that's the human level intelligence that we seek, is the discovery of these good predicates. You've talked about deep learning as a way to, um... the predicates they use and the functions are mediocre, but you can find better ones.
- VVVladimir Vapnik
Let's talk about deep learning.
- LFLex Fridman
Sure, let's do it.
- VVVladimir Vapnik
I, I know only Yann LeCun convolutional network.
- LFLex Fridman
Yeah.
- VVVladimir Vapnik
And what else? I don't know, and it's a very simple convolution. You have two pixels-
- LFLex Fridman
There's not much else to know.
- VVVladimir Vapnik
... left and right.
- LFLex Fridman
Yes.
- VVVladimir Vapnik
I can do it like that, one, with one predicate. It is-
- LFLex Fridman
Convolution is a single predicate.
- VVVladimir Vapnik
It's single, m- it's sin- it- uh, it's single predicate.
- LFLex Fridman
Yes, but that-
- VVVladimir Vapnik
It is, it, y- you know exactly, you take the derivative for translation and, and predicate is, should, should be kept.
- 50:36 – 54:31
Symbolic AI and logic-based systems
- LFLex Fridman
brute force way. What about the ideas of sym- what, uh, big umbrella term of symbolic AI? These, what in '80s with expert systems, sort of logic reasoning based systems.
- VVVladimir Vapnik
(sighs)
- LFLex Fridman
Is there hope there to find some, uh, through sort of deductive r- reasoning, to f- to find good predicates?
- VVVladimir Vapnik
I don't think so. I think that just logic is not enough.
- LFLex Fridman
It's kind of a compelling notion though, you know? That when smart people sit in a room and reason through things, it seems compelling, and making our machines do the same is also compelling.
- VVVladimir Vapnik
So, everything is very simple. When you have infinite number of predicate, you can choose the, the function you want. You have invariants and you can choose the function you want. But y- you have to have a, not too many invariants to solve the problem. So, and how from infinite number of function to select finite number, and hopefully small fini- fi- num- number of functions which is good enough to extract small set of admissible functions. So they will be admissible, it's for sure, because every function just decrease set of function and leaving it admissible. But it will be small.
- LFLex Fridman
But why do you think logic based systems don't, can't help? Uh, uh, intuition, not-
- VVVladimir Vapnik
Because you, you should know reality. You should know life. This guy like Proop-
- LFLex Fridman
Mm-hmm.
- VVVladimir Vapnik
... he knows something, and he tried to, to put in invariant his understanding.
- LFLex Fridman
So, but that's the human... yeah, yeah. But see, you're, you're putting too much value into V- Vladimir Proop's knowing something.
- VVVladimir Vapnik
No, it, it is, it is-
- LFLex Fridman
Am I, am I doing this understanding-
- VVVladimir Vapnik
... the story is that, what means you know life? What it means-
- LFLex Fridman
Common s- you know, common sense.
- VVVladimir Vapnik
No, no.
- LFLex Fridman
No.
- VVVladimir Vapnik
You, you know something. Common sense, it is some rules.
- LFLex Fridman
You think so? Common sense is simply rules? Common sense is every, it's mortality, it's know, it's, it's fear of death, it's love, it's spirituality, it's, uh, happiness and sadness. All of it is tied up into understanding gravity, which is what we think of as common sense. (laughs)
- VVVladimir Vapnik
I don't ready to discuss so wide.
- LFLex Fridman
(laughs) .
- VVVladimir Vapnik
I want to discuss, understand digit recog- u- understand
- LFLex Fridman
(laughs) .
- VVVladimir Vapnik
... digit recognition, understand-
- LFLex Fridman
Any time I bring up love and death, (laughs) you, you bring it back to digit recognition. I'm like, he- (laughs)
- VVVladimir Vapnik
Yeah. No, you know, it is doable because there is a challenge-
- LFLex Fridman
Yeah.
- VVVladimir Vapnik
... which I see how to solve it. If I will have a student concentrate on this work, I will suggest something to solve.
- LFLex Fridman
You mean handwritten recognition?
- VVVladimir Vapnik
Handwritten recognition.
- 54:31 – 1:00:23
How hard is 2D image understanding?
- LFLex Fridman
let me ask, how hard do you think is 2D image understanding? So if, if we, we can kind of intuit handwritten recognition-How big of a step, leap, journey is it from that? If I gave you good... if I solved your challenge for handwritten recognition, how long would my journey then be from that to understanding more general, natural images?
- VVVladimir Vapnik
Immediately you will understand this-
- LFLex Fridman
(laughs) .
- VVVladimir Vapnik
... as soon as you will make a record.
- LFLex Fridman
I think so.
- VVVladimir Vapnik
Because it is not for free. As soon as you will create several invariants which will help you to get the same performance that the best neural net did using 100 and maybe even more than 100 times less examples, you have to have something smart to do that.
- LFLex Fridman
And you're saying-
- VVVladimir Vapnik
That that is an invariant. It is predicate, because you should put some idea how to do that.
- LFLex Fridman
But, okay. Let me just pause. Maybe it's a trivial point, maybe not, but handwritten recognition feels like a 2D, two-dimensional problem. And it seems... Like, how much complicated is the fact that most images are a projection of a three-dimensional world onto 2D plane? It feels like for a three-dimensional world, we still, we need to start understanding common sense in order to understand an image. It's no longer visual shape and symmetry. It's having to start to understand concepts of, uh, understand life.
- VVVladimir Vapnik
Yeah. You're, you're, you're, you're, you're talking as though there are different invariant, different-
- LFLex Fridman
Different.
- VVVladimir Vapnik
... predicates. Yeah.
- LFLex Fridman
And potentially much l- larger number.
- VVVladimir Vapnik
You know, might be, but let's start from simple.
- LFLex Fridman
Well, yeah, but you said that it would be immediate.
- VVVladimir Vapnik
No, you know, I, I, I cannot think-
- LFLex Fridman
Yes.
- VVVladimir Vapnik
... about things which I don't understand.
- LFLex Fridman
Yeah. (laughs)
- VVVladimir Vapnik
This I understand, but I'm sure that I don't understand everything there.
- LFLex Fridman
Yeah. That's the difference-
- VVVladimir Vapnik
As Einstein said, uh, "Do as simple as possible, but not simpler." And that is exact case.
- LFLex Fridman
With handwritten recognition.
- VVVladimir Vapnik
With handwritten.
- LFLex Fridman
Yeah. But never... That's the difference between you and I. I, uh, (laughs) , I welcome and enjoy thinking about things I completely don't understand (laughs) . Because to me, it's a natural extension without having solved handwritten recognition, to wonder how, um, how difficult is the, the, the next step of understanding 2D, 3D images. Because ultimately, while the science of intelligence is fascinating, it's also fascinating to see how that maps to the engineering of intelligence. And recognizing handwritten digits is not, doesn't help you. It might, it may not help you with the problem of general intelligence. We don't know. It'll, it'll help you a little bit, but we don't know how much.
- VVVladimir Vapnik
It, it, it, it's unclear.
- LFLex Fridman
It's unclear.
- VVVladimir Vapnik
Yeah, yeah.
- LFLex Fridman
It might vary much.
- VVVladimir Vapnik
But I would like to make a remark.
- 1:00:23 – 1:06:39
Data
- LFLex Fridman
Okay, let me, let me put, put this on you, because I'm an emotional creature. I'm not a mathematical creature like you. I find compelling the idea... Forget the s- the space, the sea of functions. There's also a sea of data in the world, and I find compelling that there might be, like you said, teacher. Small examples of data that are most useful for discovering...... good, whether it's predicates or good functions, that the selection of data may be a, a powerful journey, a useful mechan- you know, coming up with a mechanism for selecting good data might be useful too. Do you find this idea of finding the right data set interesting at all? Or do you kind of take the data set as a given?
- VVVladimir Vapnik
I think that it is... Y- you know, my scheme is very simple. You have huge set of functions. If you will apply ... and you have not too many data.
- LFLex Fridman
Right.
- VVVladimir Vapnik
If you pick up function which describes this data, you will do not very well.
- LFLex Fridman
Like randomly pick up-
- VVVladimir Vapnik
Yeah, yeah, you will overfit here. Here, it will be overfitting. So you should s- decrease set of function from which you're picking up one. So you should go some how to admissible set of function. And this, what about weak conversions? So what... From another point of view, to, to make admissible set of function, you need just a just function, which you will take in, in a product which you will measure property of your function.
- LFLex Fridman
Hmm.
- VVVladimir Vapnik
And that is how it works. So-
- LFLex Fridman
No, I get it. I get it. I understand that. But do you... The reality is-
- VVVladimir Vapnik
But let, let's discu- let's, let's think about, uh, examples. You have huge set of function and you have several examples. If you just trying to keep, but take function which satisfies these examples, you still will overfit. You need decrease, you need admissible set of functions.
- LFLex Fridman
No, absolutely. But what... Say you have more data than functions. So, sort of consider the... I mean, maybe not more data than functions because that's-
- VVVladimir Vapnik
It's impossible.
- LFLex Fridman
... impossible (laughs) . But what, what... I was trying to be poetic for a second. I mean, you have a huge amount of data, a huge amount of examples.
- VVVladimir Vapnik
But amount of function can be even bigger.
- LFLex Fridman
Bigger, I understand.
- VVVladimir Vapnik
(laughs) Everything is.
- LFLex Fridman
There's always, there's always a bigger boat.
- VVVladimir Vapnik
All, all, all human space.
- LFLex Fridman
(laughs) I gotcha. But, okay. But you don't i- i- you don't find the world of data to be an interesting optimization space? Like, the, the optimization should be in the space of functions?
- VVVladimir Vapnik
In creating admissible set of functions.
- LFLex Fridman
Admissible set of functions.
- VVVladimir Vapnik
No. You know, even from the classical business theory, from structured risk minimization, you should or- you should organize function in the way that they will be useful for you.
- LFLex Fridman
Right.
- VVVladimir Vapnik
And that is admissible set.
- LFLex Fridman
Yeah, but th- the way you're thinking about useful is, uh, you're given a small set of examples.
- VVVladimir Vapnik
Use a small, small set of functions which contain function by looking for.
- LFLex Fridman
Yeah, but as... Looking for, based on the empirical set of small examples?
- VVVladimir Vapnik
Yeah. But that is another story. I don't touch it because I, I beli- I believe that these small examples is not too small. Say, 60 per class, that law of large numbers works. I don't need uniform law. The story is that in statistics there are two law, law of large numbers and uniform law of large numbers. So I want to be in situation where I use law of large numbers now, but not uniform law of large numbers.
- LFLex Fridman
Right. So 60 is law of large... It's large enough.
- VVVladimir Vapnik
I hope, I hope. No-
- 1:06:39 – 1:14:54
Language
- LFLex Fridman
Okay, so we talked about images a little bit, but can we talk about Noam Chomsky for a second? (laughs)
- VVVladimir Vapnik
Ugh. No, I, I, I believe I-... I don't know him very well.
- LFLex Fridman
Yeah, per- personally? Well...
- VVVladimir Vapnik
Not personally I don't know-
- LFLex Fridman
His ideas.
- VVVladimir Vapnik
... his ideas.
- LFLex Fridman
Well, let me just say, do you think language, human language is essential to expressing ideas as Noam Chomsky, uh, believes? So like, language is at the core of our formation of predicates. The human language.
- VVVladimir Vapnik
For me, language and all the story of language is very complicated. I don't understand this and I am not... I thought about.
- LFLex Fridman
Nobody does.
- VVVladimir Vapnik
I, I am not ready to work on that, because it's so huge. It is not for me, and I believe not for our century.
- LFLex Fridman
The 21st century?
- VVVladimir Vapnik
Not for 21st century.
- LFLex Fridman
So-
- VVVladimir Vapnik
We should learn something, a lot of stuff from simple task like digit recognition.
- LFLex Fridman
So you think... Okay, look, le- you think digit recognition, 2D image, what, how would you more abstractly define, uh, digit recognition? It's 2D image symbol recognition, essentially? I mean, uh, I'd like... (laughs) I'm trying to get a sense, sort of thinking about it now, having worked with MNIST forever, how c- how small of a subset is this of the general vision recognition problem and the general intelligence problem? Is it... 'Cause, yeah. Is it a giant subset? Is it not? And how far away is language?
- VVVladimir Vapnik
You know, uh, let me refer to Einstein, "Take the simplest problem, as simple as possible, but not simpler." And this is challenge, is simple problem. But it's simple by idea but not simple to, to get it. When you will m- do this, you will find some predicate which helps you to do.
- LFLex Fridman
Well, yeah, I mean, with Ei- uh, with Einstein you can, you look at, you look at general relativity, but that doesn't help you with quantum mechanics. (laughs)
- VVVladimir Vapnik
Oh, that's another story. You're, you don't have any universal instrument.
- LFLex Fridman
Yeah, so I'm, I'm trying to wonder if, uh, which space we're in, whether the, whether handwritten recognition is like general relativity and then language is like quantum mechanics, that you're still gonna have to do a lot of mess to, to, to universalize it. But, uh, I'm trying to see, one, so, what's your intuition why handwritten recognition is easier than language? Just, I, I think a lot of people would agree with that, but if you could elucidate sort of the, the intuition of why.
- VVVladimir Vapnik
I don't s- know. No, I don't think in this direction. I just thinking in directions that this is problem which if you will solve it well, we will create some abstract understanding of images. Maybe not all images. I would like to talk to guys who doing in real images in, uh, Columbia University to-
- LFLex Fridman
What kind of images? Unreal you said?
- VVVladimir Vapnik
Real images.
- LFLex Fridman
Real images.
- VVVladimir Vapnik
Yeah. What their idea is real predicate, what can be predicate. I still s- symmetry will play a role in real life images, in any real life images, 2D images. Let's talk about 2D images. Because (clears throat) um, that's what we know and neural network was created for 2D, the images.
- LFLex Fridman
So the people I know in vision science, for example, the people who study human vision-
- VVVladimir Vapnik
Yeah.
- LFLex Fridman
... that, they usually go to the world of symbols and, like, handwritten recognition, but not really as other kinds of symbols to study our visual perception system. As far as I know, not much predicate type of thinking is understood about our vision system.
- VVVladimir Vapnik
They do not think in this direction.
- LFLex Fridman
They don't, yeah.
- VVVladimir Vapnik
So-
- 1:14:54 – 1:19:28
Beautiful idea in statistical theory of learning
- LFLex Fridman
So a lot of the things we've been talking about falls... w- we've been talking about philosophy a little bit, but also about mathematics and statistics.
- VVVladimir Vapnik
Mm-hmm.
- LFLex Fridman
A lot of it falls into this idea, a universal idea of statistical theory of learning. What is the most beautiful and sort of powerful or essential idea you've come across? Even just for yourself personally in, in the world of statistics or statistic theory of learning?
- VVVladimir Vapnik
Probably u- uniform convergence, which we did with Alexey Chervonenkis.
- LFLex Fridman
Can you describe universal convergence?
- VVVladimir Vapnik
You have law of large, law of large numbers. So for any function, expectation of function, average of function converge to expectation. But if you have set of functions, for any function it is true, but it should converge simultaneously for all set of functions. And for, for learning, you need uniform convergence. Just convergence is not enough.
- LFLex Fridman
Hm.
- VVVladimir Vapnik
Because when you pick up one which gives minimum, you can pick up one function which does not converge in... and it will give you the best answer for, for this function. So you need uniform convergence to guarantee learning. So learning does not really enter your law of large numbers. It really on universal. But idea of weak convergence exist in statistics for a long time, but it is interesting that as I, I, I think about myself, how stupid I was 50 years. I did not see weak convergence. I work only on strong convergence. But now I think that most powerful is weak convergence, because it makes admissible set of functions, and even in old pro- in proverbs when people try to understand recognition about dog law, looks like a dog and so on-
- LFLex Fridman
Mm-hmm.
- VVVladimir Vapnik
... they use weak convergence. People in language, they understand this. But when we're trying to create artificial intelligence, we want event in different way. We just consider strong convergence arguments.
- LFLex Fridman
So reducing the set of admissible functions, you think there should be effort put into understanding the properties of weak convergence?
- VVVladimir Vapnik
You know, (clears throat) in classical mathematics, in Hilbert space, there are only two way, two form of convergence, strong and weak. Now we can use both. That means that we did everything. And it so happened that when we use Hilbert space, which is very rich space, space of continuous functions, uh, which has integral and square, so we can apply weak and strong convergence for learning and have closed form solution. So for... computationally simple. For me, it is sign that it is right way.... because you, you don't need any heuristic here, just do whatever you want. But now, the only what left, it is concept of what is predicate.
- LFLex Fridman
Of predicate.
- VVVladimir Vapnik
But it is not statistics.
- LFLex Fridman
By the way, I like the fact that you think that heuristics are a mess that should be removed from the system. So, closed form solution is the ultimate goal.
- VVVladimir Vapnik
No, it's okay. And when you're using right instrument, you have closed form solution.
- 1:19:28 – 1:22:23
Intelligence and heuristics
- LFLex Fridman
Do you, do you think intelligence, human level intelligence, when we create it, will, um, will have something like a (laughs) closed form solution?
- VVVladimir Vapnik
You know, I, now, I'm looking on bounds, which I gave bounds for convergence. And when I looking for bounds, I thinking what is the most appropriate kernel for this bound would be. So if you know the thing, say, allow businesses views radial basis function. But looking on the bound, I think that I start to understand that maybe we need to make corrections to radial basis function to be closer to work better for this bounds. So, I'm again trying to understand what type of kernel have best approximation, not approximation, best fit to this bound.
- LFLex Fridman
Sure. So there's a l- there's a lot of interesting work that could be done in discovering better functions than radial basis functions for, for-
- VVVladimir Vapnik
Yeah, but-
- LFLex Fridman
... the kind of bounds you can find.
- VVVladimir Vapnik
... it but, it still comes from you, you, you're looking to mass and trying to understand what-
- LFLex Fridman
From your own mind, looking at the-
- VVVladimir Vapnik
Yeah, but-
- LFLex Fridman
I don't know.
- VVVladimir Vapnik
... but then, but then I trying to understand what, what will be good for that.
- LFLex Fridman
Yeah, but to me, there's still a beauty, again, maybe I'm a descendant of Alan Turing, to heuristics. To me, ultimately, intelligence would be a mess of heuristics. And-
- VVVladimir Vapnik
No-
- LFLex Fridman
... that's the engineering answer, I guess.
- VVVladimir Vapnik
... no, absolutely. When, when you're doing, say, self-driving cars, the great guy who will do this, it does not matter what theory behind that, who has a better feeling how to apply. But by the way, it is the same story about predicate, because you cannot create rule for... Situation is much more than you have rule for that. But maybe you can have more abstract rule, then it will be less literal. It is the same story about the ideas and, and ideas applied to come... to specific cases.
- LFLex Fridman
But still you should reach-
- VVVladimir Vapnik
You cannot avoid this.
- LFLex Fridman
Yes, of course. But y- you should still reach for the ideas to understand the science.
- VVVladimir Vapnik
Yeah, yeah.
- 1:22:23 – 1:25:11
Reasoning
- VVVladimir Vapnik
- LFLex Fridman
Let me kind of ask, do you think neural networks or functions can be made to reason? Sort of, what do you think... We've been talking about intelligence, but this idea of reasoning. There's a, there's an element of sequentially dissembling, interpreting the, the images. So, when y- when you think of, um, handwritten recognition, we kind of think that there would be a single... there's an input and output. There's not a recurrence.
- VVVladimir Vapnik
Yeah.
- LFLex Fridman
What do you think about sort of the idea of recurrence, of going back to memory and thinking through this sort of sequentially, um, mangling the different representations over and over until you arrive at, um, a conclusion? Or is ultimately, all that can be wrapped up into a function? (laughs)
- VVVladimir Vapnik
No, you're, you're suggesting that let us use this type of algorithm. When I starting thinking, I, first of all, starting to understand what I want. Can I write down what I want? And then I trying to formalize. And when I do that, I think I have to solve this problem. And... till now, I did not see situation where need-
- LFLex Fridman
You need recurrence.
- VVVladimir Vapnik
... recurrent.
- LFLex Fridman
But d- do you observe human beings?
- VVVladimir Vapnik
Yeah.
- LFLex Fridman
Do you try to... It's the imitation question, right? It seems that human beings reason this kind of sequentially. Sort of, does that inspire in you a thought that we need to add that into our intelligence systems? Y- you're saying... Okay, I mean, you've, you've, you've kind of answered saying, "Until now, I haven't seen a need for it." And so, because of that, you don't see a reason to think about it.
- VVVladimir Vapnik
You know, most of things, I don't understand. In reasoning in human, it is, for me, too complicated. For me, the most difficult part is...... to ask questions, the good questions. How it works, how, how people asking questions. I don't know this.
- 1:25:11 – 1:31:40
Role of philosophy in learning theory
- LFLex Fridman
You said that machine learning is not only about technical things, speaking of questions, but it's also about philosophy. So what role does philosophy play in machine learning? We talked about Plato, but generally, thinking in this philosophical way, does it have, how does philosophy and math fit together in your mind?
- VVVladimir Vapnik
First ideas and then their implementation. It's like predicate, like, uh, say, admissible set of functions. This comes together, everything. Because (clears throat) the first iteration of theory was done 50 years ago ............................ This is ... So everything there, if you have data, you can... and you have, and your set of function is not... has or has not big capacity.
- LFLex Fridman
Mm-hmm.
- VVVladimir Vapnik
So, ............................ You can do that. You can make structural risk minimization, control capacity. But you was not able to make admissible set of function good. Now, when suddenly realize that we did not use another idea of convergence, which we can, everything comes together.
- LFLex Fridman
But those are mathematical notions. Philosophy plays a role of simply saying that we should be swimming in the space of ideas.
- VVVladimir Vapnik
Let's, let's talk what is philosophy. Philosophy means understanding of life.
- LFLex Fridman
(laughs)
- VVVladimir Vapnik
So, understanding of life, say, people like Plato, they understand on very high abstract level of life. So... and whatever I doing, just implementation of my understanding of life. But every new step, it is very difficult. For example, to find this idea that we need weak convergence was not simple for me. And I-
- LFLex Fridman
So that required thinking about life a little bit. Hard to, hard, hard to trace, but there was some thought process.
- VVVladimir Vapnik
You know, I'm working by thinking about the same problem for 50 years or more, and again and again and again.
- LFLex Fridman
(laughs)
- VVVladimir Vapnik
I'm trying to be honest, and that is very important, not to be very enthusiastic.
- LFLex Fridman
Yeah.
- VVVladimir Vapnik
But concentrate on whatever we was not able to achieve, for example.
- LFLex Fridman
Patient.
- VVVladimir Vapnik
Yeah. And understand why. And now, yeah, I understand that because I believe in maths, I believe that ... in Wigner's idea. But now, when I see that there are only two way of convergence, and we're using both, that means that we must do... as well as people doing. But now, exactly in philosophy and what we know about predicate, what we, how we understand life, can be described as a predicate. I thought about that, and that is more or less obvious level of symmetry. But next, I have a feeling it's something about structures. But I don't know how to formulate, how to measure, measure of structure and all this stuff. And guy who will solve this challenge problem then, when we were looking how he did it, probably just only symmetry is not enough.
- LFLex Fridman
But something like symmetry will be there, the structures-
- VVVladimir Vapnik
Oh, absolutely, symmetry will be there. Level of symmetry will be there. And level of symmetry, anti-symmetry, diagonal, vertical and... I, I, I even don't know how you can use in different direction idea of symmetry, it's very general. But it will be there. I think that people very sensitive to idea of symmetry. But there are several ideas like symmetry, as I would like to learn.
- LFLex Fridman
(laughs) .
- VVVladimir Vapnik
But you cannot learn just thinking about that. You should do challenging problems and then analyze them, why, why it was, we was able to solve them, and then we will see. Very simple things, it's not easy to find.
- LFLex Fridman
(laughs)
- VVVladimir Vapnik
Even with talking about this every time-
- LFLex Fridman
Yeah.
- VVVladimir Vapnik
... about your, your... I, I was surprised, I, I, I tried to understand, is people describe in language strong convergence mechanism for learning? I did not see, I don't know. But weak convergence, this dark story and story like that, when you will explain to kid, you will use weak convergence argument. It looks like it does like this.... but, but when they try to formalize, we're just ignoring this. Why? Why 50 years from start of machine learning?
- LFLex Fridman
And that's the role of philosophy. Thinking about like-
- VVVladimir Vapnik
I think, I, I think that maybe... I don't know. Maybe we should show the also we should blame for that, because empirical risk minimization and, and all this stuff. And if you read now textbooks, they're just about ￢ﾴ�.￢ﾴﾕ empirical risk minimization. They don't look in for another problem like admissible set.
- 1:31:40 – 1:35:08
Music (speaking in Russian)
- VVVladimir Vapnik
- LFLex Fridman
But on the topic of life, perhaps we, you could talk in Russian for a little bit. What's your favorite memory from childhood? (Russian)
- VVVladimir Vapnik
Ooh. Music.
- LFLex Fridman
How about... Can you try to answer in Russian?
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- VVVladimir Vapnik
(Russian)
- LFLex Fridman
(Russian)
- 1:35:08 – 1:44:49
Mortality
- LFLex Fridman
Do you ponder your own mortality?
- VVVladimir Vapnik
It-
- LFLex Fridman
Do you think about it? Do you fear it? Do you draw insight from it?
- VVVladimir Vapnik
About mortality? Oh, yeah.
- LFLex Fridman
Are you afraid of death?
- VVVladimir Vapnik
Uh, not too much. Not too much. It just pitches it, uh, you will not be able to do something which I think. I have a feeling to do that. For example, I will be very happy to work with guys, theoretician from music, to write this collection of description what, what, how they describe music, how they use predicate, and from art as well. Then take what is in common and try to understand predicate which is absolute for everything, and try to-
- LFLex Fridman
And then use that for visual recognition and see if there is a connection.
- VVVladimir Vapnik
And use that for vis- yeah, yeah, exactly.
- LFLex Fridman
Ah, th- there's still time. We got time. (laughs)
- VVVladimir Vapnik
(laughs) Yeah.
- LFLex Fridman
We got time.
- VVVladimir Vapnik
It, it, it's, it's, it's take years and years and years.
- LFLex Fridman
You think so?
- VVVladimir Vapnik
Yeah. It's, it's a long way.
- LFLex Fridman
Well, see, you've got the patient, mathematic- mathematician's mind. I think it could be done very quickly and very beautifully. I think it's a really elegant idea.
- VVVladimir Vapnik
Yeah, but also-
- LFLex Fridman
Well, one, some of many, you could say.
- VVVladimir Vapnik
... you know the, the most time, it is not to make this collection, to understand what is a common to think about that once again and again and again.
- LFLex Fridman
Again and again and again, but I think sometimes, especially just when you say this idea now, even just putting together the collection and looking at the different...... sets of data, language, trying to interpret music, criticize music, and images. I think there will be sparks of ideas that will come. Of course, again and again, you'll come up with better ideas. But even just that notion-
- VVVladimir Vapnik
Yeah.
- LFLex Fridman
... is a beautiful notion.
- VVVladimir Vapnik
I even have some example. So I have friend who was specialist in Russian poetry. She is professor of pro- of Russian poetry. He did not write, m- um, poems, but she know a lot of stuff. She make, uh, book, several books, and one of them is, uh, a collection of Russian poetry. She have images of Russian poetry. She collect all images of Russian poetry. And I ask her to do following. You have NIPS digit recognition and you get hundred digits or maybe less than hundred. I, I don't remember, maybe 50 digits. And try from poetical point of view, describe every image which she see using only words of images of Russian poetry. And she did it. And then, we tried to... I call it learning if using privileged information. I call it privileged information.
- LFLex Fridman
Mm-hmm.
- VVVladimir Vapnik
You have on two languages. One language is just image of digit and another language, poetic description of this image. And this is privileged information. And there is a algorithm when we are working using privileged information, you're doing well, web- better, much better. So-
- LFLex Fridman
(laughs) So there's something there.
- VVVladimir Vapnik
Something there. And there is a... in, and you see, she unfortunately died. Uh, the collection of digits and poetic descriptions of these digits.
- LFLex Fridman
(laughs)
- VVVladimir Vapnik
Yes.
- LFLex Fridman
So there's some- something there in that poetic description.
- VVVladimir Vapnik
But I think that there is a abstract ideas on the plateau level of ideas.
Episode duration: 1:44:55
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode bQa7hpUpMzM
Get more out of YouTube videos.
High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.
Add to Chrome