Ishan Misra: Self-Supervised Deep Learning in Computer Vision | Lex Fridman Podcast #206

Lex Fridman 0:00 The following is a conversation with Ishan Misra, research scientist at Facebook AI Research who works on self-supervised machine learning in the domain of computer vision, or in other words, making AI systems understand the visual world with minimal help from us humans. Transformers and Self-Attention has been successfully used by OpenAI's GPT-3 and other language models to do self-supervised learning in the domain of language. Ishan, together with Yann LeCun and others, is trying to achieve the same success in the domain of images and video. The goal is to leave a robot watching YouTube videos all night, and in the morning come back to a much smarter robot. I read the blog post, Self-Supervised Learning: The Dark Matter of Intelligence by Ishan and Yann LeCun, and then listened to Ishan's appearance on the excellent Machine Learning Street Talk podcast, and I knew I had to talk to him. By the way, if you're interested in machine learning and AI, I cannot recommend the ML Street Talk podcast highly enough. Those guys are great. Quick mention of our sponsors: On It, The Information, Grammarly, and Athletic Greens. Check them out in the description to support this podcast. As a side note, let me say that for those of you who may have been listening for quite a while, this podcast used to be called Artificial Intelligence podcast because my life passion has always been, will always be artificial intelligence, both narrowly and broadly defined. My goal with this podcast is still to have many conversations with world-class researchers in AI, math, physics, biology, and all the other sciences, but I also want to talk to historians, musicians, athletes, and of course, occasionally comedians. In fact, I'm trying out doing this podcast three times a week now to give me more freedom with guest selection, and maybe, uh, get a chance to have a bit more fun. Speaking of fun, in this conversation, I challenge the listener to count the number of times the word "banana" is mentioned. Ishan and I use the word "banana" as the canonical example at the core of the hard problem of computer vision, and maybe the hard problem of consciousness. This is the Lex Fridman podcast, and here is my conversation with Ishan Misra.

Lex Fridman 2:27 What is self-supervised learning? And maybe even give the, the bigger basics of what is supervised and semi-supervised learning, and maybe why is self-supervised learning a better term than unsupervised learning?

Ishan Misra 2:39 Uh, let's start with supervised learning. So, typically for machine learning systems, the way they're trained is you get a bunch of humans. The humans point out particular concepts with it. In the case of images, you want the humans to come and tell you what is pos ... like, what is present in the image, draw boxes around them, draw masks of, like, things, pixels which are of particular categories or not. Uh, for NLP, again there are, like, lots of these particular tasks, say, about sentiment analysis, about entailment and so on. So, typically for supervised learning we get a big corpus of such annotated or labeled data, and then we feed that to a system, and the system is really trying to mimic. So, it's taking this input of the data and then trying to mimic the output. So, it looks at an image, and the human has tagged that this image contains a banana, and now the system is basically trying to mimic that. So, it starts its learning signal. And so for supervised learning we try to gather lots of such data, and we train these machine learning models to imitate the input/output. And the hope is basically by doing so, now on unseen or, like, new kinds of data, this model can automatically learn to predict these concepts. So, this is a standard sort of supervised setting. For semi-supervised setting, uh, the idea typically is that you have of course all of the supervised data, but you have lots of other data which is unsupervised or which is, like, not labeled. Now, the problem basically with supervised learning and why you actually have all of these alternate sort of learning paradigms is supervised learning does ... just does not scale. So, if you look at for computer vision, the sort of largest, one of the most popular datasets is ImageNet. Right? So, the entire ImageNet dataset has about 22,000 concepts and about 14 million images. So, these concepts are j- basically just nouns, and they're annotated on images. And this entire dataset was a mammoth data collection effort. It actually, uh, gave rise to a lot of powerful learning algorithms. It's credited with, like, sort of the rise of deep learning as well. But this dataset took about 22 human years to collect, to annotate, and it's not even that many concepts, right? It's not even that many images. 14 million is nothing, really. Um, like, you have about, I think, 400 million images or so, or even more than that uploaded to most of the popular sort of social media websites today. So, now supervised learning just doesn't scale. If I want to now annotate more concepts, if I want to have this ... various types of fine-grained concepts, then it won't really scale. So, y- now you come after these sort of different learning paradigms. For example, semi-supervised learning, where the idea is y- of course, you have this annotated corpus of supervised data, and you have lots of these unlabeled images, and the idea is that the algorithm should basically try to measure some kind of consistency or really try to measure some kind of, uh, signal on this sort of unlabeled data to make itself more confident about what it's really trying to predict. So, by access to this lots of unlabeled data, the idea is that the algorithm actually learns to be more confident and actually gets better at predicting these concepts. And now, we come to the other extreme, which is, like, self-supervised learning. The idea basically is that the machine or the algorithm should really discover concepts or discover things about the world or learn representations about the world which are useful without access to explicit human supervision.

Lex Fridman 5:41 So, the word "supervision" is still in the term "self-supervised." So, what is the supervision signal? And maybe that perhaps is when Yann LeCun and you argue that unsupervised is the incorrect terminology here.

Ishan Misra 5:55 Right.

Lex Fridman 5:55 So, what is the supervision signal when the humans aren't part of the picture, or not, uh, a big part of the picture?

Ishan Misra 6:02 Right.So, self-supervised, the reason it- it has the term supervise in itself is because you're using the data itself as supervision. So, because the data serves as its own source of supervision, it's self-supervised in that way. Now, the reason a lot of people ... I mean, we did it in that blog post with Yann, but a lot of other people have also argued for using this term self-supervised. So, starting from like '94 from Virginia Dasz group, uh, at I think UCSD and now, now she's at UCSD. Uh, Jitendra Malik has sa- said this a bunch of times as well. So, you have supervised, and then unsupervised basically means everything which is not supervised, but that includes stuff like semi-supervised, that includes other ... Like transductive learning, lots of other sort of settings. So, that's the reason like now people are preferring this term self-supervised because it explicitly says what's happening. The data itself is the source of supervision and any sort of learning algorithm which tries to extract just sort of data supervision signals from the data itself is a self-supervised learning algorithm.

Lex Fridman 6:59 But there is within the data a set of tricks which unlock the supervision.

Ishan Misra 7:04 Right.

Lex Fridman 7:05 So, can you give maybe some examples? And th- there's a- there's innovation, ingenuity required to unlock that supervision.

Ishan Misra 7:12 Right.

Lex Fridman 7:12 The data doesn't just speak to you some ground truth. You have to do some kind of trick.

Ishan Misra 7:17 Right.

Lex Fridman 7:17 Uh, so I don't know what your favorite domain is. So, you specifically s- specialize in visual learning, but is there favorite examples maybe in language or other domains?

Ishan Misra 7:26 Perhaps the most successful applications have been in, uh, NLP, nat- language processing. So, the idea basically being that you can train models that can ... Uh, you have a sentence and you mask out certain words, and now these models learn to predict the masked out words. So, if you have like, "The cat jumped over the dog." So, you can basically mask out cat and now you're essentially asking the model to predict, "What was missing? What did I mask out?" So, the model is going to predict basically a distribution over all the possible words that it knows. And probably it has, like if it's a- a well-trained model, it has a h- sort of higher probability density for this word cat. For vision, I would say the sort of more, uh, I mean, the easier example which is not as widely used these days, uh, is basically say for example video prediction.

Lex Fridman 8:11 Mm-hmm.

Ishan Misra 8:12 So, video is again a sequence of things. So, you can ask the model, so if you have a video of say 10 seconds, you can feed in the first nine seconds through a model and then ask it, "Hey, what happens e- basically in the 10th second? Can you predict what's going to happen?" And the idea basically is because the model is predicting something about the data itself, of course you didn't n- didn't need any human to tell you what was happening because the 10-second video was naturally captured. Because the model is predicting what's happening there, it's g- uh, going to automatically learn something about the structure of the world. How objects move, object permanence, and these kinds of things. Uh, so like if I have something at the edge of the table, it'll fall down. Uh, things like these which you really don't have to sit and annotate. In a supervised learning setting, I would have to sit and annotate, "This is a cup. Now I move this cup. This is still a cup. And now I move this cup, it's still a cup and then it falls down, and this is a fallen down cup." So, I won't have to annotate all of these things in a self-supervised setting.

Lex Fridman 9:02 Isn't that kind of a brilliant little trick of taking a series of data that is consistent and removing one element in that series, and then, uh, teaching the algorithm to predict that element? Isn't that ... First of all, that's quite brilliant. Um, it seems to be applicable in anything that, um, has the constraint of being a- a- a sequence that is consistent with the physical reality.

Ishan Misra 9:31 Right.

Lex Fridman 9:31 Um, the question is, are there other tricks like this that can generate the, uh, self-supervision signal?

Ishan Misra 9:37 So, sequence is possibly the most widely used one in NLP. For vision, the one that is actually used for like images which is very popular these days, is basically taking an image and now taking different crops of that image. So, you can basically decide to crop say the top left corner and you crop say the bottom right corner, and asking the network to basically, uh, present it with a choice saying that, "Okay, now you have, uh, this image, you have this image. Are these the same or not?" And so the idea basically is that because different cro- like in an image, different parts of the image are going to be related. So, for example, if you have a chair and a table, uh, basically these things are going to be close by. Uh, versus if you take, uh, again, if you have like a zoomed in picture of a chair, if you're taking different crops, it's going to be different parts of the chair. So, the, uh, idea basically is that different crops of the image are related and so the features or the representations that you get from these different crops should also be related. So, this is possibly the most like widely used trick, uh, these days for, uh, self-supervised learning in computer vision.

Lex Fridman 10:35 So again, using the, uh, consistency that's inherent to physical reality in- in visual domain that's, you know, parts of an image are consistent. And then in the, uh, language domain or s- anything that has sequences like language or something that's like a time series that you can chop off parts in time.

Ishan Misra 10:55 Right.

Lex Fridman 10:55 It's similar to the story of, uh, RNNs and CNNs. Um, of RNNs and ConvNets.

Lex Fridman 11:02 You and Yann LeCun wrote the blog post in March 2021 titled Self-Supervised Learning: The Dark Matter of Intelligence. Can you summarize this blog post and maybe explain the main idea or set of ideas?

Ishan Misra 11:15 The blog post was mainly about sort of just telling ... I mean, this is really a, uh, accepted fact I would say for a lot of people now that self-supervised learning is something that is going to be ... Uh, play an important role for machine learning algorithms that come in the future, and even now.

Lex Fridman 11:30 Uh, let me just comment that, uh, we don't yet have a good understanding what dark matter is.

Ishan Misra 11:36 (laughs) That's true.

Lex Fridman 11:37 So- (laughs)

Ishan Misra 11:37 (laughs) So, the idea basically being-

Lex Fridman 11:39 So, maybe the metaphor doesn't exactly transfer but maybe it- maybe this actually perfectly transfers that we don't know. We have a- we have an inkling that it'll be a big part of whatever solving intelligence looks like.

Ishan Misra 11:51 Right. So, I think self-supervised learning the way it's done right now is I would say like the first step towards what it probably should end up like learning or what it should enable us to do.

Lex Fridman 12:00 Yeah.

Ishan Misra 12:00 So, the idea for, uh, that particular piece was ...Self-supervised learning is going to be a very powerful way to learn common sense about the world, or like stuff that is really hard to label. For example, like is this, uh, piece over here heavier than the cup? Now, for all these kinds of things you'll have to sit and label these things, so supervised learning is clearly not going to scale. So, what is the thing that's actually going to scale? It's probably going to be an agent that can either actually interact with it, so lift it up, or, uh, observe me doing it. So, if I'm basically lifting these things up it can probably reason about, "Hey, this is taking him more time to lift up," or, "The velocity's different." Where- Whereas the velocity for this is different, probably this one is heavier. So, essentially by observations of the data you should be able to infer a lot of things about the world without someone explicitly telling you, "This is heavy. This is not. Uh, this is something that can pour. This is something that cannot pour. This is somewhere that you can sit. This is not somewhere that you can sit."

Lex Fridman 12:53 But you just mentioned the ability to interact with the world. There's so many questions that are yet to be... that are still open which is, how do you select a set of data over which the self-supervised, uh, learning process works? How much interactivity, like in the active learning or the machine teaching context is there? What are the reward signals? Like how much actual interaction there is with the physical world, that kind of thing.

Ishan Misra 13:21 Right.

Lex Fridman 13:21 Uh, so that- that's a... that could be a huge que- and then on top of that, which I have a million questions about, which we don't know the answers to but it's worth talking about, is how much reasoning is involved? How much accumulation of knowledge versus something that's more akin to learning, or whether that's the- the same thing. But, so we're like... it is truly dark matter.

Ishan Misra 13:46 We don't know how exactly to do it-

Lex Fridman 13:48 Yeah.

Ishan Misra 13:49 ... but we are... I mean a lot of us are actually convinced that it's going to be a sort of major thing in machine learning and-

Lex Fridman 13:55 So let me reframe it then, that- that human supervision cannot be at large scale, the source of the solution to intelligence.

Ishan Misra 14:03 Right.

Lex Fridman 14:04 So there has... we... the machines have to discover the supervision in the natural signal of the world.

Ishan Misra 14:09 Right. I mean the other thing is also that humans are not particularly good labelers, they're not very consistent. Uh, for example, like what's the difference between a dining table and a table? Is it just the fact that one... like if you just look at a particular table, what makes us say one is a dining table and the other is not? Uh, humans are not particularly consistent, they're not like very good sources of supervision for a lot of these kind of edge cases. So, it may be also the fact that if we want a... like want an algorithm or want a machine to solve a particular task for us, we can maybe just specify the end goal. Uh, and like the stuff in between, uh, we really probably should not be specifying because we're not... maybe you're going to confuse it a lot actually.

Lex Fridman 14:49 Well, humans can't even answer the meaning of life so we don't... I'm not sure if we're good supervisors of

Lex Fridman 14:54 the end goal either. So let me ask you about categories. Humans are not very good at telling the difference between what is and isn't a table, like you mentioned.

Ishan Misra 15:01 Right.

Lex Fridman 15:02 Um, do you think it's possible... let me, let me ask you like a... pretend you're Plato. Um, is, is it possible to create a pretty good taxonomy of objects in the world? It seems like a lot of approaches in machine learning kind of assume a hopeful vision that it's possible to construct a perfect taxonomy. Or it exists, perhaps out of our reach, but we can always get closer and closer to it. Or is that a hopeless pursuit?

Ishan Misra 15:31 I think it's hopeless in some way. So the thing is, for any particular categorization that you create, if you have a discrete sort of categorization, I can always take the nearest two concepts or I can take a third concept and I can blend it in, and I can create a new category.

Lex Fridman 15:44 Yeah.

Ishan Misra 15:44 So if you were to enumerate end categories, I will always find an N plus one category for you, that's not going to be in the end categories. And I can actually create not just N plus one, I can very easily create far more than N categories. The thing is, uh, a lot of things we talk about are actually compositional, so it's really hard for us to come and sit- sit and enumerate all of these out. And they come across in various weird ways, right? Like you have... like a croissant and a donut come together to form a cronut.

Lex Fridman 16:09 Yeah.

Ishan Misra 16:09 So if you were to like enumerate all the foods up until, I don't know, whenever the cronut was, about 10 years ago or 15 years ago, then this entire thing called cronut would not exist.

Lex Fridman 16:19 Yeah, I remember there was a... the most awesome video of a cat wearing a monkey costume.

Ishan Misra 16:24 (laughs) Yeah. Yes.

Lex Fridman 16:26 (laughs) People should look it up. It's great. So is that a monkey, is, uh, or is that a cat? It's a very difficult philosophical question. So there is a concept of similarity between objects. So you think that can take us very far, just kind of getting a good function, a good way to tell which parts of things are similar and which parts of things are very different?

Ishan Misra 16:50 I think so, yeah. So you don't necessarily need to name everything or assign a name to everything to be able to use it, right? So a lot... there are like lots of-

Lex Fridman 16:59 Shakespeare said that. "What's in a name?"

Ishan Misra 17:01 What's in a name? Yeah.

Lex Fridman 17:02 Yeah, okay.

Ishan Misra 17:03 And I mean, a lot- lots of like, for example, animals, right? They don't have necessarily a well-formed like syntactic language but they're able to go about their day perfectly. The same thing happens for us. So I mean, we probably look at things and we figure out, oh, this is similar to something else that I've seen before and then I can probably learn how to use it. So I haven't seen all the possible doorknobs in the world.

Lex Fridman 17:26 Yes.

Ishan Misra 17:26 But if you show me... like I was able to get into this particular place fairly easily. I've never seen that particular doorknob. So I of course relate it to all the doorknobs that I've seen and I know exactly how it's going to open, or I have a pretty good idea of how it's going to open. And I think this kind of translation between experiences only happens because of similarity, because I'm able to relate it to a doorknob. If I related it to a hairdryer, I would probably be stuck still outside, not able to get in. (laughs)

Lex Fridman 17:49 (laughs) Again, a bit of a philosophical question, but is... can similarity take us all the way to understanding a thing? Can having a good function that compares objects get us to understand something profound about singular objects?

Ishan Misra 18:07 I think I'll ask you a question back. What does it mean to understand objects?

Lex Fridman 18:11 Well, let me tell you what that's similar to. No. (laughs)

Ishan Misra 18:15 (laughs)

Lex Fridman 18:15 Uh, (laughs) I, so there is, there is an idea of sort of reasoning by analogy kind of thing. I think understanding is the process of placing that thing in some kind of network of knowledge that you have, that it- it perhaps is fundamentally related to other concepts. So, it's not like understanding is fundamentally related by, like, composition of other concepts and maybe in relation to other concepts. Um, and maybe, like, deeper and deeper understanding is maybe just adding more edges to that, uh, to that graph somehow. Uh, so maybe it is a composition of similarities. I mean, ul- ultimately, it is ... I suppose it is a kind of embedding in that wisdom space. (laughs)

Ishan Misra 19:01 (laughs) Yeah. All right. Okay. Wisdom space is good. Uh-

Lex Fridman 19:06 (laughs)

Ishan Misra 19:06 ... I think, I do think, right, so similarity does get you very, very far.

Lex Fridman 19:10 Yeah.

Ishan Misra 19:10 Is it the answer to everything? I mean, I don't even know what everything is, but it's going to take us really far, um, and I think ... The thing is, things are similar in very different contexts, right? So, an elephant is similar to, I don't know, another sort of wild animal, let's just pick, I don't know, lion, in s- in a different way because they're both four-legged creatures, uh, they're also land animals, but of course they're very different in a lot of different ways. So, elephants are like herbivores, um, lions are not. So, similarity does, uh, similarity and particularly dissimilarity also sort- uh, actually helps us understand a lot about things. And so that's actually why I think discrete categorization is very hard, just like forming this particular category of elephant and a particular category of lion, maybe it's good for like, just like taxonomy, biological taxonomies. But when it comes to like other things which are not as, maybe, uh, for example, like grilled cheese, right?

Lex Fridman 20:01 Mm-hmm.

Ishan Misra 20:01 I have a grilled cheese, I dip it in tomato and I keep it outside. Now, is that still a grilled cheese or is that something else?

Lex Fridman 20:06 Right. So, categorization is still very useful for solving problems, but is your intuition then sort of, uh, the self-supervised should be the, to borrow Yann LeCun's terminology, uh, should be the cake and then categorization, the classification, the, maybe the supervised, like, layer should be just like the thing on top, the cherry or the icing or whatever?

Ishan Misra 20:31 Yeah.

Lex Fridman 20:31 So, if you make it the cake, it gets in the way of learning?

Ishan Misra 20:35 Yeah. If you make it the cake, then you don't, we won't be able to sit and annotate everything. That's as simple as it is. Like, that's my very practical view on it. It's just, um, I mean, in my PhD I sat down and annotated like a bunch of cars for one of my projects, and very quickly I was just like, it was in a video and I was basically drawing bound- boxes around all these cars. And I think I spent about a week doing all of that, and I barely got anything done.

Lex Fridman 20:57 (laughs)

Ishan Misra 20:58 And basically, this was I think my first year of my PhD, or like f- uh, second year of my master's. And then by the end of it, I'm like, "Okay, this is just hopeless. I can keep doing it." And when I'd done that, uh, someone came up to me and they basically told me, "Oh, th- this is a pickup truck. This is not a car." And that's when I'm like, "A-ha! This actually makes sense because a pickup truck is not re-" Like, what was I annotating? Was I annotating anything that is mobile? Uh, or was I annotating particular sedans, or was I annotating SUVs? What was I doing?

Lex Fridman 21:23 By the way, the annotation was bounding boxes, or-

Ishan Misra 21:25 Bounding boxes. Yeah.

Lex Fridman 21:27 There's so many deep profound questions here that you c- uh, you're almost cheating your way out of by doing self-supervised learning, by the way, which is like, what makes for an object?

Ishan Misra 21:37 Mm-hmm.

Lex Fridman 21:37 I suppose to solve intelligence, maybe you don't ever need to answer that question. (sniffs) I mean, this is the question that anyone that's ever done annotation, because it's so painful, gets to ask. Like, why am I doing a, um, like, drawing very careful line around this object? (laughs)

Ishan Misra 21:55 (laughs)

Lex Fridman 21:55 Like, what- what- what is the value? I remember when I first saw semantic segmentation, where you have, uh, like instant segmentation where you have a very exact line around the object in a 2D plane, of a fundamentally 3D object projected on a 2D plane. So, you're drawing a line around a car that might be occluded, there might be another thing in front of it-

Ishan Misra 22:18 Right.

Lex Fridman 22:18 ... but you're still drawing the line of- of the part of the car that you see. How is that the car? Why- why is that the car? Like, I had like an existential crisis every time. (laughs)

Ishan Misra 22:30 (laughs)

Lex Fridman 22:31 Like, how is that going to help us understand, uh, solve computer vision? I'm not sure I have a good answer to what's better, and I'm not sure I share the confidence that you have, that, um, self-supervised learning can take us far. I think I'm more and more convinced that it's a very important component, but I still feel like we need to understand what makes ... Like this, this, uh, like dream of maybe what's called like symbolic AI-

Ishan Misra 23:01 Yeah.

Lex Fridman 23:01 ... of- of arriv- Like, once you have this common sense base, be able to play with these concepts and build graphs or hierarchies of concepts on top in order to then, uh, like form a- a- a- a deep sense of this three-dimensional world, or four-dimensional world, and be able to reason and then project that onto a 2D plane in order to inter- interpret a 2D image.

Lex Fridman 23:28 Can I ask you just an, uh, out there question? I remember, I think, uh, I think Andrej Karpathy had a blog post about computer vision, uh, like being really hard.

Ishan Misra 23:38 Mm-hmm.

Lex Fridman 23:39 I forgot what the title was, but it was many, many years ago. And he had, I think President Obama stepping on a scale and there was humor, and there was a bunch of people laughing and whatever. And, uh, the interesting ... There's a lot of interesting things about that image, and I think Andrej highlighted a bunch of things about the image that us humans are able to immediately understand, like the idea, I think, of gravity and-

Ishan Misra 24:01 Yeah.

Lex Fridman 24:01 ... that you can, you have the concept of a weight, you have a, you immediately project, uh, because of our knowledge of pose and how human bodies are constructed, you understand how the forces are being applied with the human body. Uh, the really interesting other thing that you're able to understand, there's multiple people looking at each other in the image.

Ishan Misra 24:20 (laughs)

Lex Fridman 24:20 Uh, you're able to have a mental model of what the people are thinking about. You're able to infer like, "Oh, this person is probably thinks, like is laughing at how humorous the situation is, and this person is confused about what the situation is because they're looking this way." We're able to infer all of that. So, that's human vision. How difficult is computer vision? Like, in order to achieve that level of understanding. And maybe how big of a part does self-supervised learning play in that, do you think? And do you still, you know, back, that was like over a decade ago, I think Andre and I think a lot of people agreed is, (laughs) computer vision is really hard.

Ishan Misra 25:02 Yeah.

Lex Fridman 25:03 Do you still think computer vision is really hard?

Ishan Misra 25:06 I think it is, yes. And getting to that kind of understanding, uh, I mean, it's really out there. So, if you ask me to solve just that particular problem, I can do it the supervised learning route. I can always construct a data set and basically predict, "Oh, is there humor in this or not?"

Lex Fridman 25:21 (laughs)

Ishan Misra 25:21 (laughs) Of course, I can do it.

Lex Fridman 25:22 Actually, that's a good question. Do you think you can... Okay, okay, do you think you can do human-supervised annotation of humor?

Ishan Misra 25:29 To some extent, yes. I'm sure it'll work. I mean, it won't be, it won't be as bad as like randomly guessing. I'm sure it can still predict whether it's humorous or not, in some way.

Lex Fridman 25:37 Yeah, maybe like Reddit upvotes is the signal. I don't know.

Ishan Misra 25:40 Right.

Lex Fridman 25:41 Okay.

Ishan Misra 25:41 I mean, it won't do a great job, but it'll do something.

Lex Fridman 25:43 Right.

Ishan Misra 25:43 It may actually be like it may find certain things which are not humorous, humorous as well, which is going to be bad for us, but, I mean, it'll do a, it wouldn't be random.

Lex Fridman 25:52 Yeah, kind of like my sense of humor.

Ishan Misra 25:53 (laughs)

Lex Fridman 25:54 Okay, so, uh, fine. So, you can that particular problem, yes. But the general problem you're saying is hard.

Ishan Misra 25:59 The general problem is hard. And, I mean, self-supervised learning is not the answer to everything. Of course, it's not. I think, uh, if you have machines that are going to communicate with humans at the end of it, you want to understand what the algorithm is doing, right? You want it to be able to like produce an output that you can decipher, that you can understand, or it's actually useful for something else, which again, is a human... So, at, at some point in this sort of entire loop, a human steps in.

Lex Fridman 26:23 Mm-hmm.

Ishan Misra 26:23 And now, this human needs to understand what, what's going on. So at, and at that point, this entire notion of language or semantics really comes in. If the machine just spits out something and if we can't understand it, then it's not really that useful for us. So, self-supervised learning is probably going to be useful for a lot of the things before that part.

Lex Fridman 26:40 Mm-hmm.

Ishan Misra 26:40 Before the machine really needs to communicate a particular kind of output with a human. Um, because, I mean, otherwise, how, how is it going to do that without language?

Lex Fridman 26:50 Or some kind of communication. But you're saying that it's possible to build a big base of understanding or whatever, of, um, what's a better w-

Ishan Misra 26:58 Concepts.

Lex Fridman 26:58 Of concepts.

Ishan Misra 26:59 Concepts, yeah.

Lex Fridman 26:59 Of like common sense concepts.

Ishan Misra 27:01 Right.

Lex Fridman 27:02 Supervised learning in the context of computer vision is something you've focused on, but that's a really hard domain and it's kind of the cutting edge of what we're, as a community, working

Lex Fridman 27:12 on today. Can we take a, a little bit of a step back and look at language? Can you summarize the history of success of self-supervised learning in natural language processing, language modeling? What are transformers?

Ishan Misra 27:25 (laughs)

Lex Fridman 27:25 What is, uh, the masking, the sentence completion that you mentioned before? Um, how does it lead us to understand anything? Semantic meaning of words, syntactic role of words and sentences?

Ishan Misra 27:37 So, I'm, of course, not the expert in NLP. Uh, I kind of follow it, um, a little bit from the sides. So, the main sort of, uh, reason why all of this masking stuff works is, I think it's called the distributional hypothesis in NLP. The idea basically being that words that occur in the same context should have similar meaning. So, if you have the blank jumped over the blank, it basically, uh, whatever is like in the first blank is basically an object that can actually jump, is going to be something that can jump. So, a cat or a dog or, I don't know, sheep, something, all of these things can basically be in that particular context. And now, so essentially, the idea is that if you have words that are in the same con- context and you predict them, you're going to learn, uh, a lots of useful things about how words are related because you're predicting by looking at their context what the word is going to be. So, in this particular case, the blank jumped over the fence. So now, if it's a sheep, the sheep jumped over the fence, the dog jumped over the fence. So essentially, the ya- algorithm or the representation basically puts together these two concepts together. So, it says, "Okay, dogs are going to be kind of related to sheep because both of them occur in the same context." Of course, now you can decide depending on your particular application downstream. You can say that dogs are absolutely not related to sheep because, well, I don't, I really care about, you know, dog food, for example. I'm a dog food person and I really want to give this dog food to this particular animal. So, depending on what your downstream application is, of course, this notion of, uh, similarity or this notion or this common sense that you've learned may not be applicable. But the point is basically that this, um, just predicting what the blanks are is going to take you really, really far.

Lex Fridman 29:11 And so, the- there's a nice feature of language that the number of words in a particular language is very large, but it's finite and it's actually not that large in the grand scheme of things. I, I still gotta... because we take it for granted. So first of all, when you say masking, you're talking about this very process of the blank, of removing words from a sentence and then having the knowledge of what word went there in the ini- initial data set. That's the ground truth that you're training on and then you're asking the neural network to predict what goes there. That, that's, that's like a little trick.

Ishan Misra 29:49 Yeah.

Lex Fridman 29:49 It's a really powerful trick.

Ishan Misra 29:50 Yeah.

Lex Fridman 29:50 The question is how far that takes us, and the other question is, is there other tricks? 'Cause it, to me, it's very possible there's other very fascinating tricks. I'll give you an example in, um...... in autonomous driving, there's a bunch of tricks-

Ishan Misra 30:06 Right.

Lex Fridman 30:06 ... that give you the self-supervised signal back. For example, I mean, very similar to sentences, but not really, which is you have signals from humans driving the car-

Ishan Misra 30:20 Mm-hmm.

Lex Fridman 30:20 ... because a lot of us drive cars to places, and so you can ask the neural network to predict what's going to happen in the next two seconds for a safe navigation through the environment, and the, and the signal is, comes from the fact that you also have knowledge of what happened in the next two seconds because you have video of the data.

Ishan Misra 30:42 Right.

Lex Fridman 30:42 The question in autonomous driving, as it is in language, can we learn how to drive autonomously based on that kind of self-supervision? Probably the answer is no. The question is, how good can we get?

Ishan Misra 30:57 Right.

Lex Fridman 30:57 And the same with language, how good can we get? And are there other tricks? Like, we get sometimes super excited by this trick that works really well, but I wonder... It's almost like mining for gold. I wonder how many signals there are in the data that could be leveraged-

Ishan Misra 31:14 Right.

Lex Fridman 31:14 ... that are, like, there, right? Is, is that, uh... I just wanted to kind of linger on that because sometimes it's easy to think that maybe this masking process is self-supervised learning.

Ishan Misra 31:24 Right.

Lex Fridman 31:25 No, it's only-

Ishan Misra 31:25 One.

Lex Fridman 31:26 ... one method. Uh, so there could be many, many other methods, many tricky, uh, methods, maybe interesting ways to leverage human computation in very interesting ways that might actually border on semi-supervised learning, something like that. Uh, obviously, the internet is generated by humans-

Ishan Misra 31:43 Yeah.

Lex Fridman 31:43 ... at the end of the day. So, all that to say is, what's your sense, in this particular context of language, how far can that, uh, masking process take us?

Ishan Misra 31:54 So, it has stood the test of time, right? I mean, so Word2vec, uh, the initial sort of, uh, NLP technique that was using this, to now, for example, like all the BERT and all these, uh, big models that we get, um, BERT and RoBERTa, for example. All of them are still sort of based on the same principle of masking. It's taken us really far. I mean, you can actually do things like, oh, these two sentences are similar or not, whether this particular sentence follows this other sentence in terms of logic, so entailment. You can do a lot of these things with this, just this masking trick.

Lex Fridman 32:23 Yeah.

Ishan Misra 32:24 Um, so I'm not sure if I can predict how m- how far it can take us, because, like, when it first came out, when like Word2vec was out, uh, I don't think a lot of us would have imagined that this would actually help us do some kind of, like, entailment problems and really that well. And so just the fact that by just scaling up the amount of data that we're training on and, like, using better and more powerful neural network architectures has taken us from that to this is just showing you how maybe poor predictors we are. Like, uh, h- as humans, how poor we are at predicting how successful a particular technique is going to be. So, I think I can say something now, but, like, 10 years from now, I'll look completely stupid (laughs) basically predicting this.

Lex Fridman 33:02 In the language domain, is there something in your work that you find useful and insightful and, and, um, transferable to computer vision, but also just, I don't know, beautiful and profound that I think carries through to the vision domain?

Ishan Misra 33:18 I mean, the idea of masking has been very powerful. It has been used in vision as well for predicting, like you say, the next, uh, sort of, if you have N sort of frames, then you predict what going, what's going to happen in the next frame. So, that's been very powerful. In terms of modeling, like, in just terms, in terms of architecture, I think you had asked about transformers-

Lex Fridman 33:35 Mm-hmm. Yes.

Ishan Misra 33:36 ... a while back. That has really become, like, it has become super exciting for computer vision now. Like, in the past, I would say year and a half, it's become really powerful.

Lex Fridman 33:44 What's a transformer?

Ishan Misra 33:45 Right. I mean, the core part of a transformer is something called the self-attention model, so it came out of Google, and the idea basically is that if you have N elements, what you're creating is a way for all of these N elements to talk to each other. So, the idea basically is that you are paying attention, each element is paying attention to each of the other element, and basically by doing this, it's really trying to figure out, uh, you're basically getting a much better view of the data. So, for example, if you have a sentence of, like, four words, the point is if you get a representation or a feature for this entire sentence, it's constructed in a way such that each word has paid attention to everything else. Now, the reason it's, like, different from, say, what you would do in a convnet is basically that in a convnet, you would only pay attention to a local window. So, each word would only pay attention to its next neighbor or, like, one neighbor after that, and the same thing goes for images. In images, you would basically pay attention to pixels in a three across three or a seven across seven neighborhood, and that's it. Whereas with a transformer, the, that self-attention mainly, the sort of idea is that each element needs to pay attention to each other element.

Lex Fridman 34:50 And when you say attention, maybe another way to phrase that is you're considering a s- a context, a wide context-

Ishan Misra 34:59 Right.

Lex Fridman 34:59 ... in terms of the wide context of the sentence in understanding the meaning of a particular word, and in computer vision, that's understanding a larger context to understand the, the local pattern of a, uh, of a particular local part of an image.

Ishan Misra 35:13 Right. So, basically, if you have, say, again, a banana in the image, you're looking at the full image first, so whether it's like, you know, you're looking at all the pixels that are of a kitchen, of a dining table, and so on, and then you're basically looking at the banana also.

Lex Fridman 35:26 Yeah. By the way, in terms of if we were to train the funny classifier, there's something funny about the word banana.

Ishan Misra 35:31 (laughs)

Lex Fridman 35:32 Just wanted to (laughs) anticipate that. My, my-

Ishan Misra 35:34 I am wearing a banana shirt, so yeah. (laughs)

Lex Fridman 35:36 Oh, is it bananas on it? Oh. (laughs)

Ishan Misra 35:39 (laughs)

Lex Fridman 35:39 Okay, so masking has worked for the vision context as well.

Ishan Misra 35:42 And so this transformer idea has worked as well. So, basically, looking at all the elements to understand a particular element has been really powerful in vision. The reason is, like a lot of things, when you're looking at them in isolation, so if you look at just a blob of pixels, so Antonio Torralba at MIT used to have this, like, really famous image, which I looked at when I was a PhD student, where he would basically have a blob of pixels and he would ask you, "Hey, what is this?" And it looked basically like a shoe or, like, it could look like a TV remote, it could look like anything, and it turns out it was a beer bottle.... or I'm not sure. It was one of these three things-

Lex Fridman 36:14 Yeah.

Ishan Misra 36:14 ... but basically he showed you the full picture and then it was very obvious what it was. But the point is by, just by looking at that particular local window, you couldn't figure it out-

Lex Fridman 36:21 Yeah.

Ishan Misra 36:21 ... because of resolution, because of other things. It's just not easy always to just figure out by looking at just the neighborhood of pixels, what these pixels are.

Lex Fridman 36:29 Yeah.

Ishan Misra 36:29 And the same thing happens for language as well.

Lex Fridman 36:32 For the parameters that have to learn something about the data, you need to give it the capacity to learn the essential things. Like if it's not actually able to receive the signal at all, then it's not gonna be able to learn that signal. And to, in order to understand images, to understand language, you have to be able to see words in their full context. Okay.

Lex Fridman 36:52 What is harder to solve, vision or language? Visual intelligence or linguistic intelligence?

Ishan Misra 36:57 So, I'm going to say computer vision is harder. My reason for this is basically that, uh, language of course has a big structure to it because we developed it, uh, whereas vision is something that is common in a lot of animals. Everyone is able to get by... a lot of these animals on Earth are actually able to get by without language.

Lex Fridman 37:14 Mm-hmm.

Ishan Misra 37:15 And we... a lot of these animals we also deem to be intelligent. So, clearly intelligence, uh, does have like a visual component to it. And yes, of course in the case of humans it of course also has a linguistic component, but it means that there is something far more fundamental about vision than there is about language. And I'm sorry to anyone who disagrees, but yes, this is what I feel. (laughs)

Lex Fridman 37:34 So, that's being a little bit reflected in, um, the challenges that have to do with the, the progress of self-supervised learning, would you say? Or, is that just a peculiar accidents of the progress of the AI community that we focused on lang- or we discovered self-attention and transformers in the context of language first?

Ishan Misra 37:53 So, like, the self-supervised learning success was actually, uh, for vision has not much to do with the transformers part. I would say it's actually been independent-

Lex Fridman 38:01 Mm-hmm.

Ishan Misra 38:01 ... a little bit. I think it's just that the signal was a little bit different for, uh, vision than there was for like NLP and probably NLP ac- yeah, yeah, folks discovered it before. So for vision, the main success has basically been this like crops so far-

Lex Fridman 38:14 Mm-hmm.

Ishan Misra 38:14 ... like taking different crops of images. Uh, whereas for NLP it was this masking thing.

Lex Fridman 38:19 But also the level of success is still much higher for language.

Ishan Misra 38:22 Yes, it has. Uh, so that has a lot to do with... I mean, I can get into a lot of details.

Lex Fridman 38:26 Let's go.

Ishan Misra 38:26 For this particular question, let's go for it. Okay. So, the first thing is language is very structured, so you are going to produce a distribution over a finite vocabulary.

Lex Fridman 38:35 Mm-hmm.

Ishan Misra 38:36 English has a finite number of words. It's actually not that large. Uh, and you'll need to produce basically f- when you're doing this masking thing, all you need to do is basically tell me which one of these like 50,000 words it is.

Lex Fridman 38:46 Yeah.

Ishan Misra 38:46 That's it. Now for vision, let's imagine doing the same thing, okay? We're basically going to blank out a particular part of the image and we ask the network, or this neural network, to predict what is present in this missing patch. It's combinatorially large, right? You have 256 pixel values. If you're even producing basically a seven cross seven or a 14 cross 14 like window of pixels, at each of these 169 or each of these 49 locations, you have 256 values to predict.

Lex Fridman 39:13 Yeah.

Ishan Misra 39:13 And so it's really, really large. And very quickly, the kind of like, uh, prediction problems that we are setting up are going to be extremely like intractable for us. And so the thing is, for NLP, it has been really successful because we are very good at predicting, like doing this like distribution over a finite set. And the problem is when this set becomes really large, we're, we're going to become really, really bad at making these predictions and at solving basically this particular set of problems.

Lex Fridman 39:40 Mm-hmm.

Ishan Misra 39:41 So, if you were to do it exactly in the s- uh, same way as NLP for vision, there is very limited success. The way stuff is working right now is actually not by predicting these masks. It's basically by saying that you take these two like crops from the image, you get a feature representation from it, and just saying that these two features... so they're like vectors, just saying that the distance between these vectors should be small. And so it's a very different way of learning, uh, from the visual signal than there is from NLP. Okay, the other reason is the distributional hypothesis that we talked about for NLP, right? So, a word given its context, basic, the context actually supplies a lot of meaning to the word.

Lex Fridman 40:17 Mm-hmm.

Ishan Misra 40:18 Now, because there are just finite number, finite number of words and there is a finite way in like which we compose them, of course, uh, the same thing holds for pixels, but in language there's a lot of structure, right? So I always say whatever, "The dash jumped over the fence," for example. There are lots of these sentences that you'll get, and from this you can actually look at this particular sentence might occur in a lot of different contexts as well. This exact same sentence might occur in a different context. So, "The sheep jumped over the fence." "The cat jumped over the fence." "The dog jumped over the fence." So you immediately get a lot of these words which are... because this particular token itself has so much meaning, you get a lot of these tokens or these words which are actually going to have a, have sort of this related meaning across, given this context. Whereas for vision it's much harder because just by like pure, like the way we capture images, lighting can be different. Um, there might be like different noise in the sensor. So the thing is you're capturing a physical phenomenon and then you're basically g- going through a very complicated pipeline-

Lex Fridman 41:15 Mm-hmm.

Ishan Misra 41:15 ... of like image processing and then you're translating that into some kind of like digital signal.

Lex Fridman 41:19 Mm-hmm.

Ishan Misra 41:20 Whereas with language, you write it down and you transfer it to a digital signal almost like it's a lossless like transfer.

Lex Fridman 41:27 Yeah.

Ishan Misra 41:27 And each of these tokens are very, very well defined.

Lex Fridman 41:30 There could be a little bit of an argument there because language as written down is a projection of thought. This is one of the open questions is if you perfectly can solve language, are you getting close to being able to solve, you know, easily with flying colors past the touring test kind of thing?

Ishan Misra 41:52 Right.

Lex Fridman 41:52 So that's... and it's similar, uh, but different than, uh, the computer vision problem is in the 2D plane is a projection with three dimensional world. So, perhaps there's similar, uh, uh, similar problems there. Maybe this is good-

Ishan Misra 42:06 I mean-

Lex Fridman 42:06 Yeah.

Ishan Misra 42:06 ... I think what I'm saying is NLP is not easy, of course, don't get me wrong. Like, abstract thought expressed in knowledge, uh, or knowledge basically expressed in language is really hard to understand, right? I mean, we've been communicating with language for so long and it's... it is of course a very complicated concept. The thing is, um, at least getting like some- somewhat reasonable, um, like being able to solve some kind of reasonable tasks with language-... I would say slightly easier than it is with computer vision.

Lex Fridman 42:33 Yeah, I would say, yeah. So, that, that's well put. I would say getting impressive performance on language is, uh, easier. There, I feel like for both language and computer vision, there's going to be this wall of, like, that you, like a, this hump you have to overcome to achieve superhuman level performance, or human level performance. And I feel like for language, it, that wall is farther away. So, you can get pretty nice... You can, you can do a lot of tricks. You can show really impressive performance. You can even fool people that you're tweeting, or you write blog posts, writing, or your question answering, uh, is, has intelligence behind it. But to truly demonstrate understanding of dialogue, uh, of continuous long-form dialogue, that would require perhaps big breakthroughs. In the same way, way in computer vision, I think the big breakthroughs need to happen earlier to p- to achieve, uh, impressive performance.

Ishan Misra 43:36 Right.

Lex Fridman 43:36 This might be a good place to, uh, you already mentioned it, but what is contrastive learning and what are energy-based models?

Ishan Misra 43:43 Contrastive learning is sort of the, a paradigm of learning where the idea is that you are learning this embedding space, or so you're learning this sort of vector space of all your concepts, and the way you learn that is basically by contrasting. So, the idea is that you have a sample, you have another sample that's related to it, so that's, uh, called a positive, and you have another sample that's not related to it, so that's negative. So, for example, let's just take an NLP or one, and a simple example in, uh, computer vision. So, you have an image of a cat, you have an image of a dog, and for whatever application that you're doing, say you're trying to figure out what, uh, pets are, you're saying that these two images are related. So, image of a cat and dog are related, but now you have another third image of a banana, uh, because you don't like that word.

Lex Fridman 44:27 Thank you.

Ishan Misra 44:27 (laughs) So now you basically have this banana-

Lex Fridman 44:28 Thank you for speaking to the crowd.

Ishan Misra 44:30 And so you take both of these images and you take the image from the cat, the image from the dog, you get a feature from both of them.

Lex Fridman 44:36 Mm-hmm.

Ishan Misra 44:36 And now what you're training the network to do is basically, uh, pull both of these features together while pushing them away from the feature of a banana.

Lex Fridman 44:44 Mm-hmm.

Ishan Misra 44:44 So this is the contrastive part. So, you're contrasting against the banana. So, there's always this notion of a negative and a positive.

Lex Fridman 44:50 Mm-hmm.

Ishan Misra 44:51 Now, energy-based models are, like, like one way that, uh, Yan sort of explains a lot of these methods. So, uh, Yan basically, I think a couple of years or more than that, like when I joined Facebook, uh, Yan used to keep mentioning this word, energy-based models. And of course, I had no idea what he was talking about.

Lex Fridman 45:07 Yeah.

Ishan Misra 45:07 So then one day, I caught him in one of the conference rooms and I'm like, "Can you please tell me what this is?" So then, like very patiently, he sat down with, like a marker and a whiteboard. And his idea basically is that rather than talking about probability distributions, you can talk about energies of models. So, models are trying to minimize certain energies in certain space, or they're trying to maximize a certain kind of, uh, energy. And the idea basically is that you can explain a lot of the contrastive models, GANs, for example, which are like generative adversarial networks. Uh, a lot of these modern learning methods, or VAEs, which are variational autoencoders, you can really explain them very nicely in terms of an energy function that they're trying to minimize or maximize. And so by putting this common sort of language for all of these models, what looks very different in machine learning that, oh, VAEs are very different from what GANs are-

Lex Fridman 45:54 Mm-hmm.

Ishan Misra 45:54 ... or very, very different from what contrastive models are, you actually get a sense of like, oh, these are actually very, very related.

Lex Fridman 45:59 Mm-hmm.

Ishan Misra 46:00 It's just that the v- way or the mechanism in which they're, uh, sort of maximizing or minimizing this energy function is slightly different.

Lex Fridman 46:07 So, revealing the, the, the commonalities between all these approaches-

Ishan Misra 46:10 Right.

Lex Fridman 46:10 ... and putting a sexy word on top of it, like energy. And so similarities, so- two things that are similar have low energy, like the low energies signifying similarity.

Ishan Misra 46:20 Right. Exactly. So, basically the idea is that if you were to imagine, like the embedding as a manifold, a 2D manifold, you would get a hill or like a high sort of peak in the energy manifold wherever two things are not related, and basically you would have like a dip where two things are, are related. So, you'd get a dip in the ƒ.

Lex Fridman 46:37 And, uh, in the self-supervised context, how do you know two things are related and two things are not related?

Ishan Misra 46:44 Right. So, this is where all the sort of ingenuity or tricks comes in, right? So, for example, like, uh, you can take the fill-in-the-blank problem or you can take in, like the context problem and you, what you can say is two words that are in the same context are related, two words that are in different contexts are not related. For images, basically two crops from the same image are related and, whereas a third image is not related at all. Or for a video, it can be two frames from that video are related because they're likely to c- contain the same sort of concepts in them. Whereas a third frame from a different video is not related. So, it basically is, it's a very general term. Contrastive learning has nothing really to do with self-supervised learning. It actually is very popular in, for u- for example, like any kind of metric learning or any kind of embedding learning. So, it's also used in supervised learning. It's als-... And the thing is because we are not really using labels to get these positive or negative pairs, uh, it can basically also be used for self-supervised learning.

Lex Fridman 47:37 So, you mentioned one of the ideas in the vision context to, uh, that works is to have different crops. So, you could think of that as a way to sort of, uh, manipulating the data-

Ishan Misra 47:49 Right.

Lex Fridman 47:49 ... to generate, uh, examples that are similar. Obviously, uh, there's a bunch of other techniques. You mentioned lighting as a very, you know, i- in images, lighting is something that varies a lot, and you can artificially, uh, change those kinds of things. There's the whole broad field of data augmentation which, uh, manipulates images in order to increase arbitrarily the size of the dataset. First of all, what is data augmentation? And second of all, what's the role of data augmentation in self-supervised learning and contrastive learning?

Ishan Misra 48:22 So, data augmentation is just a way, like you said, it's basically a way to augment the data. So, you have say, N samples, and what you do is you basically define some kind of transforms for the sample. So, you take your, say, image, and then you define a transform where you can just increase, say, the colors, like the colors or the brightness of the image, or increase or decrease the contrast of the image, for example.

Lex Fridman 48:41 Mm-hmm.

Ishan Misra 48:42 Or take different crops of it. Uh, so data augmentation is just a process to, like, basically perturb the data, or, like, augment the data, right? And so, it has played a fundamental role for computer vision, uh, for self-supervised learning especially. The way most of the current methods work, contrastive or otherwise, is by taking an image, uh, in the case of images, uh, is by taking an image and then computing basically two perturbations of it. So, these can be two different crops of the image, uh, with, like, different types of lighting or different contrasts or different colors, so you j- jitter the colors a little bit, and so on. And now, the idea is basically because it's the same object or because it's, like, related concepts in both of these perturbations, you want the features from both of- both of these perturbations to be similar. So, now you can use a variety of different ways to enforce this constraint, like these features being similar. You can do this by contrastive learning. So, basically both of these things are positives. A third sort of image is negative. You can do this basically by, like, clustering, for example. You can say that both of these images should, uh, the features from both of these images should belong in the same cluster because they're related, whereas image ... like, another image should belong to a different cluster. So, there's a variety of different ways to basically enforce this particular constraint.

Lex Fridman 49:57 By the way, when you say features, it means there's a very large neural network that extracting patterns from the image and the kind of patterns it extracts should be v- ... either identical or very similar.

Ishan Misra 50:08 Right.

Lex Fridman 50:08 That's what that means.

Ishan Misra 50:09 Right. So, the neural network basically takes in the image and then outputs a- a set of, like, basically a vector of, like-

Lex Fridman 50:15 Mm-hmm.

Ishan Misra 50:16 ... numbers, and that's the feature.

Lex Fridman 50:17 Right.

Ishan Misra 50:17 And you want this feature for both of these, like, different crops that your computed to be similar.

Lex Fridman 50:22 Mm-hmm.

Ishan Misra 50:22 So, you want this vector to be identical, uh, in its, like, entries, for example.

Lex Fridman 50:26 Be, like, literally close in-

Ishan Misra 50:28 Yeah.

Lex Fridman 50:28 ... this multi-dimensional space to each other.

Ishan Misra 50:30 Space. Right.

Lex Fridman 50:31 And like you said, close can mean part of the same cluster or something like that-

Ishan Misra 50:36 Right.

Lex Fridman 50:36 ... in the- in this large space. First of all, that ... I wonder if there is connection to the way humans learn to this, almost like maybe subconsciously in order to understand a thing, you kind of have to see it from two, three, multiple angles. I wonder. There's a l- ... I have a lot of friends who are neuroscientists maybe or ... and, and cognitive scientists. I wo- I wonder if that's in there somewhere. Like, in order for us to place a concept in its proper place, we have to basically crop it in (laughs) all kinds of ways, uh, do basic data augmentation on it in whatever very clever ways that the brain likes to do.

Ishan Misra 51:17 Right.

Lex Fridman 51:18 Um, like spin it around in our minds somehow. That, that is very effective.

Ishan Misra 51:23 So, I think for some of them we, like, need to do it. So, like, babies for example, pick up objects, like move them, put them close to ƒ and whatnot.

Lex Fridman 51:29 Yeah.

Ishan Misra 51:30 But for certain other things actually, we are good at imagining it as well, right?

Lex Fridman 51:33 Yes.

Ishan Misra 51:33 So, if you ... I have never seen, for example, an elephant from the top. I've never basically looked at it from, like, top down.

Lex Fridman 51:39 Yeah.

Ishan Misra 51:39 But if you showed me a picture of it, I could very well tell you that it- that's an elephant. Um, so I think some of it we're just, like, we naturally build it or transfer it from other objects that we've seen to i- imagine what it's going to look like.

Lex Fridman 51:50 Has anyone done that with, uh, augmentation? Like, (laughs) imagine all the possible things

Lex Fridman 51:57 that are occluded or not there, but not just, like, normal things, like wild things, but they're nevertheless physically consistent.

Ishan Misra 52:07 So, s- ... I mean, people do kind of, like, o- occlusion-based augmentation as well.

Lex Fridman 52:11 Yeah.

Ishan Misra 52:11 So, you place in, like, a random, like, box, gray box to sort of mask out a certain part of the image, and the thing is basically you're kind of occluding it. For example, you place it, say, on half of a person's face, so basically saying that, you know, something below their nose is occluded-

Lex Fridman 52:26 Mm-hmm.

Ishan Misra 52:26 ... 'cause it's grayed out.

Lex Fridman 52:27 No.

Ishan Misra 52:28 Uh, so this is kind of-

Lex Fridman 52:29 No. I meant, like, you have, like, what is it? A table and you can't see behind the table and, and you imagine there's a bunch of, uh, elves with bananas behind the table.

Ishan Misra 52:38 (laughs)

Lex Fridman 52:38 Like, I wonder if there's useful to have, uh, a wild imagination for the network, because that's possible. Well, maybe not elves, but, like, puppies and kittens or something like that. Just have a wild imagination and, like, constantly be generating that wild imagination 'cause in, in terms of data augmentation as currently applied, it's super, ultra, very boring. It's very basic data augmentation. I wonder if, I wonder if there's a benefit to being wildly imaginable while trying to be, uh, consistent with physical reality.

Ishan Misra 53:11 I think it's a kind of a chicken and egg problem, right? Because to have, like, amazing data augmentation, you need to understand what the scene is.

Lex Fridman 53:17 Right.

Ishan Misra 53:18 And what we're trying to do data augmentation to learn (laughs) what a scene is anyway.

Lex Fridman 53:22 Just-

Ishan Misra 53:22 So, it's basically-

Lex Fridman 53:22 (laughs)

Ishan Misra 53:22 ... just keeps going on and on.

Lex Fridman 53:23 Before you understand it, just put elves with bananas until, until you know it's not to be true. (laughs)

Ishan Misra 53:28 (laughs)

Lex Fridman 53:29 (laughs) Just like children have a wild imagination until the adults ruin it all. Okay. So, what are the different kinds of data augmentation that you've seen to be effective in, uh, visual intelligence?

Ishan Misra 53:40 For, like, vision, it's a lot of these image filtering operations, so, like, blurring the image, uh, you know, all the kind of Instagram filters that you can think of.

Lex Fridman 53:49 (laughs)

Ishan Misra 53:49 (laughs) So, like, arbitrarily, like, make the red super red, make the green super greens, like saturate the image, uh-

Lex Fridman 53:55 Rotation, cropping ƒ-

Ishan Misra 53:57 Rotation, cropping, exactly. All of these kind of things. Uh-

Lex Fridman 53:59 Like you said, light- lighting is a really interesting one-

Ishan Misra 54:01 Yes.

Lex Fridman 54:01 ... to me. Like, that feels, like, really complicated to do.

Ishan Misra 54:04 So, I mean, they don't ... The augmentations that we work on aren't, like, that involved 'cause they're not going to be, like, physically realistic versions of lighting. It's not a ... We are assuming that there's a light source up and then you're moving it to the right and then what does the thing look like.

Lex Fridman 54:16 Yeah.

Ishan Misra 54:17 It's really more about, like, brightness of the image, overall brightness of the image or overall contrast of the image and so on.

Lex Fridman 54:22 But this is a really important-... point to me. I always thought that data augmentation holds an important key, um, to big improvements in machine learning, and it seems that it is an important aspect of, uh, self-supervised learning. So, I wonder if there's big improvements to be achieved and much more intelligent kinds of data augmentation. For example, currently, maybe you can correct me if I'm wrong, data augmentation is not parameterized.

Ishan Misra 54:52 Yeah.

Lex Fridman 54:53 You're not learning.

Ishan Misra 54:54 You're not learning.

Lex Fridman 54:55 To me, it seems like data augmentation potentially should involve more learning than the learning process itself.

Ishan Misra 55:04 Right.

Lex Fridman 55:05 Um, you're almost like thinking of like generative kind of ... It's the elves with bananas. You're trying to ... It's like very active imagination of messing with the world and teaching that mechanism for messing with the world to be, uh, realistic.

Ishan Misra 55:19 Right.

Lex Fridman 55:19 Uh, because that feels like ... I mean, it's imagination. It's j- just as you said, if things... It feels like us humans are able to, um, maybe sometimes subconsciously imagine before we see the thing, imagine what we're expecting to see, like maybe several options.

Ishan Misra 55:37 Right.

Lex Fridman 55:37 And especially, we probably forgot, but when we were younger, probably the possibilities were wilder and more numerous. And then as we get older, we become to understand the world and, uh, our, the possibilities of what we might see becomes less and less and less. So, I, I wonder if, if you think there's a lot of breakthroughs yet to be had in data augmentation? And maybe also can you just comment on the stuff we have, is that a big part of self-supervised learning?

Ishan Misra 56:02 Yes. So, data augmentation is like key to self-supervised learning. That is like the kind of augmentations that we're using, and basically this, the fact that we are trying to learn these neural networks that are predicting these features from images that are robust under data augmentation-

Lex Fridman 56:16 Mm-hmm.

Ishan Misra 56:16 ... uh, has been the key for visual self-supervised learning, and they play, play a fairly fundamental role to it. Now, the irony of all of this is that for like deep learning purists will say the entire point of deep learning is that you feed in the pixels to the network, neural network, and it should figure out the patterns on its own. So, if it really wants to look at edges, it should look at edges. You shouldn't really like really go and handcraft these like features, right?

Lex Fridman 56:38 Mm-hmm.

Ishan Misra 56:38 You shouldn't go tell it that look at edges. So, data augmentation should basically be in the same category, right? Why should we tell the network or tell this like entire learning paradigm what kind of data augmentation that we are looking for? We are encoding a very sort of human-specific bias there, that we know things are ... Like if you change the contrast of the image, it should still be an apple-

Lex Fridman 57:00 Mm-hmm.

Ishan Misra 57:00 ... or it should still see apple, not banana.

Lex Fridman 57:02 Thank you.

Ishan Misra 57:02 And, uh, (laughs) basically if we change like colors, uh, it should still be the same kind of concept.

Lex Fridman 57:07 Yes.

Ishan Misra 57:08 Of course, this is not ... One, this is, doesn't feel like super satisfactory because a lot of our human knowledge or our human supervision is actually going into the data augmentation.

Lex Fridman 57:17 Mm-hmm.

Ishan Misra 57:17 So, although we are calling it self-supervised learning, a lot of the human knowledge is actually being encoded in the data augmentation process. So, it's really like we've kind of sneaked away the supervision at the input, and we are like really designing these nice list of data augmentations that are working very well.

Lex Fridman 57:31 Of course, the idea is that it's much easier to design a list of data augmentation than it is to do ... so humans are doing, nevertheless doing less and less work-

Ishan Misra 57:39 Right.

Lex Fridman 57:39 ... and maybe leveraging their creativity more and more. And when we say data augmentation is not parameterized, it means it's not part of the learning process. Do you think it's possible to integrate some of the data augmentation into the learning process?

Ishan Misra 57:53 I think so. I think so. And in, in fact, it will be really beneficial for us because, uh, a lot of these data augmentation that we use in vision are very extreme. For example, like when you have certain concepts, (laughs) again, a banana. (laughs) You take the banana and then basically you change the color of the banana, right?

Lex Fridman 58:10 Yeah.

Ishan Misra 58:10 So, you make it a purple banana. Now, this data augmentation process is actually independent of the, like ... It has no, uh, notion of what is present in the image, so it can change this color arbitrarily. It can make it a red banana as well. And now, what we are doing is we are telling the neural network that this red banana and, uh, uh, so a crop of this image which has the red banana and a crop of this image where I change the color to a purple banana should be, the features should be the same.

Lex Fridman 58:33 Mm-hmm.

Ishan Misra 58:34 Now, bananas aren't red or purple, mostly. So, really, the data augmentation process should take into account what is present in the image-

Lex Fridman 58:40 Mm-hmm.

Ishan Misra 58:41 ... and what are the kinds of physical realities that are possible. It shouldn't be completely independent of the image.

Lex Fridman 58:45 So, you might get big gains if you, instead of being drastic, do subtle augmentation, but realistic augmentation?

Ishan Misra 58:53 Right. Realistic. I'm not sure if it's subtle, but like realistic for sure.

Lex Fridman 58:56 If it's realistic, then even subtle augmentation will give you big benefits.

Ishan Misra 59:00 Exactly, yeah. And it will be ... Like for particular domains, you might actually see like if ... For example, now we are doing medical imaging, there are going to be certain kinds of like geometric augmentations which are not really going to be very valid for the human body.

Lex Fridman 59:14 Mm-hmm.

Ishan Misra 59:15 So, if you were to like actually loop in data augmentation into the learning process, it will actually be much more useful.

Lex Fridman 59:21 Yeah.

Ishan Misra 59:21 Now, this actually does take us to maybe a semi-supervised kind of a setting because you do want to understand what is it that you're trying to solve. So, currently, self-supervised learning kind of operates in the wild, right? So, you do the self-supervised learning, you re- and the purists and all of us basically say that, "Okay, this should learn useful representations, and they should be useful for any kind of end task, no matter it's like banana recognition or like autonomous driving." Uh, it's a tall order. Maybe the first baby step for us should be that, okay, if we are trying to loop in this data augmentation into the learning process, then we at least need to have some sense of what we are trying to do. Are we trying to distinguish between different types of bananas, or are we trying to distinguish between banana and apple, or are we trying to do all of these things at once? And so some notion of like what happens at the end might actually help us do much better at,

Ishan Misra 1:00:10 at this side.

Lex Fridman 1:00:10 Let me ask you, uh, a ridiculous question. If I were to give you, like a black box, like a choice to have an arbitrary large dataset of real natural data versus really good data augmentation algorithms, which would you like to train in a self-supervised way on?... so natural data from the internet, ar- arbitrarily large, so unlimited data, or it's like more controlled, good data augmentation on the finite dataset.

Ishan Misra 1:00:43 The thing is like because our learning algorithms for vision right now really rely on data augmentation, even if you were to give me like an infinite source of like image data, I still need a good data augmentation algorithm to learn from it.

Lex Fridman 1:00:54 You need something that tells you that two things are similar.

Ishan Misra 1:00:57 Right. And so something... because you've given me an arbitrarily large dataset, I still need to use data augmentation to take that image, construct like these two perturbations of it, and then learn from it. So, the thing is our learning paradigm is very primitive right now.

Lex Fridman 1:01:11 Yeah.

Ishan Misra 1:01:11 Even if you were to give me lots of images, it's still not really useful. A good data augmentation algorithm is actually going to be more useful. So, you can like reduce down the amount of da- data that you give me by like 10 times, but if you were to give me a good data augmentation algorithm, that would probably do better than giving me like 10 times the size of that data but me having to rely on like a very primitive data augmentation algorithm.

Lex Fridman 1:01:32 Like through tagging and all those kinds of things, is there a way to discover things that are semantically similar on the internet? Obviously there is, but they might be extremely noisy.

Ishan Misra 1:01:42 Right.

Lex Fridman 1:01:42 And the difference might be farther away than you would be comfortable with.

Ishan Misra 1:01:47 So, I mean, yes, tagging will help you a lot. It'll actually go a very long way in figuring out what images are related or not. Um, and then, so, but then the purists would argue that when you're using human tags because these tags are like supervision, is it really, really self-supervised learning now?

Lex Fridman 1:02:04 Yeah.

Ishan Misra 1:02:04 Because you're using human tags to figure out which images are like similar. Perhaps-

Lex Fridman 1:02:08 Hashtag no filter means a lot of things.

Ishan Misra 1:02:10 Yes.

Lex Fridman 1:02:11 (laughs)

Ishan Misra 1:02:11 I mean, there are certain tags which are going to be applicable pretty much to anything.

Lex Fridman 1:02:14 Yeah. (laughs)

Ishan Misra 1:02:15 (laughs) So they're pretty useless for learning.

Lex Fridman 1:02:17 Yeah.

Ishan Misra 1:02:18 Uh, but I mean, f- certain tags are actually like the eye filter, for example, or the Taj Mahal, for example. These tags are like very indicative of what's going on.

Lex Fridman 1:02:26 Mm-hmm.

Ishan Misra 1:02:26 And they are... I mean, they are human supervision.

Lex Fridman 1:02:29 Yeah. Well, this is one of the task of discovering from human-generated data strong signals that could be leveraged, uh, for self-supervision. Like humans are doing so much work already. Like many years ago, there was something that was called I guess human computation back in the day. Humans are doing so much work. It's- it'd be exciting to discover ways to leverage the work they're doing to teach machines without any extra effort from them. An example could be, like we said, driving, humans driving and machines can learn from their driving. I always hope that there could be some supervision signal discovered in video games because there's so many people-

Ishan Misra 1:03:09 (laughs)

Lex Fridman 1:03:09 ... that play video games that it feels like so much effort is put into video games, uh, into playing video games, and you can design video games somewhat cheaply-

Ishan Misra 1:03:21 Right.

Lex Fridman 1:03:22 ... and, and to, to include whatever signals you want. It feels like, uh, that could be leveraged somehow.

Ishan Misra 1:03:27 So, people are using that.

Lex Fridman 1:03:28 Yeah.

Ishan Misra 1:03:28 Like there are actually folks right here in UT Austin, like Philip Granbull is a professor at UT Austin. Uh, he's been like working on video games, uh, as a source of supervision. (laughs)

Lex Fridman 1:03:37 Yeah.

Ishan Misra 1:03:38 I mean, it's, it's really fun like as a PhD student getting to basically-

Lex Fridman 1:03:41 (laughs)

Ishan Misra 1:03:41 ... play video games all day.

Lex Fridman 1:03:42 Yeah. But so I do hope it- that kind of thing scales and like ultimately it boils down to discovering some undeniably very good signal. It's like masking in NLP.

Ishan Misra 1:03:53 Right.

Lex Fridman 1:03:54 But that said, there's non-contrastive methods.

Ishan Misra 1:03:57 Right.

Lex Fridman 1:03:57 What do non-contrastive energy-based self-supervised learning methods look like? And why are they promising?

Ishan Misra 1:04:05 So, like I said about contrastive learning, you have this notion of a positive and a negative. Now, the thing is this entire learning paradigm really requires access to a lot of negatives, uh, to learn a good sort of feature space. The idea is if I tell you, um, okay, so a cat and a dog are similar and they're very different from a banana. The thing is this is a fairly simple analogy, right? Because, well, bananas look visually very different from what cats and dogs do. So very quickly, if this is the only source of supervision that I'm giving you, your learning is not going to be like... after a point, the neural network is really not going to learn a lot, uh, because the negative that you're getting is going to be so random. So, it can be, oh, a cat and a dog are very- are similar, but they're very different from a Volkswagen Beetle. Now, like this car looks very different from these animals again. So, the thing is in contrastive learning, the quality of the negative sample really matters a lot. And so what has happened is basically that typically these methods that are contrastive really require access to lots of negatives, which becomes harder and harder to sort of scale when designing a learning algorithm. So, that's w- been one of the reasons why non-contrastive methods have become like popular and why people think that they're going to be more useful. So, a non-contrastive method, for example, like clustering is one non-contrastive method. The idea basically being that you have two of these, uh, two of these, uh, samples, and so the cat and dog are two crops of this image, they belong to the same cluster. Uh, and so essentially you're basically doing clustering online when you're lea- learning this network, and which is very different from having access to a lot of negatives explicitly.

Lex Fridman 1:05:38 Mm-hmm.

Ishan Misra 1:05:39 The other way which has become really popular is something called self-distillation. So, the idea basically is that you have a teacher network and a student network, and the teacher network produces a feature, so it takes in the image and it... basically the neural network figures out the patterns, gets the feature out. And there's another, uh, neural network which is the student neural network, and that also produces a feature. And now all you're doing is basically saying that the, uh, features produced by the teacher network and the student network should be very similar. That's it. There is no notion of a, a negative anymore.

Lex Fridman 1:06:08 Mm-hmm.

Ishan Misra 1:06:09 And that's it. So, it's all about similarity maximization between these two features. And so all I need to now do is figure out how to have these two sorts of parallel networks, a student network and a teacher network. And, uh, basically researchers have figured out very cheap methods to do this. So, you can actually have for free really two types of neural networks. Uh, they're kind of related, but there's different enough that you can actually basically have a learning problem set up.

Lex Fridman 1:06:34 So, you can ensure that they always remain different enough?So, the thing doesn't collapse into something boring.

Ishan Misra 1:06:41 Exactly. So, the main sort of enemy of self-supervised learning a- any kind of similarity maximization technique is collapse. And so, collapse means that you learn the same feature representation for all the images in the world.

Lex Fridman 1:06:53 (laughs)

Ishan Misra 1:06:53 Which is completely useless. It-

Lex Fridman 1:06:54 Everything is a banana.

Ishan Misra 1:06:55 Everything is a banana, everything is a cat, everything is a car.

Lex Fridman 1:06:58 Yeah.

Ishan Misra 1:06:58 Uh, and so all we need to do is basically come up with ways to prevent collapse, contrastive learning is one way of doing it. And then, for example, like clustering or self-distillation are other ways of doing it. We also had a recent paper where we used like decorrelation, um, between like two sets of features to prevent collapse. So, that's inspired a little bit by like Horace Barlow's neuroscience principles.

Lex Fridman 1:07:20 By the way, I should comment that whoever counts the number of times the n- the word banana, apple, cat, and dog were used in this conversation wins the internet. I wish you luck.

Ishan Misra 1:07:31 (laughs)

Lex Fridman 1:07:32 What, uh, what is SUAVE and- and the main improvement proposed in, uh, the paper, Unsupervised Learning of Visual Features by Contrasting Cluster Assignments?

Ishan Misra 1:07:43 SUAVE basically is a clustering base technique, uh, which is for, again, the same thing, for self-supervised learning in vision where we have two crops. And the idea basically is that you want the features from these two crops of an image to lie in the same cluster. Uh, and basically, uh, crops that are coming from different images to be in different clusters.

Lex Fridman 1:08:04 Okay.

Ishan Misra 1:08:04 Now, typically, you know, sort of if you were to do this clustering, you would perform clustering offline. What that means is, you would, if you have a data set of N examples, you would run over all of these N e- N examples, get features for them, perform clustering, so basically get some clusters, and then repeat the process again.

Lex Fridman 1:08:21 Mm-hmm.

Ishan Misra 1:08:22 So, this is offline basically because I need to do one pass through the data to compute its clusters.

Lex Fridman 1:08:26 Mm-hmm.

Ishan Misra 1:08:27 SUAVE is basically just a simple way of doing this online. So, as you're going through the data, you're actually computing these clusters online.

Lex Fridman 1:08:34 Mm-hmm.

Ishan Misra 1:08:34 And so, of course, there is like a lot of tricks involved in how to do this in a robust manner without collapsing, but this is the s- sort of key idea to it.

Lex Fridman 1:08:42 Is there a nice way to say what is the key methodology of the clustering that enables that?

Ishan Misra 1:08:47 Right. So, the idea basically is that, um, when you have N samples, we assume that we- we have access to like, there are always K clusters in a data set. K is a fixed number. So, for example, K is 3,000. And so if you have any n- when you look at any sort of small number of examples, all of them must belong to one of these K clusters, and we impose this equipartition constraint. What this means is, that, uh, basically, uh, your entire set of N samples should be equally partitioned into K clusters. So, all your K clusters are basically equal, will have equal contribution to these N samples. And this ensures that we never collapse. So, collapse can be viewed as a way in which all samples belong to one cluster, right?

Lex Fridman 1:09:30 Mm-hmm.

Ishan Misra 1:09:30 So, all this, if all features become the same, then you have basically just one mega cluster, you don't even have like 10 clusters or 3,000 clusters. So, SUAVE basically ensures that at each point, all these 3,000 clusters are being used in the clustering process, and that's it. Basically just, uh, figure out how to do this online, and-

Lex Fridman 1:09:48 Mm-hmm.

Ishan Misra 1:09:49 ... um, again, basically just make sure two crops from the same image belong to the same cluster, uh, and others don't.

Lex Fridman 1:09:55 And the fact they have a fixed K makes things simpler.

Ishan Misra 1:09:58 Fixed K makes things simpler. Our clustering is not like really hard clustering, it's soft clustering.

Lex Fridman 1:10:03 Mm-hmm.

Ishan Misra 1:10:03 So, basically you can be point two to cluster number one and point eight to cluster number two, so it's not really hard. Uh, so essentially even though we have like 3,000 clusters, we can actually represent a lot

Ishan Misra 1:10:14 of clusters.

Lex Fridman 1:10:15 What is SEER? S-E-E-R. And, uh, what are the key results and insights in the paper Self-Supervised Pre-training of Visual Features in the Wild? What is this big, beautiful SEER system?

Ishan Misra 1:10:30 SEER... So, I'll first go to SUAVE because SUAVE is actually like, uh, one of the key components for SEER.

Lex Fridman 1:10:35 Mm-hmm.

Ishan Misra 1:10:35 So, SUAVE was, when we used SUAVE, it was demonstrated on ImageNet.

Lex Fridman 1:10:39 Mm-hmm.

Ishan Misra 1:10:39 So, typically, like, uh, self-supervised methods, uh, the way we sort of operate is, uh, like in the research community, we kind of cheat. So, we take ImageNet, which of course I talked about as having lots of labels, and then we throw away the labels, like throw away all the hard work that went behind basically the labeling process, and we pretend that it is self- like unsupervised. But the problem here is that we have when, like when we collected these images, uh, the ImageNet data set has a particular distribution of concepts, right?

Lex Fridman 1:11:09 Mm-hmm.

Ishan Misra 1:11:10 So, these images are very curated. And what that means is, these images, uh, of course belong to a certain set of known concepts. And also, ImageNet has this bias that all images contain an object which is like very big and it's typically in the center.

Lex Fridman 1:11:23 Mm-hmm.

Ishan Misra 1:11:24 So, when you're talking about a dog, it's a well-framed dog, it's towards the center of the image. So, a lot of the data augmentation, a lot of the sort of hidden assumptions in self-supervised learning, uh, actually really, uh, exploit this bias of ImageNet.

Lex Fridman 1:11:36 Mm-hmm.

Ishan Misra 1:11:37 And so, I mean, a lot of my work, a lot of work from other people always uses ImageNet sort of as the benchmark to show s- the success of self-supervised learning.

Lex Fridman 1:11:45 So, you're implying that there's particular limitations to this kind of data set?

Ishan Misra 1:11:49 Yes. I mean, it's basically because our data augmentations that we designed, uh, like in the h- like all, uh, data augmentation that we designed for self-supervised learning and vision are kind of overfed to ImageNet.

Lex Fridman 1:11:59 But yet you're saying a little bit hardcoded like the cropping.

Ishan Misra 1:12:03 Exactly. The cropping parameters, the kind of lighting that we're using, the kind of blurring that we're using.

Lex Fridman 1:12:08 Yeah. But you would, uh, for more in the wild data set, you would need to be, uh, uh, clever or more careful in setting the range of parameters and those kinds of things.

Ishan Misra 1:12:19 Right. So, for SEER, our main goal was two-fold. One, basically to move away from ImageNet for training. Uh, so the images that we used were like uncurated images. Now, there's a lot of debate whether they're actually curated or not, but I'll talk about that later. Uh, but the idea was basically these are going to be random internet images, uh, that we are not going to filter out based on like ca- particular categories.

Lex Fridman 1:12:39 Mm-hmm.

Ishan Misra 1:12:40 So, we did not say that, "Oh, images that belong to dogs and cats should be the only images that come in this data set."... banana. Uh, and basically, other images should be thrown out, so we didn't d- and do any of that. So, these are random internet images and of course, uh, it also goes back to like, the problem of scale that you talked about.

Lex Fridman 1:12:57 Mm-hmm.

Ishan Misra 1:12:57 So, these were basically about a billion or so images and for context, ImageNet, uh, the ImageNet version that we used was one million images earlier, so this is basically going like three orders of magnitude more. The idea was basically to see if we can train a very large convolutional model in a self-supervised way on this uncurated, but really large set of images.

Lex Fridman 1:13:16 Mm-hmm.

Ishan Misra 1:13:16 And how well would this model do? So, is self-supervised learning really overfit to ImageNet? Uh, or, or can it actually work in the wild? And it was also out of curiosity, what kind of things will this model learn? Will it actually be able to still figure out, you know, different types of objects and so on? Would there be particular kinds of tasks it would actually, uh, do better than, uh, an ImageNet, uh, trained model? And so for SEAR, one of our main findings was that we can actually train very large models in a completely self-supervised way on lots of internet images without really necessarily filtering them out. Which was in itself a good thing, because it's a fairly simple process, right? So, you get images which are uploaded, and you basically can immediately use them to train a model in an unsupervised way. You don't really need to sit and filter them out. These images can be cartoons, these can be memes, these can be actual pictures uploaded by people, and you don't really care about what these images are. You don't even care about what concepts they contain. So, this was a very sort of simple setup.

Lex Fridman 1:14:10 What image selection mechanism would you say is there, like, um, inherent in some aspect of the process? So, you're kind of implying that there's almost none, but, uh, what, what is there would you say if you were to introspect?

Ishan Misra 1:14:25 So... Right. So, it's not like uncurated can basically like... One way of imagining uncurated is basically you have