Cristos Goodrow: YouTube Algorithm | Lex Fridman Podcast #68

1. 0:00 – 3:26

   Introduction

   1. LFLex Fridman0:00

      The following is a conversation with Christos Goudreau, Vice President of Engineering at Google and head of search and discovery at YouTube, also known as the YouTube algorithm. YouTube has approximately 1.9 billion users, and every day people watch over one billion hours of YouTube video. It is the second most popular search engine behind Google itself. For many people, it is not only a source of entertainment, but also how we learn new ideas from math and physics videos, to podcasts, to debates, opinions, ideas from out of the box thinkers and activists on some of the most tense, challenging and impactful topics in the world today. YouTube and other content platforms receive criticism from both viewers and creators, as they should, because the engineering task before them is hard and they don't always succeed. And the impact of their work is truly world changing. To me, YouTube has been an incredible wellspring of knowledge. I've watched hundreds if not thousands of lectures that changed the way I see many fundamental ideas in math, science, engineering and philosophy. But it does put a mirror to ourselves, and keeps the responsibility of the steps we take in each of our online educational journeys into the hands of each of us. The YouTube algorithm has an important role in that journey of helping us find new exciting ideas to learn about. That's a difficult and an exciting problem for an artificial intelligence system. As I've said in lectures and other forums, recommendation systems will be one of the most impactful areas of AI in the 21st century, and YouTube is one of the biggest recommendation systems in the world. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, give it five stars on Apple Podcasts, follow on Spotify, support it on Patreon, or simply connect with me on Twitter @lexfridman, spelled F-R-I-D-M-A-N. I recently started doing ads at the end of the introduction. I'll do one or two minutes after introducing the episode, and never any ads in the middle that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. I personally use Cash App to send money to friends, but you can also use it to buy, sell and deposit Bitcoin in just seconds. Cash App also has a new investing feature. You can buy fractions of a stock, say $1 worth, no matter what the stock price is. Broker services are provided by Cash App Investing, a subsidiary of Square and member SIPC. I'm excited to be working with Cash App to support one of my favorite organizations called FIRST, best known for their FIRST robotics and Lego competitions. They educate and inspire hundreds of thousands of students in over 110 countries, and have a perfect rating at Charity Navigator, which means the donated money is used to maximum effectiveness. When you get Cash App from the App Store or Google Play, and use code LEXPODCAST, you'll get $10, and Cash App will also donate $10 to FIRST, which again is an organization that I've personally seen inspire girls and boys to dream of engineering a better world. And now, here's my conversation with Christos Goudreau.

2. 3:26 – 7:30

   Life-long trajectory through YouTube

   1. LFLex Fridman3:26

      YouTube is the world's second most popular search engine, behind Google of course. We watch more than one billion hours of YouTube videos a day, more than Netflix and Facebook Video combined. YouTube creators upload over 500,000 hours of video every day. The average lifespan of a human being, just for comparison, is about 700,000 hours. So what's uploaded every single day is just enough for a human to watch in a lifetime. So let me ask an absurd philosophical question. If from birth, when I was born, and there's many people born today with the internet, I watched YouTube videos non-stop, do you think their trajectories through YouTube video space that can maximize my average happiness or maybe education or my growth as a human being?

   2. CGCristos Goodrow4:19

      I think there are some great trajectories through YouTube, uh, videos, but I wouldn't recommend that anyone spend all of their waking hours or all of their hours watching YouTube. I mean, I think about the fact that YouTube has been really great for my kids, for instance. Uh, my oldest daughter, uh, you know, she's been watching YouTube for several years. She watches Tyler Oakley and the Vlog Brothers, and I know that it's had a very profound and positive impact on her character. And my younger daughter, she's a ballerina, and her teachers tell her that YouTube is a huge advantage for her, because she can practice a routine and watch, like, professional dancers do that same routine, and stop it and back it up and rewind and all that stuff, right? So it's been really good for them, and then even my son is a sophomore in college. He, um, he got through his linear algebra class because of a channel called 3Blue1Brown.

   3. LFLex Fridman5:19

      Mm-hmm.

   4. CGCristos Goodrow5:20

      Which, you know, helps you understand linear algebra, but in a way that would be very hard for anyone to do on a white board or a chalkboard. And so I think that those experiences, from my point of view, were very good, and, and so I can imagine really good trajectories through YouTube, yes.

   5. LFLex Fridman5:38

      Have you looked at or do you think of broadly about that trajectory over a period? 'Cause YouTube is grown up now, so over a period of years you just kind of gave a few anecdotal examples. But, you know, I used to watch certain shows on YouTube. I don't anymore, I've moved on to other shows. And ultimately you want people to, uh, from YouTube's perspective, to stay on YouTube, to grow as human beings on YouTube.So you have to think not just what makes them engage in- in today, or this month, but also over a period of years. Do you-

   6. CGCristos Goodrow6:12

      Absolutely, that's right. I mean, if YouTube is going to continue to enrich people's lives, then, um, you know, then it has to grow with them. And, uh, and people's interests change over time. And so, I think we've, we've been working on this problem, and I'll just say it broadly as like how to intro- introduce diversity and introduce people who are watching one thing to something else they might like. We've been working on that problem all the eight years I've been at YouTube. Um, it's a hard problem, because, uh, I mean, of course it's trivial to introduce diversity that doesn't help.

   7. LFLex Fridman6:52

      Yeah, it's just-

   8. CGCristos Goodrow6:53

      Right? I could-

   9. LFLex Fridman6:53

      ... add a random video.

   10. CGCristos Goodrow6:54

       I could just randomly select a video from the billions that we have. Uh, it's likely not to even be in your language. So... (laughs)

   11. LFLex Fridman7:02

       (laughs)

   12. CGCristos Goodrow7:03

       Uh, the likelihood that you would watch it and develop a new interest is very, very low. And so, uh, what you want to do when you're trying to increase diversity is find something that is not too similar to the things that you've watched, but also something that you might be likely to watch. And that balance, finding that spot between those two things, is quite challenging.

   13. LFLex Fridman7:29

       So,

3. 7:30 – 13:33

   Discovering new ideas on YouTube

   1. LFLex Fridman7:30

      so diversity of content, diversity of ideas, it's, uh, it's a really difficult... It's a thing, like, that's almost impossible to define, right? Like, what's different? So how, how do you think about that? So two examples is, um, I'm a huge fan of 3Blue1Brown, say, and then one diversity... Uh, I, I wasn't even aware of a channel called Veritasium, whi- which is a great science, physics, whatever channel. So one version of diversity is showing me Derek's Veritasium channel, which I was really excited to discover, actually, and now I watch a lot of his videos.

   2. CGCristos Goodrow8:07

      Okay, so you're a person who's watching some math channels, and you might be interested in some other science or math channels. So like you mentioned, the first kind of diversity is just show you some, some things from other channels that are related. Uh, but not just, you know, not all the, uh, 3Blue1Brown channel, throw in a couple others. So, so that's the, maybe the first kind of diversity that we started with many, many years ago. Um, taking a bigger leap is, uh, is about... I mean, the, the mechanisms we do, we use for that is, is we basically cluster videos and channels together, mostly videos. We do every, almost everything at the video level. And so we'll, we'll make some kind of a cluster via some embedding process, and then, um, and then measure, you know, what is the likelihood that a, that users who watch one cluster might also watch another cluster that's very distinct.

   3. LFLex Fridman9:06

      Mm-hmm.

   4. CGCristos Goodrow9:06

      So we may come to find that, um, that people who watch, uh, science videos also like, um, jazz.

   5. LFLex Fridman9:15

      Mm-hmm.

   6. CGCristos Goodrow9:15

      Uh, this is possible, right? And so, and so, um, because of that relationship that we've identified, um, through the, uh, measur- through the embeddings and then the measurement of the people who watch both, we might recommend a jazz video once in a while.

   7. LFLex Fridman9:31

      So there's this clustering in the embedding space of jazz videos and science videos, and so you kind of try to look at aggregate statistics where if a lot of people that jump from science cluster to the jazz cluster tend to remain as engaged or become more engaged, then that's, that means those two are, they should hop back and forth and they'll be, they'll be happy.

   8. CGCristos Goodrow9:57

      Right. There's a higher likelihood that a person from, who's watching science would like jazz than the person watching science would like, I don't know, backyard railroads or, or something else, right? And so we can try to measure these likelihoods and use that to make the best recommendation we can.

   9. LFLex Fridman10:16

      So, okay, so w- we'll talk about the machine learning of that, but I have to linger on things that neither you or anyone have an answer to. There's gray areas of truth, which is, for example, now I can't believe I'm going there, but, uh, politics. It, it... (laughs) It happens so that certain people believe certain things, and they're very certain about them. Let's move outside the red versus blue politics of today's world, but there's different ideologies. For example, in college, I, I read quite a lot of Ayn Rand. I studied, and that's a particular philosophical ideology I find, I found it interesting to explore. Okay, so that was that kind of space. I've kind of moved on from that cluster, uh, intellectually, but it nevertheless is an interesting cluster. There's, I was born in the Soviet Union. Socialism, communism is a certain kind of political ideology that's really interesting to explore. Again, objectively just, there's a set of beliefs about how the economy should work and so on. And so it's hard to know what's true or not in terms of people within those communities are often advocating that this is how we achieve utopia in this world, and they're pretty certain about it. So how do you try to manage politics in this chaotic, divisive world? Not political, any kind of ideas in terms of filtering what people should watch next and in terms of also not letting certain things be on YouTube. This is an exceptionally difficult responsibility. (laughs)

   10. CGCristos Goodrow11:49

       Right. Well, um, the responsibility to get this right is our top priority. And, um, and the first comes down to making sure that we have good, clear rules of the road, right? Like, just because we have freedom of speech doesn't mean that you can literally say anything, right? Like, we as a society have accepted certain, um, restrictions on our freedom of speech.... there are things like libel laws and, and things like that. And so, um, where we can draw a clear line, we do, and we continue to evolve that line over time. Um, however, as you pointed out, wherever you draw the line, there's gonna be a borderline. And in that borderline area, we are going to maybe not remove videos, but we will try to reduce the recommendations of them or the proliferation of them, um, by demoting them. And then alternatively, in those situations, try to raise what we would call authoritative or credible sources of information. So, we're not trying to w- w- I mean, you mentioned Ayn Rand and, um, communism. Uh, you know, those are, those are two, like, valid points of view that people are gonna debate and discuss. And, and of course, people who, uh, believe in one or the other of those things are gonna try to (laughs) persuade other people-

   11. LFLex Fridman13:14

       Right.

   12. CGCristos Goodrow13:14

       ... to their point of view. And so, um, we're not trying to settle that or choose a side or anything like that. What we're trying to do is make sure that the, the people who are expressing those point of view and, and, um, offering those positions are authoritative and credible.

4. 13:33 – 23:02

   Managing healthy conversation

   1. LFLex Fridman13:34

      So, let me ask (sighs) a question about people I don't like personally. You heard me. I don't care if you leave comments on this.

   2. CGCristos Goodrow13:42

      (laughs)

   3. LFLex Fridman13:42

      Uh, is, uh... And but they... Sometimes they're brilliantly funny, which is trolls. So, eh, (laughs) people who kind of mock... I mean, the internet is full, Reddit, of mock-style comedy, where people just kind of make fun of, um... Point out that the emperor has no clothes. And there's brilliant comedy in that, but sometimes it can get cruel and mean. So, on that, on the mean point... And sorry to linger on these things that have no good answers, but actually it's... I, I, I totally hear you that this is really important that you're trying to solve it. But how do you reduce the meanness of people on YouTube? (laughs)

   4. CGCristos Goodrow14:26

      (laughs) Um, I understand that anyone who uploads YouTube videos has to become resilient to a certain amount of meanness.

   5. LFLex Fridman14:35

      Mm-hmm.

   6. CGCristos Goodrow14:35

      Like, I've heard that from many creators. And, um, we would... We are trying in various ways, comment ranking, um, allowing certain features to block people to, to reduce or, or make that, that meanness or that trolling behavior, um, less effective on YouTube.

   7. LFLex Fridman14:56

      Yeah.

   8. CGCristos Goodrow14:57

      And so, uh, I mean, it's, it's very important, um, but it's something that we're, we're gonna keep having to work on and, and... You know, as, as we improve it, like, maybe we'll get to a point where, uh, where people don't have to suffer this sort of meanness when they upload YouTube videos. I hope we do, but, um, uh, you know, but it just does seem to be something that you have to be able to deal with as a YouTube creator nowadays.

   9. LFLex Fridman15:26

      Do, do you ever hope that... So, you mentioned two things that I kinda agree with is... So, there's like a machine learning approach of ranking comments based on whatever, uh, based on how much they contribute to the healthy conversation. Let's put it that way. And then the other is almost an interface question of how do you... How does the creator filter? So, block or... How, how does, how do humans themselves, the users of YouTube, manage their own conversation? Do you have hope that these two tools will create a better society without limiting freedom of speech too much? Without sort of... And I can't even, like, h- saying that. People are like, "What do you mean limiting f-"

   10. CGCristos Goodrow16:08

       (laughs)

   11. LFLex Fridman16:09

       Uh, sort of, uh, curating speech.

   12. CGCristos Goodrow16:12

       I mean, I think that that overall is our whole project here at YouTube.

   13. LFLex Fridman16:17

       Right.

   14. CGCristos Goodrow16:17

       Like-

   15. LFLex Fridman16:18

       Yeah.

   16. CGCristos Goodrow16:18

       ... we fundamentally believe, and I personally believe very much, that YouTube can be great. It's been great for my kids. I think it can be great for society. Um, but it's absolutely critical that we get this responsibility part right. And that's why it's our top priority. Susan Wojcicki, who's the CEO of YouTube, um, she says something that I personally find very inspiring, which is that we wanna do our jobs today in a manner so that people 20 and 30 years from now will look back and say, "You know, YouTube, they, they really figured this out. They really found a way to strike the right balance between the openness and the value that the openness has, and also making sure that we are meeting our responsibility to users in society."

   17. LFLex Fridman17:09

       So, the burden (laughs) on YouTube actually is quite incredible. And, uh, uh, the one thing that people don't, uh, don't give enough credit to the seriousness and the magnitude of the problem, I think. So, uh, I, I personally hope that you do solve it 'cause a lot is in your ha- uh, on... (laughs)

   18. CGCristos Goodrow17:25

       (laughs)

   19. LFLex Fridman17:26

       A lot is riding on your success or failure, so it's... Uh, besides, of course, running a successful company, you're also curating the content of the internet and the conversation on the internet. That's a, that's a powerful thing. So, o- one thing that people wonder about is how much of it can be solved with pure machine learning? So, looking at the data, studying the data, and creating algorithms that curate, uh, the comments, curate the content? And how much of it needs human intervention? Meaning, people here at YouTube in a room sitting and thinking about, "What is the nature of truth? (laughs) What is... Uh, what are the ideals that we should be promoting?" That kind of thing. Uh, so algorithm versus human input.

   20. CGCristos Goodrow18:18

       Ah.

   21. LFLex Fridman18:18

       What's your sense?

   22. CGCristos Goodrow18:20

       I mean, my own experience has demonstrated that you need both of those things. Um, algorithms, I mean, you're familiar with machine learning algorithms and the thing they need most is data, and the data is generated by humans. And so for instance, um, when we're building a system to try to figure out which are the videos that are misinformation or borderline policy violations, well, the first thing we need to do is get human beings to make decisions about which, which of those videos are in which category. And then we use that data and, and basically, you know, take that information that's, that's determined and governed by humans and, and extrapolate it or, or apply it, uh, to the entire set of billions of YouTube videos. And we couldn't, we, we, we couldn't get to all the videos on YouTube well without the humans, and we, we couldn't use the humans to get to all the (laughs) videos of YouTube. So there's no world in which you have only one or the other of these things. Um, and just as you said, uh, a lot of it comes down to, um, people at YouTube spending a lot of time trying to figure out what are the right policies, um, you know, what are the outcomes based on those policies? Are they the kinds of things we want to see? Uh, and then once we kind of get a, get an agreement or, or build some consensus around, around what the policies are, well, then we've got to find a way to implement those policies across all of YouTube. And that's where both the human beings, um, we call them evaluators or reviewers, come into play to help us with that. And then, and then once we get a lot of training data from them, then we apply the machine learning techniques to take it even further.

   23. LFLex Fridman20:17

       Do you have a sense that these human beings have a bias in some kind of direction? Sort of, um... I mean, that's an interesting question. We do sort of, in, in autonomous vehicles and computer vision in general, a lot of annotation, and we rarely ask what bias do the annotators have. You know, they're... it... even in the sense that they're better than- th- they're better at annotating certain things than others. For example, people are much better at fors- annotating segmentation at segmenting cars in a scene versus segmenting bushes or trees. (laughs) You know, there's specific mechanical reasons for that, but also because the sema- it's semantic gray area and, and just for a lot of reasons, people are just terrible at annotating trees. Okay, so in that same kind of sense, do you think of, in terms of people reviewing videos or annotating the content of videos, is there some kind of bias that you're aware of or seek out in that human input?

   24. CGCristos Goodrow21:24

       Well, we take steps to try to overcome these kinds of biases or biases that we think would be problematic. Um, so for instance, like we ask people to have a bias towards scientific consensus. That's something that we, we instruct them to do. Um, we ask them to have a bias towards, uh, demonstration of expertise or credibility or authoritativeness. Um, but there are other biases that we, that we want to make sure to try to remove, and there's many techniques for doing this. One of them is you, you send the same thing to be reviewed to many people. And so, um, you know, that's one technique. Another is that you make sure that the people that are doing these sorts of, uh, tasks are from different backgrounds and different areas of the United States or of the world. But then even with all of that, it's possible for certain kinds of, uh, what we would call, um, unfair biases to creep into machine learning systems, primarily, as you said, uh, because maybe the training data itself comes in-

   25. LFLex Fridman22:32

       Right.

   26. CGCristos Goodrow22:32

       ... in a, in a biased way. And so we also have, um, worked very hard on the ma- on improving the machine learning systems to remove and reduce unfair biases when it's, um, when it goes against or, or has- involves some protected class, for instance.

   27. LFLex Fridman22:51

       Thank you for exploring with me (laughs) some of the more challenging things. I'm sure there's a few more that we'll jump back to, but let me jump into the fun part, which is, um,

5. 23:02 – 38:00

   YouTube Algorithm

   1. LFLex Fridman23:02

      maybe the basics of the "YouTube algorithm." What does the YouTube algorithm look at to make recommendation for what to watch next? I mean, just from a machine learning perspective. Or when you search for a particular term, how does it know what to show you next? Because it seems to, at least for me, do an incredible job of both.

   2. CGCristos Goodrow23:25

      (laughs) Well, that's kind of you to say. It didn't used to do a very good job, (laughs) um, but it's gotten better over the years. Even, even I observed that it's improved quite a bit. Um, those are two different situations. Like when you search for something, uh, YouTube uses the best technology we can get from Google, um, to make sure that, that the YouTube search system finds what someone's looking for. And of course, the very first things that one thinks about is, okay, well, does the word occur in the title, for instance? Um, uh, you know, but there, but there are much more sophisticated things, um, where we're mostly trying to do some syntactic match or, or maybe a semantic match based on, uh, words that we can add, um, to the document itself. For instance, uh, you know, maybe is, is this video, uh, watched a lot after this query?

   3. LFLex Fridman24:23

      ... hmm.

   4. CGCristos Goodrow24:23

      Right? That's something that, uh, we can observe and then as a result, uh, make sure that that- that, uh, document would be retrieved for that query. Um, now when you talk about what kind of videos would be recommended to watch next, um, that's something, again, we've been working on for many years. And probably the first, um, the first real attempt to do that well was to use collaborative filtering. So you-

   5. LFLex Fridman24:57

      Can you describe what collaborative filtering is?

   6. CGCristos Goodrow24:59

      Sure. It's just, um... Basically, what we do is we observe which videos get watched close together by the same person. And if you observe that, and if you can imagine creating a graph where the videos that get watched close together by the most people are sort of very close to one another in this graph, and videos that don't frequently get watched close to- close together by the same person or- or the same people are far apart, then you end up with this, um, gr- graph that we call the related graph that basically represents videos that are very similar or related in some way. And what's amazing about that is that, uh, it puts all the videos that are in the same language together, for instance.

   7. LFLex Fridman25:49

      Hmm.

   8. CGCristos Goodrow25:50

      And we didn't even have to think about language.

   9. LFLex Fridman25:53

      Yeah.

   10. CGCristos Goodrow25:53

       It just does it.

   11. LFLex Fridman25:54

       Yeah.

   12. CGCristos Goodrow25:55

       Right? And it puts all the videos that are about sports together, and it puts most of the music videos together, and it puts all of these sorts of videos together, um, just because that's sort of the way the people using YouTube behave.

   13. LFLex Fridman26:08

       So that already cleans up a lot of the problem. It- it- it takes care of the lowest hanging fruit, which is- uh, uh, happens to be a huge one of just managing these millions of videos.

   14. CGCristos Goodrow26:20

       That's right. Um, I remember a few years ago, I was talking to someone who was, um, uh, trying to propose that we do a wo- a research project concerning people who, um, who are bilingual. And this person was, uh, making this proposal based on the idea that YouTube could not possibly be good at recommending videos well to people who are bilingual. And so, um, she was telling me, uh, about this and I said, "Well, can you give me an example of what problem do you think we have on YouTube with the recommendations?" And so she said, "Well, I'm a, um, a researcher in- in the US and- and when I'm looking for academic topics, I wanna look, I wanna see them in English." And so she searched for one, found a video, and then looked at the What's Next suggestions, and they were all in En- English.

   15. LFLex Fridman27:16

       Mm-hmm.

   16. CGCristos Goodrow27:16

       And so she said, "Oh, I see. YouTube must think that I speak only English." And so she said, "Now, I'm actually originally from Turkey. And sometimes when I'm cooking, let's say, I wanna make some baklava, I really like to watch videos that are in Turkish." And so she searched for a video about making the baklava and then- and then selected it, and it was in Turkish, and the What's Next recommendations were in Turkish.

   17. LFLex Fridman27:38

       Hmm.

   18. CGCristos Goodrow27:38

       And she just couldn't believe how this was possible (laughs) . And, and, "How is it that you know that I speak both these two languages?" And put all the videos together and it's just as a re- uh, uh, sort of an outcome of this related graph that's created through collaborative filtering.

   19. LFLex Fridman27:54

       So for me, one of my huge interests is just human psychology, right? And- a- and that's such a powerful platform on which to utilize human psychology to- to discover what people, individual people wanna watch next. But it's also be just fascinating to me... You know, I've, uh, Google Search has the ability to look at your own history. And I've done that before, just- just what I've searched, three years, for many, many years, and it's a fascinating picture of who I am actually. And, um, I don't think anyone's ever summarized, now, I personally would love that, a summary of who I am as a person on the internet to me-

   20. CGCristos Goodrow28:35

       (laughs)

   21. LFLex Fridman28:35

       ... because I think it reveals... I- I think it puts a mirror to me or to- to others, you know, that's actually quite revealing and interesting. You know, uh, just, uh, maybe the number of... It's- it's a joke, but not really, is the number of cat videos I've watched-

   22. CGCristos Goodrow28:53

       (laughs)

   23. LFLex Fridman28:53

       ... or videos of people falling, you know? It's stuff that's absurd, um, th- that kind of stuff, it's really interesting. And of course, it's really good from the machine learning aspect to, uh, to- to- to show, uh, to figure out what to show next, but it's interesting. Um, hey, have you, just as a tangent, played around with the idea of giving a map to people sort of as opposed to just using this information to show what's next, showing them, "Here are the clusters you've loved over the years," kind of thing?

   24. CGCristos Goodrow29:25

       Well, we do provide the history of all the videos that you've watched.

   25. LFLex Fridman29:29

       Yes.

   26. CGCristos Goodrow29:29

       So you can definitely search through that, and- and look through it and search through it to see what it is that you've been watching on YouTube. Uh, we have actually, in various times, um, experimented with this sort of cluster idea, finding ways to demonstrate or show people, um, what topics they've been interested in or what- what clusters they've watched from. It's interesting that you bring this up because, um, in some sense, the- the way the recommendation system of YouTube sees a user is exactly as the history of all the videos they've watched on YouTube. And so you can think of, um, yourself or- or- or any user on YouTube as kind of like a- a DNA strand-

   27. LFLex Fridman30:16

       (laughs) Strand, yeah.

   28. CGCristos Goodrow30:17

       ... of all your videos, right? Um, that sort of represents you. Uh, you can also think of it as maybe a vector in the space of all the videos on YouTube.And so, you know, now, once you think of it as a vector in the space of all the videos on YouTube, then you can start to say, "Okay, well, you know, which videos, which, which other vectors are close to me? And, uh, to my vector?" And, um, and that's one of the ways that we generate some diverse recommendations.

   29. LFLex Fridman30:45

       Yeah.

   30. CGCristos Goodrow30:45

       It's because you're like, "Okay, well, you know, these, these people seem to be close with respect to the videos they watch on YouTube, but, you know, here's a topic or a video that one of them has watched and enjoyed, but the other one hasn't." That could be an opportunity, uh, to make a good recommendation.
6. 38:00 – 44:38

   Analyzing the content of video itself

   1. LFLex Fridman38:00

      uh, using for the search and for the recommendation, you've mentioned titles and like meta data, like text data that people provide. Description and title, and maybe keywords. So maybe you can speak to the value of those things in search, and also this incredible, fascinating area of the content itself. So the video content itself, trying to understand what's happening in the video. So YouTube will release a data set that, you know, in the, in the machine learning, computer vision world, this is just an exciting space. How much is that currently, how much are you playing with that currently? How much is your hope for the future of being able to analyze the content of the video itself?

   2. CGCristos Goodrow38:42

      Well, we have been working on that also since I came to YouTube. So that's-

   3. LFLex Fridman38:46

      Analyzing the content of the video?

   4. CGCristos Goodrow38:47

      Analyzing the content of the video.

   5. LFLex Fridman38:48

      Wow. Awesome.

   6. CGCristos Goodrow38:48

      Right? Um, and, uh, what I can tell you is that, uh, our ability to do it well is still somewhat crude.

   7. LFLex Fridman38:57

      Hmm.

   8. CGCristos Goodrow38:57

      We can, we can tell if it's a music video. We can tell if it's a sports video. We can probably tell you that people are playing soccer. Um, we probably can't tell whether it's, uh, Manchester United or my daughter's soccer team. So these things are kind of difficult, and, and using them, we, we can use them in some ways. So for instance, we use that kind of information to understand and inform these clusters that I talked about. Uh, and also maybe to add some words, like soccer, for instance, to the video, if, if it doesn't occur in the title or the description. Which is remarkable that often it doesn't. Um...

   9. LFLex Fridman39:39

      Right.

   10. CGCristos Goodrow39:40

       I, one of the things that I ask, uh, creators to do is, is please help us out with the title and the description. Um, for instance, we were, um, a few years ago, having a livestream of some competition for World of Warcraft on YouTube. And, um, it was a very important competition. But if you typed World of Warcraft in search, you wouldn't find it.

   11. LFLex Fridman40:05

       World of Warcraft wasn't in the title?

   12. CGCristos Goodrow40:07

       World of Warcraft wasn't in the title.

   13. LFLex Fridman40:09

       Oh. Okay.

   14. CGCristos Goodrow40:09

       It was match 478, you know, A team versus B team, and World of Warcraft wasn't in the title.

   15. LFLex Fridman40:15

       Yeah.

   16. CGCristos Goodrow40:15

       (laughs) I'm just like, "Come on, give me, give me a..."

   17. LFLex Fridman40:16

       But being literal, being literal on the internet is actually s- very uncool. Which is the problem. (laughs)

   18. CGCristos Goodrow40:22

       Oh, is that right? (laughs)

   19. LFLex Fridman40:23

       Well, I mean, uh, i- in some sense, well, some of the greatest videos, I mean, there's a humor to just being indirect, being witty and so on, and actually being, um, you know, machine learning algorithms want you to be, you know, literal, right? You, you just want to say what's in the thing, be very, very simple. And in, in some sense, that gets away from wit and humor, so you have to play with both, right? So, but you're saying that for now, sort of, um, the content of the title, the content of the description, the actual text, is, is one of the best ways to, uh, for the, for the algorithm to find your video and put him in the right cluster.

   20. CGCristos Goodrow41:03

       That's right. And, and I would go further and say that, uh, if you want people, human beings, to select your video in search, then it helps to have, let's say, World of Warcraft in the title. Because why would a person s- you know, if they're looking at a bunch, they type World of Warcraft, and they have a bunch of videos, all of whom say World of Warcraft, except the one that you uploaded-

   21. LFLex Fridman41:24

       Mm-hmm.

   22. CGCristos Goodrow41:25

       ... well, even the person is gonna think, "Well, maybe this isn't, somehow search made a mistake. This isn't really about World of Warcraft." So it's important not just for the machine learning systems, but also for the people who might be looking for this sort of thing. They get a, a clue that it's what they're looking for by seeing that same thing prominently in the title of the video.

   23. LFLex Fridman41:46

       Okay, let me push back on that. So I think from the algorithm perspective, yes, but if they typed in World of Warcraft, and saw a video that, with the title simply, "Winning!" and, and, and the thumbnail has like a sad, um, orc or something. I don't know.

   24. CGCristos Goodrow42:03

       (laughs)

   25. LFLex Fridman42:04

       Right? Like, I think that's much, it's gr- it er- it gets your curiosity up. And then if they could trust that the algorithm was smart enough to figure out somehow that this is indeed a World of Warcraft video, that would have created the most beautiful experience. I, I think in terms of just the wit and the humor and the curiosity that we human beings naturally have. But you're saying, I mean, realistically speaking, it's really hard for the algorithm to figure out that the content of that video will be a World of Warcraft video.

   26. CGCristos Goodrow42:34

       And you have to accept that some people are gonna skip it.

   27. LFLex Fridman42:37

       Yeah. It-

   28. CGCristos Goodrow42:38

       Right? I mean, and so, you're right. Uh, the people who don't skip it and select it are gonna be delighted.

   29. LFLex Fridman42:45

       Yeah.

   30. CGCristos Goodrow42:45

       Um, but other people might say-
7. 44:38 – 47:50

   Clickbait thumbnails and titles

   1. LFLex Fridman44:38

      speaking to that, there's a lot of people that, i- in a positive way perhaps, I don't know, I, I don't like it, but like to game... Want to try to game the system to get more attention. Everybody, creators in a positive sense, want to get attention, right? So how do you, how do you work in this space when people create more and more, um, sort of clickbait-y titles and thumbnails? Sort of, uh, Veritasium Derek has made a video where basically describes that it seems what works is to create a high quality video, really good video, what people would want to watch and wants to click on it, but have clickbait-y titles and thumbnails to get them to click on it in the first place. And he's saying, "I'm embracing this fact and I'm just gonna keep doing it, and I hope you forgive me for doing it."

   2. CGCristos Goodrow45:25

      (laughs)

   3. LFLex Fridman45:26

      "And you will enjoy my videos once you click on them." So in what sense do you see this kind of clickbait style attempt to manipulate... To, to, to get people in the door, to manipulate the algorithm or play with the algorithm or game the algorithm?

   4. CGCristos Goodrow45:43

      I think that, that you can look at it as an attempt to game the algorithm. But, um, even if you were to take the algorithm out of it and just say, "Okay, well all these videos happen to be lined up," which the algorithm didn't make any decision about which one to put at the top or the bottom, but they're all lined up there. Which one are the people gonna choose? And, and I'll tell you the same thing that I told Derek is, um, you know, I have a bookshelf and they have two kinds of books on them. Uh, science books.

   5. LFLex Fridman46:12

      Mm-hmm.

   6. CGCristos Goodrow46:12

      Um, I have my math books from when I was a student, and they all look identical except for the titles on the covers.

   7. LFLex Fridman46:21

      Mm.

   8. CGCristos Goodrow46:21

      They're all yellow. They're all from Springer, and they're... Every single one of them, the cover is totally the same.

   9. LFLex Fridman46:27

      Yes. (laughs)

   10. CGCristos Goodrow46:28

       Right?

   11. LFLex Fridman46:28

       Yeah.

   12. CGCristos Goodrow46:29

       On the other hand, I have other more pop science type books, and they all have very interesting covers, right?

   13. LFLex Fridman46:35

       Yeah.

   14. CGCristos Goodrow46:35

       And, and they have provocative, uh, titles and things like that. I mean, I wouldn't say that they're clickbait-y because (laughs) they are indeed good books.

   15. LFLex Fridman46:44

       Yeah.

   16. CGCristos Goodrow46:44

       Um, and I don't think that they cross any line, but, uh, but, you know, the- that's just a decision you have to make, right? Like, the people who, who write... Classical Recursion Theory by Piero di Freddi-

   17. LFLex Fridman46:57

       Oh.

   18. CGCristos Goodrow46:57

       He was fine with the yellow title and the (laughs) and, and nothing more. Whereas, I think other people who, who wrote a more popular type book, um, understand that they need to have a compelling cover and a compelling title, and, uh, and, you know, I, I don't think there's anything really wrong with that. We, we do, we do take steps to make sure that there is a line that you don't cross. And if you go too far, maybe your thumbnail's especially racy or, or, um, you know, it's all caps with too many exclamation points-

   19. LFLex Fridman47:33

       (laughs)

   20. CGCristos Goodrow47:34

       ... we observe that, um, users are kind of, uh, you know, sometimes offended by that.

   21. LFLex Fridman47:41

       Mm-hmm.

   22. CGCristos Goodrow47:41

       And so, um, so for the users who are offended by that, we will then depress or suppress those videos.

8. 47:50 – 50:14

   Feeling like I'm helping the YouTube algorithm get smarter

   1. CGCristos Goodrow47:51

   2. LFLex Fridman47:51

      And which reminds me, there's also another signal where users can say... I, I don't know if it was recently added, but I really enjoy it. Just saying, I don't, I didn't re- something like, "I, I don't want to see this video anymore." Or something like-

   3. CGCristos Goodrow48:03

      (laughs)

   4. LFLex Fridman48:04

      (laughs) Like this is, uh... Like there's certain videos that just cut me the wrong way. Like, just, just jump out at me. It's like, "I don't wanna, I don't want this." And it feels really good to clean that out.

   5. CGCristos Goodrow48:14

      (laughs) .

   6. LFLex Fridman48:14

      (laughs) To be like, "I don't... That's not, that's not for me." I don't know. I, I think that might have been recently added, but that's, that's also a really strong signal.

   7. CGCristos Goodrow48:22

      Yes, absolutely. Right. We don't want to make a recommendation that, uh, people are unhappy with.

   8. LFLex Fridman48:29

      And that makes me... That particular one makes me feel good as a user in general, a- and as a machine learning person, 'cause I feel like I'm helping the algorithm. My interaction on YouTube don't always feel like I'm helping the algorithm. Like, I'm not reminded of that fact. Uh, like for example-... uh, Tesla and Autopilot and Elon Musk create a feeling, uh, for their customers, for people that own Teslas, that they're helping the algorithm of Tesla vehicles.

   9. CGCristos Goodrow48:53

      Right.

   10. LFLex Fridman48:54

       Like, they're all, like are really proud they're helping-

   11. CGCristos Goodrow48:56

       Right.

   12. LFLex Fridman48:56

       ... the fleet learn. I think YouTube doesn't always remind people that you're helping the algorithm get smarter. And for me, I, I love that idea. Like, we're all collaboratively ... Like, Wikipedia gives that sense through altogether creating a, a beautiful thing. YouTube is, uh, doesn't always remind me of that. It's, uh, this conversation is reminding me of that, but, uh-

   13. CGCristos Goodrow49:18

       (laughs) Well, that's a good tip. We should keep that fact in mind when we design these features. I, I'm not sure I, I really thought about it that way, but that's a very interesting perspective.

   14. LFLex Fridman49:27

       It's an interesting question of personalization that I feel like when I click like on a video, I'm just improving my experience. It would be great ... It would make me personally, people are different, but make me feel great if I was helping also the YouTube algorithm broadly say something. You know what I'm saying? Like, there's a, that ... I don't know if that's human nature, but you want the products you love, and I certainly love YouTube. Like, you want to help it get smarter and smarter and smarter 'cause there's some hunk kind of coupling between our lives together-

   15. CGCristos Goodrow50:03

       (laughs)

   16. LFLex Fridman50:03

       ... (laughs) being better. If, if YouTube was better, then I will, my life will be better, and there's that kind of reasoning. I'm not sure what that is, and I'm not sure how many people share that feeling. That could be just a machine learning feeling.

9. 50:14 – 51:44

   Personalization

   1. LFLex Fridman50:14

      But n- on that point, how much personalization is there in terms of next video recommendations? So, is it kind of all really boiling down to clustering? Like, you find near as clusters to me and so on and that-

   2. CGCristos Goodrow50:31

      Um-

   3. LFLex Fridman50:31

      ... that kind of thing or is it-

   4. CGCristos Goodrow50:33

      (clears throat)

   5. LFLex Fridman50:33

      ... how much is personalized to me, the individual, completely?

   6. CGCristos Goodrow50:36

      It's very, very personalized. So, um, your experience will be quite a bit different from anybody else's who's watching that same video, uh, at least when they're logged in. And, um, the reason is, is that we, we found that, that users often want two different kinds of things when they're watching a video. Sometimes, they wanna keep watching more on that topic or more in that genre, and other times, they just are done and they're ready to move on to something else. And so the question is, well, what is the something else? And one of the first things one can imagine is, well, maybe something else is, um, the latest video from some channel to which you've subscribed, and that's gonna be very different from, for you than it is for me, right? And, and even if it's not something that you subscribe to, it's something that you watch a lot. And again, that'll be very different on a person by person basis. And so, um, even the Watch Next, as well as the homepage, of course, is quite personalized.

   7. LFLex Fridman51:43

      So

10. 51:44 – 54:32

    What does success look like for the algorithm?

    1. LFLex Fridman51:44

       what ... W- we mentioned some of the signals, but what does success look like? Wh- what does success look like in terms of the algorithm creating a great longterm experience for a user? Or, put another way, if you look at the videos I've watched this month, how do you know the algorithm succeeded for me?

    2. CGCristos Goodrow52:03

       I think, first of all, if you come back and watch more YouTube, then that's one indication that you found some value from it.

    3. LFLex Fridman52:11

       So, just the number of hours is a powerful indicator?

    4. CGCristos Goodrow52:13

       Well, I mean, not the hours themselves, but, um, uh, the fact that you return on another day.

    5. LFLex Fridman52:21

       Hm.

    6. CGCristos Goodrow52:21

       Um, so that's probably the most simple indicator. Uh, people don't come back to things that they don't find value in, right? There's a lot of other things that they could do. Um, but like I said, I mean, ideally, we would like everybody to feel that YouTube enriches their lives and that every video they watched is the best one they've ever watched since they've started watching YouTube. And so that's why, uh, we survey them and ask them, like, "Is this one to five stars?" And so our version of success is, uh, every time someone takes that survey, they say it's five stars. And if we ask them, "Is this the best video you've ever seen on YouTube?" They say yes-

    7. LFLex Fridman53:04

       (laughs)

    8. CGCristos Goodrow53:05

       ... every single time. So, um, it's hard to imagine that we would actually achieve that. Maybe asymptotically (laughs) we would get there, but, uh, but that would be what we think success is.

    9. LFLex Fridman53:16

       It's funny, I've recently said somewhere, I don't know, maybe tweeted, but, uh, that, uh, Ray Dalio has this video on the ec- the economic machine. I forget what it's called, but it's a 30-minute video, and I said it's the, the greatest video I've ever watched on YouTube. It's, it's like I watched the whole thing, and my mind was blown. It's a very crisp, clean des- description of how the, at least the American economic system works. It's a beautiful video, and I was just ... I wanted to click on something that say this is the best thing.

    10. CGCristos Goodrow53:46

        (laughs)

    11. LFLex Fridman53:47

        This is the best thing ever. Please let me ... I c- can't believe I discovered it. Uh, I mean, th- the views and the likes reflect its quality, but I was almost upset that I haven't found it earlier and wanted to find other things like it. I don't think I've ever felt that this is the best video I've ever watched (laughs) that was that. And, uh, to me, the ultimate utopia, the best experience, is where every single video ... where I don't see any of the videos I regret, and every single video I watch is one that actually helps me grow, helps me enjoy life, be happy and so on. Um, well ... (laughs) S- so that's, that's, that's a heck of a ... That's a, that's one of the most beautiful and ambitious, I think, machine learning tasks.

11. 54:32 – 57:24

    Effect of YouTube on society

    1. LFLex Fridman54:32

       So, when you look at a society as opposed to an individual user, do you think of how YouTube is changing society when you have these millions of people watching videos, growing, learning, changing, having debates? Do, do you have a sense of-... yeah, what the big impact on society is? 'Cause I think it's huge, but do you have a sense of what direction (laughs) we're taking this world?

    2. CGCristos Goodrow54:56

       Well, I mean, I think, you know, openness has had an impact on society already. Uh, there's a lot of, um-

    3. LFLex Fridman55:04

       What do you mean by openness?

    4. CGCristos Goodrow55:05

       Well, the fact that, um, uh, unlike other mediums, there's not someone sitting at YouTube who decides before you can upload your video, whether it's worth having you upload it.

    5. LFLex Fridman55:18

       Right.

    6. CGCristos Goodrow55:19

       Um, or- or worth anybody seeing it really, right? Um, and so, uh, you know, there are some creators who say, like, "I- I wouldn't have this opportunity to- to reach an audience." Uh, Tyler Oakley often said that, you know, he wouldn't have had this opportunity to reach this audience if it weren't for YouTube. Um, and, uh, and so I think that's one way in which YouTube has changed society. Um, I know that there are people that I work with from, uh, outside the United States, especially in, from places where, uh, literacy is low.

    7. LFLex Fridman55:59

       Mm-hmm.

    8. CGCristos Goodrow56:00

       And they think that YouTube can help in those places because you don't need to be able to read and write in order to learn something important for your life. Maybe, um, you know, how to do some job or how to fix something. Uh, and so that's another way in which I think YouTube is possibly changing society. So, I've- I've worked at YouTube for eight, almost nine years now, and it's fun because I meet people and, you know, you tell them where they, where you work, and you say you work on YouTube, and they immediately say, "I love YouTube."

    9. LFLex Fridman56:35

       Yeah.

    10. CGCristos Goodrow56:36

        Right? Which is great. Makes me feel great. Uh, but then, of course, when I ask them, "Well, what is it that you love about YouTube?" Not one time ever has anybody said that the search work's outstanding or that the recommendations are great. Um, what they always say when I ask them, "What do you love about YouTube?" Is they immediately start talking about some channel or some creator or some topic or some community that they found on YouTube and that they just love.

    11. LFLex Fridman57:06

        Yeah.

    12. CGCristos Goodrow57:07

        And so that has made me realize that YouTube is really about the video and connecting the people with the videos, and then everything else kind of gets out of the way.

    13. LFLex Fridman57:22

        So beyond the video,

12. 57:24 – 59:33

    Creators

    1. LFLex Fridman57:24

       it's interesting 'cause you kind of mentioned creator. What about the connection with just the individual creators as opposed to just individual videos? So, like I gave the example of a Ray Dalio video, that the video itself is incredible, but there's some people who are just creators that, uh, I love that they're... One of the cool things about people who call themselves YouTubers or whatever is they have a journey. They usually, almost all of them are hor- they suck horribly in the beginning.

    2. CGCristos Goodrow57:57

       (laughs)

    3. LFLex Fridman57:57

       And then they kind of grow, you know? And then there's that genuineness in their growth. So h- you know, uh, YouTube clearly wants to help creators connect with their audience in this kind of way. So, how do you think about that process of helping creators grow, helping them connect with their audience, develop not just individual videos, but the entirety of a creator's life on YouTube?

    4. CGCristos Goodrow58:18

       Well, I mean, we're trying to help creators find the biggest audience that they can find. And the reason why that's... You, you brought up creator versus video.

    5. LFLex Fridman58:29

       Yeah.

    6. CGCristos Goodrow58:29

       The reason why creator channel is so important is because, um, if we have a hope of- of people coming back to YouTube, well, they have to have in their minds some sense of what they're going to find when they come back to YouTube. If YouTube were just the next viral video and I have no concept of what the next viral video could be... One time it's a cat playing a piano, and the next day it's, uh, uh, some children interrupting a reporter, and the next day it's, you know-

    7. LFLex Fridman59:03

       (laughs)

    8. CGCristos Goodrow59:03

       ... uh, some other thing happening. Um, then- then it's hard for me to- to, when I'm not watching YouTube say, "Gosh, I- I really, you know, would like to see something from someone or about something." Right? And so that's why I think this connection between fans and creators is so important, um, uh, for both because it's- it's a way of, uh, of sort of fostering a relationship that can play out into the future.

13. 59:33 – 1:03:27

    Burnout

    1. CGCristos Goodrow59:33

    2. LFLex Fridman59:33

       Let me talk about kind of a- a dark and interesting question in general. And again, a topic that you or nobody has an answer to. But social media has a sense of... You know, it gives us highs and it gives us lows in the sense that... So, creators often speak about having sort of burn- burnout and having psychological ups and downs and challenges mentally in terms of continuing the creation process. There's a momentum. There's a huge excited audience that makes everybody feel, that makes creators feel great. And I- I think it's more than just financial. I think it's literally just they love that sense of community. It's part of the reason I upload to YouTube. I don't care about money. Never will. What I care about is the- the community. But some people feel like this momentum, and even when there's times in their life when they don't feel... You know, for some reason don't feel like creating. So, how do you think about burnout, this mental exhaustion that some YouTube creators go through? Is that something we have an answer for? Is that something... How do we even think about that?

    3. CGCristos Goodrow1:00:42

       Well, the first thing is we want to make sure that the YouTube systems are not contributing to this-

    4. LFLex Fridman1:00:48

       Right.

    5. CGCristos Goodrow1:00:49

       ... sense, right? And so, um, we've done a fair amount of research to demonstrate that you can absolutely take a break.... if you are a creator and you've been uploading a lot, uh, we have just as many examples of people who took a break and came back more popular than they were before, as we have examples of going the other way.

    6. LFLex Fridman1:01:10

       Yeah. Can we pause on that for a second? So, the feeling that people have, I think, is, "If I take a break, everybody will, the party will leave," right? So, ca- ca- ca- if you could just linger on that. So, in your sense that taking a break is okay.

    7. CGCristos Goodrow1:01:24

       Yes. Taking a break is absolutely okay, and the reason I say that is because, um, we ha- we can observe many examples of being, of creators, um, coming back very strong and even stronger after they have taken some sort of break. And so I just want to dispel the myth that this somehow, um, necessarily, uh, means that your channel is gonna go down or, or lose views. That is not the case. We know for sure that this is not a necessary outcome. Um, and so-

    8. LFLex Fridman1:02:00

       Th-

    9. CGCristos Goodrow1:02:00

       ... we, we want to encourage people to make sure that they take care of themselves. That is job one, right? Like, you, you have to look after yourself and your mental health. Um, and, you know, I think that it probably, in some of these cases, uh, contributes to, uh, better videos once they come back, right? Because a lot of people, I mean, I know myself, if I'm burnt out on something then I'm probably not doing my best work, even though I can keep working, uh, until I pass out. And so, um, I think that the, the taking a break, uh, may even improve the creative ideas that someone has.

    10. LFLex Fridman1:02:39

        Okay. I think that's a really important thing to sort of, to dispel. I think that applies to all of social media. Like, literally, I've taken a break for a day every once in a while. (laughs)

    11. CGCristos Goodrow1:02:50

        (laughs)

    12. LFLex Fridman1:02:51

        Sorry, th- uh, sorry if that sounds like a short time. (laughs)

    13. CGCristos Goodrow1:02:54

        (laughs)

    14. LFLex Fridman1:02:54

        But even like a, sort of, email, just taking a break from email or only checking email once a day, eh, especially when you're going through something psychologically in your personal life or so on, or really not sleeping much 'cause of work deadlines. It can refresh you in a way that's, uh, that's profound. And so the same applies-

    15. CGCristos Goodrow1:03:11

        And it was there when you came back, right?

    16. LFLex Fridman1:03:12

        It's there. So when ... And, and it's, looks different actually when you come back. You're sort of brighter-eyed with some coffee. Everything, the world looks better. So it, it's important to take a break when you need it. So

14. 1:03:27 – 1:08:36

    YouTube algorithm: heuristics, machine learning, human behavior

    1. LFLex Fridman1:03:27

       you've mentioned kind of the, the YouTube algorithm isn't, you know, E equals MC squared. It's not a single equation. It's, it's potentially sort of more than a million lines of code. (sighs) Sort of, is it more akin to what autonomous, successful autonomous vehicles today are, which is, they're just basically patches on top of patches of heuristics and human experts really tuning the algorithm and have some machine learning modules? Or is it becoming more and more a giant machine learning system, with humans just doing a little bit of tweaking here and there? What's your sense ... First of all, do you even have a sense of what is the YouTube algorithm at this point, and whichev- however much you do have a sense, what does it look like?

    2. CGCristos Goodrow1:04:15

       Well, we don't usually think about it as the algorithm.

    3. LFLex Fridman1:04:19

       (laughs)

    4. CGCristos Goodrow1:04:19

       Um, because it's a bunch of systems that-

    5. LFLex Fridman1:04:21

       Right.

    6. CGCristos Goodrow1:04:22

       ... work on different services. Uh, the other thing that I think people don't understand is that what you might refer to as the YouTube algorithm from outside of YouTube is actually a, you know, a bunch of code and machine learning systems and heuristics, but that's married with the behavior of all the people who come to YouTube every day.

    7. LFLex Fridman1:04:44

       So, the people are part of the code, essentially. (laughs)

    8. CGCristos Goodrow1:04:46

       Exactly. Right? Like, if there were no people who came to YouTube tomorrow, then their, the algorithm wouldn't work anymore.

    9. LFLex Fridman1:04:52

       Right.

    10. CGCristos Goodrow1:04:52

        Right? So that's a critical part of the algorithm. And so when people talk about, "Well, the algorithm does this, the algorithm does that," it's sometimes hard to understand. Well, you know, it could be the, the, the viewers are doing that and the algorithm is mostly just keeping track of what the viewers do, and then reacting to those things, um, in, in sort of more fine-grained situations. And I, and I think that this is the way that the recommendation system and the search system and, and probably many machine learning systems evolve, is, you know, you start trying to solve a problem, and the first way to solve a problem is often with a simple heuristic. Right? And, and, you know, you wanna say, "What are the videos we're gonna recommend? Well, how about the most popular ones?" Right? (laughs)

    11. LFLex Fridman1:05:40

        (laughs) Yeah.

    12. CGCristos Goodrow1:05:40

        And that's where you start.

    13. LFLex Fridman1:05:41

        Right.

    14. CGCristos Goodrow1:05:41

        Um, and, and over time, you collect some data and you refine your situations so that you're making less heuristics and you are, you're building a system that can actually learn what to do in different situations, based on some observations of those situations in the past. And, uh, and you keep chipping away at these heuristics over time. And so I think that, um, just like with diversity, uh, you know, I think the first diversity, uh, measure we took was, okay, not more than three videos in a row from the same channel. Right? It's a pretty simple heuristic-

    15. LFLex Fridman1:06:16

        Yeah.

    16. CGCristos Goodrow1:06:17

        ... to encourage diversity, but it worked. Right? Y- who needs to see four, five, six videos in a row from the same channel? Um, and over time, we try to chip away at that and, and make it more fine-grained and, and basically have it remove the heuristics in favor of something that can react to individuals and individual sit- situations.

    17. LFLex Fridman1:06:41

        So, how do you ... You mentioned, you know, we, we know that something worked. How do you get a sense when decisions or the kind of A/B testing that this idea was a good one, this was not so good? Uh, what's, h- how do you measure that and, uh, across which time scale, across how many users, that kind of, that kind of thing?

    18. CGCristos Goodrow1:07:02

        Well, you mentioned the A/B experiments, and so, uh, just about every single change we make to YouTube, uh, we do it only after we've run a A/B experiment.

    19. LFLex Fridman1:07:13

        Mm-hmm.

    20. CGCristos Goodrow1:07:13

        And so-In those experiments, which run from one week to months, um, we measure hundreds, literally hundreds of different variables and, and measure changes with confidence intervals in all of them, because we really are trying to get a sense for ultimately, does this improve the experience for viewers? That's the question we're trying to answer. And an experiment is one way, um, because we can see certain things go up and down. So, for instance, um, if we notice in the experiment people are dismissing videos less frequently or they're, um, saying that they're more satisfied, they're giving more videos five stars after they watch them, then those would be indications of, uh, that the experiment is successful, that it's improving the situation for viewers. Um, but we can also look at other things. Like, we might do, uh, user studies, where we invite some people in and ask them, like, "What do you think about this? What do you think about that? How do you feel about this?" Um, and other various kinds of user research. But ultimately, before we launch something, we're gonna want to run an experiment so we get a sense for, uh, what the impact is gonna be, not just to the viewers, but also to the different channels and all of that.

15. 1:08:36 – 1:10:27

    How to make a viral video?

    1. LFLex Fridman1:08:36

       An absurd question, nobody knows... Well, actually it's interesting, maybe there is an answer. But, uh, if I want to make a viral video, how do I do it?

    2. CGCristos Goodrow1:08:45

       I don't know how you make a viral video. I, I know that we have, in the past, tried to figure out if we could detect when a video, video was going to go viral. You know, and those were, you take the first and second derivatives of the view count and maybe use that to, um, do some prediction. But, um, but I can't say we ever got very good at that. (laughs) Uh, oftentimes we look at where the traffic was coming from, you know, if it's, if it, if a lot of the viewership is coming from something like Twitter, uh, then, then maybe it has a higher chance of becoming viral than maybe if, than, than if it were coming from search or something. Um, but that was just trying to detect a video that might be viral. How to make one?

    3. LFLex Fridman1:09:31

       (laughs)

    4. CGCristos Goodrow1:09:31

       Like, I have no idea. (laughs) I mean, you, you get your kids to interrupt you while you're-

    5. LFLex Fridman1:09:35

       Yeah. (laughs)

    6. CGCristos Goodrow1:09:35

       ... on the new- on the news or something.

    7. LFLex Fridman1:09:37

       Absolutely. Uh, but after the fact, on one individual video, sort of ahead of time predicting is a really hard task, but after the video went viral, in analysis, can you sometimes understand why it went viral? From the perspective of YouTube broadly. First of all, is it even interesting for YouTube that a particular video is vi- uh, is viral? Or is, does that not matter for the individual, for, for the experience of people?

    8. CGCristos Goodrow1:10:05

       Well, I think people expect that if a video, video is going viral and it's something they would be interested in, then I would, I think they would expect YouTube to recommend it to them.

    9. LFLex Fridman1:10:16

       Right. So if something's going viral, it's good to just let the wave-

    10. CGCristos Goodrow1:10:20

        (laughs)

    11. LFLex Fridman1:10:20

        ... let, let people ride the wave of it, its virality.

    12. CGCristos Goodrow1:10:23

        Well, I mean, we want to meet people's expectations in that way, of course.

16. 1:10:27 – 1:13:20

    Veritasium: Why Are 96,000,000 Black Balls on This Reservoir?

    1. CGCristos Goodrow1:10:27

    2. LFLex Fridman1:10:27

       So, like, like I mentioned, I hung out with Derek Muller a while ago, uh, a couple of months back. He's actually the person who suggested I talk to you on this podcast.

    3. CGCristos Goodrow1:10:38

       All right. Well, thank you, Derek.

    4. LFLex Fridman1:10:40

       (laughs) At, at that time, he'd just recently posted, uh, an awesome science video titled, Why Are 96 Million Black Balls on This Reservoir? And in a matter of, I don't know how long, but like a few days, he got 38 million views, and it's still growing. Is this something you can analyze and understand why it happened? This video, or any one particular video like it?

    5. CGCristos Goodrow1:11:06

       I mean, we can surely see where it was recommended, where it was found, who watched it, and those sorts of things.

    6. LFLex Fridman1:11:14

       So it's actually, sorry to interrupt, it is the video which helped me discover who Derek is. I didn't know who he is before.

    7. CGCristos Goodrow1:11:22

       (laughs)

    8. LFLex Fridman1:11:22

       So, I, I remember, you know, usually I just have all of these technical, boring MIT Stanford talks in my recommendation, 'cause that's what I watch. And then all of a sudden there's this black balls in a reservoir video with like an excited nerd-

    9. CGCristos Goodrow1:11:36

       (laughs)

    10. LFLex Fridman1:11:37

        ... and uh, with like just ... And why is this being recommended to me? So I clicked on it and watched the whole thing and it was awesome. But, and then a lot of people had that experience, like, "Why was I recommended this?" (laughs) But they all, of course, watched it and enjoyed it, which is ... What's your sense of this just wave of recommendation that, that comes with this viral video that ultimately people get enjoy after they click on it?

    11. CGCristos Goodrow1:12:00

        Well, I think it's the system, you know, basically doing what anybody who's recommending something would do, which is you show it to some people, and if they like it you say, "Okay, well, can I find some more people who are a little bit like them? Okay, I'm gonna try it with them. Oh, they like it too."

    12. LFLex Fridman1:12:15

        Yeah.

    13. CGCristos Goodrow1:12:15

        "Let me expand the circle some more, find some more people. Oh, it turns out they like it too."

    14. LFLex Fridman1:12:19

        So interesting.

    15. CGCristos Goodrow1:12:19

        And you just keep going until you get some feedback that says that, "No, now you've gone too far. These people don't like it anymore."

    16. LFLex Fridman1:12:25

        Right.

    17. CGCristos Goodrow1:12:25

        Um, and so I, I think that's basically what happened. Now, um, you asked me about how to make a video go viral or make a viral video. I don't think that, uh, if you or I decided to make a video about 96 million balls, that it would also go viral.

    18. LFLex Fridman1:12:42

        (laughs)

    19. CGCristos Goodrow1:12:42

        It's possible that Derek made like (laughs) , um, the canonical video-

    20. LFLex Fridman1:12:47

        Yeah.

    21. CGCristos Goodrow1:12:47

        ... about those black balls in the lake.

    22. LFLex Fridman1:12:50

        Yeah. Exactly.

    23. CGCristos Goodrow1:12:50

        And so um-

    24. LFLex Fridman1:12:51

        He did actually.

    25. CGCristos Goodrow1:12:52

        Right.

    26. LFLex Fridman1:12:52

        Exactly.

    27. CGCristos Goodrow1:12:52

        And, and so I don't know whether or not, uh, just following along is the secret.

    28. LFLex Fridman1:12:58

        Right. Yeah, but it's fascinating. I mean, just like you said, the algorithm sort of expanding that circle and then figuring out that more and more people did enjoy it, and that sort of, uh, phase shift of just a huge number of people enjoying it, and the algorithm quickly, automatically, I assume, figuring that out, that's a ... I don't know, the dynamics, the psychology of that is a beautiful thing. And-

17. 1:13:20 – 1:18:07

    Making clips from long-form podcasts

    1. LFLex Fridman1:13:20

    2. CGCristos Goodrow1:13:20

       (laughs)

    3. LFLex Fridman1:13:21

       So what do you think about the idea of, of clipping? Like, and, and too many people en- annoyed me into doing it, which is they were requesting it.

    4. CGCristos Goodrow1:13:29

       (laughs)

    5. LFLex Fridman1:13:29

       And they said it would be very beneficial to add clips in, in like, the coolest points, and actually have explicit videos. Like I'm re-uploading a video, like a short clip, which is what the, the podcasts are doing. Yeah, do, do you see... as opposed to like, I also add timestamps for the topics, you know, people want the clip. Do you see YouTube somehow helping creators with that process, or helping connect clips to the original videos?

    6. CGCristos Goodrow1:13:56

       Well-

    7. LFLex Fridman1:13:56

       Or is that just on a long list of amazing features-

    8. CGCristos Goodrow1:13:59

       (laughs) .

    9. LFLex Fridman1:13:59

       ... to, to work towards? (laughs)

    10. CGCristos Goodrow1:14:00

        Yeah, I mean, it's not something that I think we've, we've done yet. But I can tell you that, um, I think clipping is great, and I think it's actually great for you as a creator.

    11. LFLex Fridman1:14:12

        Right.

    12. CGCristos Goodrow1:14:12

        And here's the reason, um, if you think about... I mean, let's, let's say the NBA is uploading, uh, videos of, of its games. Well, people might search for Warriors versus Rockets, or they might search for Steph Curry.

    13. LFLex Fridman1:14:30

        Hmm.

    14. CGCristos Goodrow1:14:31

        And so a highlight from the game in which Steph Curry makes an amazing shot, um, is an opportunity for someone to find a portion of that video. And so I think that, um, you never know how people are gonna search for something that you've created. And so you wanna... I would say, you wanna make clips and, and add titles and things like that, so that they can find it, um, as easily as possible.

    15. LFLex Fridman1:14:58

        Do you ever dream of a future, perhaps a distant future, when the YouTube algorithm figures that out? Sort of automatically detects the parts of the video that are really interesting, exciting, potentially exciting for people, and sort of clip them out in this incredibly space? 'Cause if you talk about... if you talk... even just this conversation, we probably covered 30, 40 little topics. And there's a huge space of users that would find, you know, 30% of those topics really interesting, and that space is very different. It, it's something that's beyond my ability to clip out, right?

    16. CGCristos Goodrow1:15:37

        (laughs) .

    17. LFLex Fridman1:15:37

        But the algorithm might be able to figure all that out, sort of expand into clips. Do you have a... do you think about this kind of thing? Do you have a hope, a dream that one day the algorithm will be able to do that kind of deep content analysis?

    18. CGCristos Goodrow1:15:51

        Well, we've actually had projects that attempt to achieve this. Uh, but it really does depend on understanding the video well, and our understanding of the video right now is quite crude. And so, um, I think it would be especially hard to do it with a conversation like this. Uh, one might be able to do it with, um, let's say a soccer match more easily, right? You could probably find out where the goals were scored. And then, of course, you, you need to figure out who it was that scored the goal (laughs) -

    19. LFLex Fridman1:16:25

        Yeah.

    20. CGCristos Goodrow1:16:25

        ... and, and that might require a human to do some annotation. But I think that, um, trying to identify coherent topics in a transcript, like, like the one-

    21. LFLex Fridman1:16:35

        Yeah.

    22. CGCristos Goodrow1:16:35

        ... of our conversation is, um, is not something that w- we're gonna be very good at right away.

    23. LFLex Fridman1:16:42

        And I was speaking more to the general problem actually, of being able to do both the soccer match and our conversation-

    24. CGCristos Goodrow1:16:48

        Right. (laughs)

    25. LFLex Fridman1:16:48

        ... without explicit... sort of, uh, almost my, my hope was that there exists an algorithm that's able to find exciting things in video.

    26. CGCristos Goodrow1:17:00

        So Google now, on, um, Google Search, will help you find the segment of the video that you're interested in. So if you, uh, search for something like how to change the filter in my dishwasher, then if there's a long video about your dishwasher, and this is the part where the person shows you how to change the filter, then, then it will highlight that area-

    27. LFLex Fridman1:17:21

        And-

    28. CGCristos Goodrow1:17:22

        ... and provide a link directly to it.

    29. LFLex Fridman1:17:24

        And do you know, uh, d- d... if... from your recollection, do you know if the thumbnail reflects... like, what's the difference between showing the full video and the shorter clip? Do you know wh- how it's presented in the search results?

    30. CGCristos Goodrow1:17:34

        I don't remember how it's presented. And the other thing I would, uh, say is that right now, it's based on creator annotations.
18. 1:18:07 – 1:20:04

    Moment-by-moment signal of viewer interest

    1. LFLex Fridman1:18:07

       And I wish... I, I know there's privacy concerns, but I wish, um, YouTube explored this space, which is sort of putting a camera on the users if they allowed it, right?

    2. CGCristos Goodrow1:18:19

       (laughs)

    3. LFLex Fridman1:18:19

       To study their, uh, uh... like I did, I did a lot of emotion recognition work and so on. Uh, to study actual sort of richer signal. One of the cool things when you upload 360, like VR video to YouTube, and I've done this a few times, so I've, I've uploaded myself. It's a horrible idea, uh, some people enjoyed it, but whatever, the video of me giving a lecture in, in 360 of the 360 camera. And it's cool because YouTube allows you to then watch wh- where did people look at? There's a heat map of where, you know-

    4. CGCristos Goodrow1:18:50

       Right.

    5. LFLex Fridman1:18:50

       ... of where the center of the VR experience was. And it's interesting 'cause that reveals to you, like what people looked at, and it's v- it's very-

    6. CGCristos Goodrow1:18:58

       It's not always what you were expecting.

    7. LFLex Fridman1:18:59

       ... even though it's not... in the case of the lecture, it's pretty boring, it is what we're expecting. But we did a few funny videos where there's a bunch of people doing things, and they... everybody tracks those people, you know? In the beginning, they all look at the main person, and then they start spreading around and looking at the other people. It's fascinating. So that kind of... that's a really strong signal of what p- people found exciting in the video. I don't know how you get that from people just watching, except they tuned out at this point. Like it's hard to measure this moment was super exciting for people. I don't know how you get that signal. Maybe comment... is there a way to get that signal where this was like... this was when their eyes opened up and they're like-

    8. CGCristos Goodrow1:19:40

       (laughs)

    9. LFLex Fridman1:19:40

       Like for me with the Ray Dalio video, right? Like at first I was like, "Oh, okay, this is another one of these, like, dumb it down for you videos." And then you like start watching, it's like, okay, there's this really crisp, clean, deep explanation of how the economy works. That's where I like set up and started watch... right? That moment, is there a way to detect that moment?

    10. CGCristos Goodrow1:19:59

        ... the only way I can think of is by asking people to-

    11. LFLex Fridman1:20:02

        Just ask.

    12. CGCristos Goodrow1:20:02

        ... label it.

    13. LFLex Fridman1:20:03

        Yeah.

19. 1:20:04 – 1:21:54

    Why is video understanding such a difficult AI problem?

    1. LFLex Fridman1:20:05

       You mentioned that we're quite far away in terms of doing video analysis, deep video analysis. Like w- of course, Google, YouTube, you know, uh, we're quite far away from solving the autonomous driving problem too.

    2. CGCristos Goodrow1:20:19

       (laughs)

    3. LFLex Fridman1:20:19

       So it's, uh-

    4. CGCristos Goodrow1:20:20

       I don't know. I think we're closer to that. (laughs)

    5. LFLex Fridman1:20:21

       Well, (laughs) the, the, you know, you never know. And, uh, the Wright brothers thought they're never, they're not gonna fly for 50 years three years before they flew. So, uh, what are the biggest challenges would you say? Is it the broad challenge of understanding video, understanding natural language, understanding the, the, the challenge before the entire machine learning community of just being able to understand data? Or is there something specific to video that's even more challenging than u-understanding natural language understanding? What's your sense of what the biggest challenges are?

    6. CGCristos Goodrow1:20:54

       I mean, video is just so much information. And so precision becomes a real problem. It's like a, you know, you're, you're trying to classify something and you've got a million classes. And you, the distinctions among them, at least from a, from a machine learning perspective, are often pretty small, right? Like, um, uh, you know, you need to see this person's number in order to know which player it is.

    7. LFLex Fridman1:21:27

       Mm-hmm.

    8. CGCristos Goodrow1:21:27

       And, and there's a lot of players. Um, or you need to see, uh, you know, the, the, the logo on their chest in order to know (laughs) like, which, which team they play for. And so, um, and that's just figuring out who's who, right? And then you go further in saying, "Okay, well, you know, was that a goal? Was it not a goal?" Like, "Is that an interesting moment?" as you said.

    9. LFLex Fridman1:21:48

       (laughs)

    10. CGCristos Goodrow1:21:48

        Or, "Is that not an interesting moment?" Um, these things can be pretty hard.

    11. LFLex Fridman1:21:53

        So,

20. 1:21:54 – 1:25:44

    Self-supervised learning on video

    1. LFLex Fridman1:21:54

       okay, so Yann LeCun, I'm not sure if you're familiar sort of with his current thinking and work. So he, he believes that self, w-w what he's referring to as self-supervised learning will be the solution sort of to achieving this kind of greater level of intelligence. In fact, the thing he's focusing on is watching video and predicting the next frame. So predicting the future of video, right?

    2. CGCristos Goodrow1:22:17

       (laughs)

    3. LFLex Fridman1:22:17

       Uh, so for now, we're very far from that. But his thought is because it's unsupervised-

    4. CGCristos Goodrow1:22:23

       Uh-huh.

    5. LFLex Fridman1:22:23

       ... or as he, he refers to as self-supervised, you know, if you watch enough video, essentially if you watch YouTube, you'll be able to learn about the nature of reality, the physics, the common sense, the reasoning required, by just teaching a system to predict the next frame. So he's confident-

    6. CGCristos Goodrow1:22:41

       Yeah.

    7. LFLex Fridman1:22:41

       ... this is the way to go. So, so you from the perspective of just working with this video, how, uh, d-do you think an algorithm that just watches all of YouTube, stays up all day and night-

    8. CGCristos Goodrow1:22:54

       (laughs)

    9. LFLex Fridman1:22:54

       ... watching YouTube would be able to understand enough of the physics of the world about the way this world works, be able to do common sense reasoning and so on?

    10. CGCristos Goodrow1:23:04

        Um, well, I mean, we have systems that already watch all the videos on YouTube, right? But they're just looking for very specific things, right? They're, um, supervised learning systems that are trying to identify something or classify something. Um, and I don't know if-

    11. LFLex Fridman1:23:22

        But-

    12. CGCristos Goodrow1:23:22

        ... I don't know if predicting the next frame is really gonna get there, because, uh, I don't, I'm not an expert on compression algorithms. But I understand that that's kind of what compression, video compression algorithms do is they basically try to predict the next frame and, and, um, and then fix up the places where they got it wrong. Uh, and that leads to higher compression than if you actually put all the bits for the next frame there. So, so I, I don't know if I believe that just being able to predict the next frame, uh, is gonna be enough. Because, because there's so many frames, and even a tiny bit of error on a per frame basis can lead (laughs) to wildly different videos.

    13. LFLex Fridman1:24:02

        So the thing is, the idea of compression is, uh, one way to do compression is to describe through text what's contained in the video. That's the ultimate high level of compression. So the idea is, uh, traditionally when you think of video image compression y- you're trying to maintain the same visual quality while reducing the size. But if you think of deep learning from a bigger perspective of what compression is, is you're trying to summarize the video. And the idea there is if you have a big enough neural network, just by watching the next, try to predict the next frame, you'll be able to form a compression of actually understanding what's going on in the scene. If there's two people talking, you can just reduce that entire video into the fact that two people are talking and maybe the content of what they're saying and so on. That, that's kind of the, the open-ended dream.

    14. CGCristos Goodrow1:24:55

        (laughs)

    15. LFLex Fridman1:24:55

        So, uh, I just wanted to sort of express that 'cause it's an interesting, compelling notion. But, uh, it, it is nevertheless true that video, our world is a lot more complicated than we get it credit for.

    16. CGCristos Goodrow1:25:07

        Well, I mean, in terms of search and discovery, we have been working on trying to summarize videos in text or, or with some kind of labels for eight years at least. And, uh, you know, and we're kind of so-so. (laughs) And so...

    17. LFLex Fridman1:25:25

        So if you were to say it's, the problem is 100% solved and eight years ago it was 0% solved, how f- where are we on that timeline would you say?

    18. CGCristos Goodrow1:25:37

        Yeah. To summarize a video well, uh, maybe less than a quarter of the way.

21. 1:25:44 – 1:26:46

    What does YouTube look like 10, 20, 30 years from now?

    1. LFLex Fridman1:25:44

       So on that topic, what does YouTube look like 10, 20, 30 years from now?

    2. CGCristos Goodrow1:25:51

       I mean, I think that YouTube is evolving to take the place of TV. Um, you know, I grew up as a kid in the '70s, and I watched a tremendous amount of television. And, uh-... I feel sorry for my poor mom because, uh, people told her at the time that it was going to rot my brain and that she should kill her te- television. Um, but anyway, I mean, I think that YouTube is, at least for my family, a- a better version of television, right? It's one that is on demand. It's, uh, more tailored to the things that my kids, uh, want to watch, and also they can find things that, um, they would never have found on television. And so, I think that, at least from just observing my own family, that's where we're headed, is that people watch YouTube kind of in the same way that I watched television when I was younger.

Episode duration: 1:30:51