Peter Norvig: Artificial Intelligence: A Modern Approach | Lex Fridman Podcast #42

1. 0:00 – 0:37

   Introduction

   1. LFLex Fridman0:00

      The following is a conversation with Peter Norvig. He's the director of research at Google and the co-author with Stuart Russell of the book Artificial Intelligence: A Modern Approach that educated and inspired a whole generation of researchers, including myself, to get into the field of artificial intelligence. This is the Artificial Intelligence podcast. If you enjoy it, subscribe on YouTube, give it five stars on iTunes, support on Patreon, or simply connect with me on Twitter @lexfridman, spelled F-R-I-D-M-A-N. And now, here's my conversation with Peter Norvig.

2. 0:37 – 9:11

   Artificial Intelligence: A Modern Approach

   1. LFLex Fridman0:37

      Most researchers in the AI community, including myself, own all three editions, red, green, and blue, of the, uh, Artificial Intelligence: A Modern Approach. It's a field-defining textbook, as many people are aware, that you wrote with Stuart Russell. How has the book changed and how have you changed-

   2. PNPeter Norvig0:55

      (laughs) Yeah.

   3. LFLex Fridman0:55

      ... uh, in relation to it from the first edition to the second to the third and now fourth edition as you work on it?

   4. PNPeter Norvig1:00

      Yeah. So it's, so it's been a lot of years, a lot of changes. One of the things changing from the first to m- m- maybe the second or third was just the rise of, uh, computing power, right? So I think in the, in the first edition we said, uh, "Here's predicate logic, but, uh, that only goes so far 'cause pretty soon you have millions of, uh, short little predicate expressions and they couldn't possibly fit in memory. Uh, so we're gonna use first-order logic that's, uh, more concise." And then we quickly r- realized, "Oh, predicate logic is pretty nice because there are really fast SAT solvers and other things, and look, there's only millions of expressions and that fits easily into memory, or maybe even billions fit into memory now." So, that was a change of, uh, the type of technology we needed just because the hardware expanded.

   5. LFLex Fridman1:56

      Even to the second edition?

   6. PNPeter Norvig1:58

      Yeah. Yeah.

   7. LFLex Fridman1:58

      So resource constraints were loosened significantly for the second edition?

   8. PNPeter Norvig2:01

      Yeah. Yeah. And then-

   9. LFLex Fridman2:02

      And that was early 2000s, second edition?

   10. PNPeter Norvig2:05

       Right. So '95-

   11. LFLex Fridman2:06

       Yeah.

   12. PNPeter Norvig2:06

       ... was the first and then, uh, 2000, 2001 or so. And then, uh, moving on from there, I think we're s- we're starting to see that again with the, uh, GPUs and then, uh, more specific type of, uh, machinery like the TPUs and w- we're seeing custom ASICs and so on, uh, for deep learning. So, we're seeing another advance in terms of the hardware. Then I think another thing that we especially noticed this time around is in all three of the first editions, we kind of said, "Well, we're gonna find AI as maximizing expected utility."

   13. LFLex Fridman2:42

       Mm-hmm.

   14. PNPeter Norvig2:43

       "And you tell me your utility function and now we've got 27 chapters worth of cool techniques for how to optimize that." I think in this edition, we're saying more, "You know what? Maybe that optimization part is the easy part-

   15. LFLex Fridman2:57

       Mm-hmm.

   16. PNPeter Norvig2:58

       ... and the hard part is deciding what is my utility function? What do I want? And if I'm a collection of agents or a society, uh, what do we want as a whole?"

   17. LFLex Fridman3:08

       So, you touched that topic in this edition. You get-

   18. PNPeter Norvig3:10

       Yeah.

   19. LFLex Fridman3:10

       ... a little bit more into utility.

   20. PNPeter Norvig3:12

       Yeah. Yeah.

   21. LFLex Fridman3:12

       That's really interesting. Uh, on a, uh, a technical level or almost pushing the philosophical?

   22. PNPeter Norvig3:17

       I guess it, it is philosophical, right? So we, we've always had a philosophy chapter, which, which I was, uh, glad to s- that we were supporting. And now, it's less kind of the, uh, you know, Chinese room-type argument-

   23. LFLex Fridman3:33

       Mm-hmm.

   24. PNPeter Norvig3:33

       ... and more of these, uh, ethical and societal-type issues. Uh, so we get into, uh, the issues of, uh, fairness and bias and, uh, and just the issue of, uh, aggregating utilities.

   25. LFLex Fridman3:46

       So, how do you encode human values into a utility function?

   26. PNPeter Norvig3:49

       (laughs)

   27. LFLex Fridman3:50

       Is, is there something that you can do purely through data in a learned way or is there some systematic... Obviously, there's no good answers yet. There's just, uh-

   28. PNPeter Norvig3:59

       Yeah.

   29. LFLex Fridman4:00

       ... beginnings to this, uh, to, to even opening the door to these questions.

   30. PNPeter Norvig4:02

       Right. So, there is no one answer. Yes, there are techniques, uh, to try to learn that. So we talk about inverse reinforcement learning.
3. 9:11 – 15:42

   Covering the entire field of AI

   1. LFLex Fridman9:12

      so maybe taking a quick step back, uh, at the beginning of the Artificial Intelligence: The Modern Approach book or writing, so here you are in the '90s when you first sa- uh, sat down with Stuart to write the book, uh, to cover an entire field, which is one of the only books that has successfully done that for AI and actually in- in a lot of other computer science fields. You know, it's a diff- it's a h- it's a huge undertaking. So, m- th- it must've been quite daunting. What was that process like? Did you envision that you would be trying to cover the entire field? Was there a systematic approach to it that was more step-by-step? How was-

   2. PNPeter Norvig9:50

      Yeah.

   3. LFLex Fridman9:50

      ... how did it feel?

   4. PNPeter Norvig9:52

      So I guess it came about, you know, I'd go to lunch with the other AI faculty at Berkeley and we'd say, uh, you know, "The field is changing. Seems like the current books are a little bit behind. Nobody's come out with a new book recently. We should do that." And everybody said, "Yeah, yeah. That's a great thing to do." Uh, and we never did anything.

   5. LFLex Fridman10:10

      (laughs) Right.

   6. PNPeter Norvig10:11

      And then I ended up, uh, heading off to, uh, industry. I went to, uh, Sun Labs, so I thought, "Well, that's the end of my possible academic publishing career."

   7. LFLex Fridman10:20

      Mm-hmm.

   8. PNPeter Norvig10:21

      But I met Stuart again at a conference, like a year later and said, "You know that book we were always talking about? You guys must be half done with it by now, right?"

   9. LFLex Fridman10:30

      (laughs)

   10. PNPeter Norvig10:32

       And he said, "Well, we keep talking. We never do anything."

   11. LFLex Fridman10:34

       Right.

   12. PNPeter Norvig10:34

       So I said, "Well, you know, we should do it." And I think the reason is that we all felt it was a time where the field was changing, and th- that was in two ways. So, you know, the good old-fashioned AI was based, uh, primarily on Boolean logic, and you had a few tricks to deal with uncertainty. And it was based pr- primarily on knowledge engineering-

   13. LFLex Fridman11:00

       Mm-hmm.

   14. PNPeter Norvig11:00

       ... that the way you got something done is you went out and you interviewed an expert and you wrote down by hand everything they knew. And we saw in- in, uh, '95 that the field was changing in- in two ways. One, we were moving more towards probability ru- rather than Boolean logic, and we were moving more towards machine learning rather than knowledge engineering.

   15. LFLex Fridman11:20

       Mm-hmm.

   16. PNPeter Norvig11:21

       Uh, and the other books, uh, hadn't caught that wave. They were still in the, uh, more in the- in the old school, although certainly they had part of that, uh, on the way. But we said, "If we start now completely taking that point of view, we can have a- a different kind of book." And we were able to put that together.

   17. LFLex Fridman11:40

       And, uh, what was literally the process if you remember? What... Did you start writing a chapter? Did you outline s-

   18. PNPeter Norvig11:48

       Yeah. I guess, I guess we did an- an outline and then we sort of assigned chapters to each person. At the time, uh, I had moved to Boston and Stuart- Stuart was in Berkeley, so basically, uh, we did it, uh, uh, over the internet and- and, uh, you know, that's n- that wasn't the same as doing it today. (laughs) It meant, uh, you know, uh, dial-up lines and Telnetting in-

   19. LFLex Fridman12:13

       Right.

   20. PNPeter Norvig12:13

       ... and, and (laughs) -

   21. LFLex Fridman12:13

       (laughs)

   22. PNPeter Norvig12:14

       ... you know, you, uh, you Telnet it into, uh, one shell and you type cat filename and you-

   23. LFLex Fridman12:21

       Right.

   24. PNPeter Norvig12:21

       ... hoped it was captured at the other end and...

   25. LFLex Fridman12:23

       And certainly you're not sending, uh, images and figures back and forth.

   26. PNPeter Norvig12:27

       Right, right, that didn't work.

   27. LFLex Fridman12:29

       (laughs) But, you know, did you anticipate where the field would go, uh, from that day, uh, from, uh, from the '90s? Did you see the growth into learning-based methods, into data-driven methods, that followed in the future decades?

   28. PNPeter Norvig12:47

       W- we certainly thought that, uh, learning was important. I guess we, w- we missed it as, uh, being as important as it, as it is today. We mi- we missed this idea of big data. We missed it, uh, uh... the idea of deep learning hadn't been invented yet. We could have, uh, taken the book from a complete, uh, machine learning point of view right from the start. We chose to do it more from a point of view of, we're gonna first develop the different types of representations and we're gonna talk about different types of environments, of, uh, is it fully observable or partially observable and is it, uh, deterministic or stochastic and so on. And we, uh, made it more complex along those axes rather than, uh, focusing on the machine learning axis first.

   29. LFLex Fridman13:38

       Do you think... You know, there's some sense in which the deep learning craze, uh, is extremely successful for a particular set of problems and, you know, eventually it's going to, in the general case, hit challenges. And so in terms of the difference between, uh, perception systems and robots that have to act in the world, do you think, uh, we're gonna return to AI modern approach type breadth in edition five and six-

   30. PNPeter Norvig14:08

       Yeah.
4. 15:42 – 18:31

   Expert systems and knowledge representation

   1. LFLex Fridman15:42

      looking at the, some success but certainly, uh, eventual demise, the partial demise of experts as symbolic, uh, systems in the '80s, do you think there is kernels of wisdom in the work that was done there with logic and reasoning and so on that will rise again in your view?

   2. PNPeter Norvig16:05

      So certainly I think the idea of representation and reasoning is crucial, that, uh, you know, sometimes you just don't have enough data about the world to learn de novo, uh, so you've got to have a, a, some idea of representation, whether that was programmed in or told or whatever, and then be able to take, uh, steps of reasoning. I, I think the problem, uh, with, uh, you know, the good old-fashioned AI was, uh, one, we tried to base everything on these, uh, symbols that were atomic. And that's great if you're, like, trying to define the properties of a triangle.

   3. LFLex Fridman16:47

      Right.

   4. PNPeter Norvig16:47

      Right? Because they have necessary and sufficient conditions. Uh, but things in the real world don't. The real world is, is messy and doesn't have sharp edges, and atomic symbols do. So that was a, a poor match. And then the other aspect was that the, uh, reasoning was universal and applied anywhere, which in some sense is good, but it also means there's no guidance as to where to apply.

   5. LFLex Fridman17:14

      Mm-hmm.

   6. PNPeter Norvig17:15

      And so you, you know, you started getting these paradoxes like, uh, uh, "Well, if I have a mountain and I remove one grain of sand, uh, then it's still a mountain," and, "But if I do that repeatedly, at some point it's not," right? And, uh, with logic, you know, there's nothing to stop you from applying things, uh, repeatedly. Uh, but maybe with, uh, something like, uh, deep learning, and I don't really know what the right name for it is, uh, we could separate out those ideas. So, one, we could say, uh, you know, a mountain isn't just an atomic notion. It, it's some sort of, something like a word embedding that, uh, uh, has a, uh, a more complex representation.

   7. LFLex Fridman18:01

      Mm-hmm. Yeah.

   8. PNPeter Norvig18:02

      And secondly, we could somehow learn, yeah, there's this rule that you can remove one grain of sand, uh, and you can do that a bunch of times but you can't do it, uh, a near infinite amount of times. But on the other hand, when you're doing induction on the integers, sure, then it's fine to do it an infinite number of times. And if we could l- uh, somehow we have to learn when these strategies are applicable-... rather than having the strategies be completely neutral and avai- uh, available everywhere.

5. 18:31 – 23:15

   Explainable AI

   1. PNPeter Norvig18:31

   2. LFLex Fridman18:31

      Anytime you use neural networks, anytime you learn from data or form representation from data in an automated way, it's not very explainable as to, uh, or it's not introspective to us humans in terms of, uh, how this neural network sees the world. Where... Why does it succeed so brilliantly on so many, in so many cases and fail so miserably in surprising-

   3. PNPeter Norvig18:55

      Yeah.

   4. LFLex Fridman18:55

      ... ways and small. So, what do you think is this... is, um, the future there? Can simply more data, better data, more organized data solve that problem, or is there elements of symbolic systems that need to be brought in which are a little bit more explainable?

   5. PNPeter Norvig19:12

      Yeah. So, I prefer to talk about trust and, uh, validation and verification rather than just about explainability. And then I think, uh, explanations are one tool that you use towards those goals. And I think it is an important issue that, uh, we don't want to use these systems unless we trust them and we want to understand where they work and where they don't work, and- and an explanation can be part of that, right? So, I apply for loan and I get, uh, denied, uh, I want some explanation of why, and uh, you have, uh, in Europe we have the GDPR that says, uh, you're required to be able to get that. But on the other hand, an explanation alone is not enough, right? So, you know, we were used to dealing with people and with, uh, organizations and corporations and so on, and they can give you an explanation and you have no guarantee that that explanation relates to reality.

   6. LFLex Fridman20:10

      Right.

   7. PNPeter Norvig20:10

      Right? So, the bank can tell me, "Well, you didn't get the loan 'cause you didn't have enough collateral," and that may be true or it may be true that they just didn't like my, uh, religion or- or something else. Uh, I can't tell from the explanation, and that's, uh, that's true whether the decision was made by a computer or by a person. So, I want more. I do want to have the explanations and I want to be able to, uh, have a conversation to go back and forth-

   8. LFLex Fridman20:38

      Mm-hmm.

   9. PNPeter Norvig20:38

      ... and said, "Well, you gave this explanation, but what about this?"

   10. LFLex Fridman20:41

       Mm-hmm.

   11. PNPeter Norvig20:41

       "And what would have happened if this had happened? And, uh, what would- what would I need to change that?" So, I think a conversation is- is a better way to think about it than just, uh, an explanation as a single output. Uh, and I think we need testing of various kinds, right? So, in order to know, was the decision really based on my collateral or was it based on my, uh, religion or skin color or whatever? I can't tell if I'm only looking at my case, but if I look across all the cases, then I can detect a pattern.

   12. LFLex Fridman21:15

       Right.

   13. PNPeter Norvig21:15

       Right? So, you want to have that kind of capability. Uh, you want to have these adversarial testing, right? So, we thought we were doing pretty good at, uh, object recognition in- in images. We said, "Look, we're- we're at sort of pretty close to human level of performance on ImageNet," and so on. Uh, and then you start seeing these adversarial images and you say, "Wait a minute, that part is nothing like (laughs) human performance." Uh-

   14. LFLex Fridman21:39

       Yeah, you can mess with it really easily.

   15. PNPeter Norvig21:40

       You can mess with it really easily, right?

   16. LFLex Fridman21:42

       Yeah.

   17. PNPeter Norvig21:42

       And, uh, yeah, you can do that to humans too, right? So...

   18. LFLex Fridman21:46

       In a different way, perhaps.

   19. PNPeter Norvig21:47

       Right. Humans don't know what color the dress was.

   20. LFLex Fridman21:49

       Right.

   21. PNPeter Norvig21:50

       And so they're vulnerable to certain attacks that are different than the attacks on the- on the machines, but the, you know, the attacks on the machines are so striking, uh, they really change the way you think about what we've done, right?

   22. LFLex Fridman22:03

       Mm-hmm.

   23. PNPeter Norvig22:03

       And the- and the way I think about it is, I think part of the problem is we're seduced by, uh, our low-dimensional metaphors, right?

   24. LFLex Fridman22:13

       (laughs) Yeah. I like-

   25. PNPeter Norvig22:14

       So, you know, you look-

   26. LFLex Fridman22:15

       I like that phrase. (laughs)

   27. PNPeter Norvig22:15

       You look in a- in a textbook and you say, "Okay, now we've mapped out the space and, you know, uh, cat is here and dog is here, and maybe there's a tiny little spot in the middle-

   28. LFLex Fridman22:27

       Yeah.

   29. PNPeter Norvig22:27

       ... where you can't tell the difference, but mostly we've got it all covered." And if you believe that metaphor, uh, then you say, "Well, we're nearly there," and, uh, you know, there's only going to be a couple adversarial images.

   30. LFLex Fridman22:38

       Yeah.
6. 23:15 – 25:47

   Trust

   1. LFLex Fridman23:15

      so on. But take it back to the... this, uh, this word trust. Uh, do you think we're a little too hard on our robots in terms of-

   2. PNPeter Norvig23:23

      (laughs) .

   3. LFLex Fridman23:23

      ... uh, the standards we apply? So, you know, of, uh, there's a dance, there's a- there's a- there's a dance in non-verbal and verbal communication between humans. You know, if we apply the same kind of standard in terms of humans, you know, we trust each other pretty quickly. Uh, you know, you and I haven't met before and there's some degree of trust (laughs) .

   4. PNPeter Norvig23:48

      Yeah.

   5. LFLex Fridman23:48

      Right? That, uh, nothing's gonna go crazy wrong. And yet to AI, when we look at AI systems or... we seem to approach, uh, through skepticism always, always.

   6. PNPeter Norvig23:58

      Yeah.

   7. LFLex Fridman23:58

      And it's like they have to prove through a lot (laughs) of hard work that they're even worthy of, uh, even inkling of our trust. What do- what do you- what do you think about that? How- how do we break that barrier, close that gap?

   8. PNPeter Norvig24:11

      I think that's right. I think that's a big issue. Uh, just listening, uh, my friend, uh, Mark Moffett is a naturalist and he says, uh, "The most amazing thing about humans is that you can walk into a- a coffee shop or a, uh, a busy street in a city..."... and there's lots of people around you that you've never met before, and you don't kill each other.

   9. LFLex Fridman24:33

      (laughs) Yeah.

   10. PNPeter Norvig24:34

       He says, "Chimpanzees cannot do that."

   11. LFLex Fridman24:36

       Yeah, right. (laughs)

   12. PNPeter Norvig24:37

       Right? If a chimpanzee is in a situation where, "Here's some, uh, that aren't from my tribe..." Bad things happen.

   13. LFLex Fridman24:46

       Especially in a coffee shop, there's delicious food around, you never know.

   14. PNPeter Norvig24:49

       Yeah, yeah. But, but we humans have figured that out.

   15. LFLex Fridman24:52

       Yeah.

   16. PNPeter Norvig24:52

       Right? Uh, and you know-

   17. LFLex Fridman24:54

       For the most part.

   18. PNPeter Norvig24:55

       ... for the most part. We still go to war, we still do terrible things, uh, but for the most part, we've learned to trust each other and, and live together. Uh, so that's gonna be important for our, uh, our AI systems as well, and I th- also, I think, uh, you know, a lot of the emphasis is on AI, uh, but in many cases, uh, AI is part of the technology, but isn't really the main thing. So a lot of, of what we've seen is more due to communications technology-

   19. LFLex Fridman25:24

       Mm-hmm.

   20. PNPeter Norvig25:25

       ... than AI te- AI technology. Yeah, you wanna make these good decisions, but the reason, uh, we're able to have any kind of system at all is we've got the communication so that we're collecting the data and so that we can reach lots of people around the world. I think that's a, a bigger change that we're dealing with.

   21. LFLex Fridman25:45

       Speaking of reaching a lot of people around

7. 25:47 – 32:43

   Education - Intro to AI - MOOC

   1. LFLex Fridman25:47

      the world, on the side of education, you've, uh ... one of the many things in terms of education you've done, you taught the Intro to Artificial Intelligence course that signed up 100,000- 160,000 students. It was one of the first successful example of an massive, uh, of a MOOC, massive open online course. What did you learn from that experience? Uh, what do you think is the future of MOOCs, of education online?

   2. PNPeter Norvig26:12

      Yeah. It was a great fun doing it, particularly, uh, being right at the start just because it was exciting and new, but it also meant that we had less competition.

   3. LFLex Fridman26:24

      (laughs) Yeah.

   4. PNPeter Norvig26:24

      Right? So, uh, one of the things you hear about, uh, "Well, the problem with MOOCs is, uh, the completion rates are, are so low, so they must be a failure." And, and I got to admit, I'm a prime contributor, right?

   5. LFLex Fridman26:37

      Mm-hmm.

   6. PNPeter Norvig26:37

      I've probably, uh, started 50 different courses that I haven't finished, but I got exactly what I wanted out of them because I had never intended to finish them. I just wanted to, uh, dabble in a little bit, either to see the topic matter or just to see the pedagogy of, "How are they doing this class?" So I guess the main thing I learned is, when I came in, I thought, uh, the challenge was information, saying, "If I'm just to take the stuff I want you to know and I'm very clear and explain it well, then my job is done and, uh, good things are gonna happen." Uh, and then in, in doing the course, I learned, uh, well, yeah, you gotta have the information, but really, the motivation is the most important thing, that, uh, if students don't stick with it, then it doesn't matter how good the content is.

   7. LFLex Fridman27:29

      Mm-hmm.

   8. PNPeter Norvig27:29

      Uh, and I think being one of the first classes, we were helped by a sort of exterior motivation. So, we tried to do a good job at making it enticing and setting up, uh, uh, ways for, uh, you know, the community to work with each other, to make it more motivating, but really, a lot of it was, "Hey, this is a, a new thing, and I'm really excited to be part of a new thing." And so the students brought their own motivation. And so I think this is great because there's lots of people around the world who have never had this before-

   9. LFLex Fridman28:00

      Mm-hmm.

   10. PNPeter Norvig28:00

       ... you know, uh, who'd never have the opportunity to, uh, go to Stanford and take a class, or go to MIT, or go to one of the other schools. Uh, but now we can bring that to them, and if they bring their own motivation, uh, they can be successful in a way they couldn't before. But that's really just the top tier of people that are ready to do that. The rest of the people, uh, j- just don't see or, you know, don't have the motivation and don't see how, if they push through and were able to do it, what advantage that would get them. Uh, so I think we got a long way to go before we're able to do that, and I think it'll be m- some of it is based on technology, but more of it's based on the idea of community, that you gotta actually get people together. Some of that getting together can be done online. I think some of it really has to be done in person to be able to, in order to build that type of, uh, community and trust.

   11. LFLex Fridman28:56

       You know, there's an intentional mechanism that we've developed, uh, a short attention span, especially younger people, um, because sort of shorter and shorter videos online, uh, there's a, whatever the, th- the way the brain is dev- is developing now with people that have grown up with the internet, they have a, quite a short attention span. So, and I, I would say I had the same when I was growing up too, probably for different reasons. So, I probably wouldn't have, uh, learned as much as I have if I wasn't forced to sit in a physical classroom-

   12. PNPeter Norvig29:30

       Mm-hmm.

   13. LFLex Fridman29:31

       ... sort of bored, sometimes falling asleep-

   14. PNPeter Norvig29:34

       Sure. (laughs)

   15. LFLex Fridman29:34

       ... but sort of forcing myself through that process in sometimes extremely difficult computer science courses. What, what's the difference, in your view, between in-person education experience, which you, uh, first of all, yourself had and you yourself taught-

   16. PNPeter Norvig29:50

       Yeah.

   17. LFLex Fridman29:50

       ... and online education?

   18. PNPeter Norvig29:52

       Right.

   19. LFLex Fridman29:52

       And how do we close that gap, if it's even possible?

   20. PNPeter Norvig29:54

       Yeah. So, I think there's two issues. One is whether it's in person or online, so sort of the physical location, and then the other is, uh, kind of the affiliation, right? So, you stuck with it in part because you were in the classroom and you saw everybody else was suffering-

   21. LFLex Fridman30:14

       Right. (laughs)

   22. PNPeter Norvig30:15

       ... the same, the same way you were.

   23. LFLex Fridman30:16

       Yeah.

   24. PNPeter Norvig30:16

       Uh, but also because you were enrolled, you had paid tuition.

   25. LFLex Fridman30:21

       Yeah.

   26. PNPeter Norvig30:22

       Sort of everybody was expecting you to stick with it. Uh-

   27. LFLex Fridman30:25

       Mm-hmm. Society, parents-

   28. PNPeter Norvig30:26

       Yeah.

   29. LFLex Fridman30:27

       ... c- uh, peers.

   30. PNPeter Norvig30:28

       Right.
8. 32:43 – 37:12

   Learning to program in 10 years

   1. LFLex Fridman32:43

      So our field, programming, you've also done a lot of, you've done a lot of programming yourself. In, uh, 2001, you wrote a great article about programming called Teach Yourself Programming in 10 Years. Sort of responds to-

   2. PNPeter Norvig32:58

      Yeah.

   3. LFLex Fridman32:58

      ... all the books that say Teach Yourself Programming in 21 Days. So, if you were giving advice to someone getting into programming today, this is, uh, a few years since you've written that article, what's the best way to undertake that journey?

   4. PNPeter Norvig33:10

      I think there's lots of different ways and I think, uh, programming means more things now. And I guess, you know, when I wrote that article, I was thinking more about becoming a professional software engineer. And I thought that's a, you know, a c- sort of a career-long, uh, field of study. Uh, but I think there's lots of things now that people can do where programming is a part of solving what they wanna solve, uh, without it achieving that professional-level status.

   5. LFLex Fridman33:44

      Yeah.

   6. PNPeter Norvig33:44

      Right? So, I'm not gonna be going and writing a million lines of code, but, you know, I'm a biologist or a physicist or something, or a, even a historian, and I've got some data, and I wanna ask a question of that data. And I think for that, uh, you don't need 10 years, right? So, eith- there are many shortcuts to, uh, being able to a- answer those kinds of questions. And, and you know, you see today a lot of, uh, emphasis on, uh, learning to code-

   7. LFLex Fridman34:13

      Mm-hmm.

   8. PNPeter Norvig34:14

      ... teaching kids how to code. Uh, I think that's great. Uh, but I wish they would change the message a little bit, right? So, I think code isn't the main thing. I don't really care if you know the syntax of JavaScript or if you can, uh, connect these blocks together in this visual language. Uh, but what I do care about is that you can analyze a problem, uh, you can, uh, think of a solution, you can, uh, carry out, uh, you know, make a model, run that model, test the model, see the results, uh, uh, verify that they're reasonable, uh, ask questions and answer them. Right? So, it's more, uh, modeling and problem-solving-

   9. LFLex Fridman34:58

      Mm-hmm.

   10. PNPeter Norvig34:58

       ... and you use coding in order to do that, uh, but it's not just learning coding for its own sake.

   11. LFLex Fridman35:04

       That's really interesting. So, it's actually almost, in many cases, it's learning to work with data, to extract-

   12. PNPeter Norvig35:10

       Yeah.

   13. LFLex Fridman35:10

       ... something useful-

   14. PNPeter Norvig35:11

       Yeah.

   15. LFLex Fridman35:11

       ... out of data. So, when you say problem-solving, you really mean taking some kind of, maybe collecting some kind of dataset, cleaning it up, and saying something interesting about it-

   16. PNPeter Norvig35:20

       Yeah.

   17. LFLex Fridman35:20

       ... which is useful in all kinds of domains.

   18. PNPeter Norvig35:23

       And, uh, you know, and I see myself, uh, being stuck sometimes in kind of the, the old ways.

   19. LFLex Fridman35:29

       Mm-hmm.

   20. PNPeter Norvig35:30

       Right? So, you know, I'll be working on a project, uh, maybe with a, a younger employee and we say, "Oh, well, here's this new package that could help solve this, uh, problem." And I'll go and I'll start reading the manuals and, you know, I'll be-

   21. LFLex Fridman35:45

       (laughs)

   22. PNPeter Norvig35:45

       ... two hours into reading the manuals and then, uh, my colleague comes back and says, "I'm done."

   23. LFLex Fridman35:51

       Yup.

   24. PNPeter Norvig35:51

       You know? "I downloaded the package. I installed it. I tried calling some things. The first one didn't work. The second one worked. Now I'm done." And, and I say, "But I have 100 questions about how does this work and how does that work?" And they say, "Who cares," right? "I don't need to understand the whole thing. I unders- I answered my question. It's a big complicated package. I don't understand the rest of it, but I got the right answer." And I'm just, it's hard for me to get into that mindset. I want to understand the whole thing and, you know, if they wrote a manual, I should probably read it. And, but that's not necessarily the right way. And I, I think I have to get used to dealing with more, being more comfortable with uncertainty and not knowing everything.

   25. LFLex Fridman36:32

       Yeah. So, I struggle with the same. It's sort of the, the spectrum between Donald, Don Knuth-

   26. PNPeter Norvig36:37

       Yeah.

   27. LFLex Fridman36:37

       ... who's kind of the very, you know, b- before he can say anything m-... about a problem, he really has to get down to the machine code as, uh, assembly.

   28. PNPeter Norvig36:45

       Yeah.

   29. LFLex Fridman36:46

       Versus exactly what you said, of, uh... of several students in my group that, uh, you know, 20 years old and they can solve almost any problem within a few hours that would take me probably weeks, because I would try to, as you said, read the manual. So, do you think the nature of mastery is... You're, you're mentioning biology, sort of outside disciplines, applying programming, but

9. 37:12 – 40:01

   Changing nature of mastery

   1. LFLex Fridman37:12

      computer scientists... So over time, there's higher and higher levels of abstraction available now. So with, with, uh, this week-

   2. PNPeter Norvig37:20

      Hm. Yeah.

   3. LFLex Fridman37:20

      ... there's the, the TensorFlows Summit, right?

   4. PNPeter Norvig37:23

      Yeah.

   5. LFLex Fridman37:23

      (laughs) So if you're, if you're not particularly into deep learning, but you're still a computer scientist, uh, you can accomplish an incredible amount with, uh, TensorFlow without really knowing any fundamental internals of machine learning. Do you think the nature of mastery is, is changing, uh, even for computer scientists, like what it means to be an expert programmer?

   6. PNPeter Norvig37:45

      Yeah, I think that's true. You know, we never really should have focused on programmer, right? Because that's still... it's, it's a skill and what we really want to focus on is the result. So we, we built this, uh, ecosystem where the way you can get stuff done is by programming it yourself.

   7. LFLex Fridman38:03

      Right.

   8. PNPeter Norvig38:04

      At least when I started, it... you know, library functions meant you had square root and that was about it.

   9. LFLex Fridman38:10

      (laughs)

   10. PNPeter Norvig38:10

       (laughs) Right?

   11. LFLex Fridman38:10

       Yeah.

   12. PNPeter Norvig38:10

       Everything else you built from scratch.

   13. LFLex Fridman38:12

       Yeah.

   14. PNPeter Norvig38:13

       And then we built up an ecosystem where a lot of times, well, you can download a lot of stuff that does-

   15. LFLex Fridman38:18

       Yeah.

   16. PNPeter Norvig38:18

       ... a big part of what you need. And so now, it's more a question of, uh, assembly rather than, uh, uh, manufacturing, and, uh, that's a different way of looking at problems.

   17. LFLex Fridman38:32

       From another perspective, in terms of mastery and looking at programmers or people that reason about problems in a computational way, so Google, uh, you know, the... from the hiring perspective, from the perspective of hiring or building a team of programmers, uh, how do you determine if someone's a good programmer? Or if somebody... again-

   18. PNPeter Norvig38:52

       Yeah.

   19. LFLex Fridman38:52

       ... because I don't want to deviate from... I want to move away from the word programmer, but somebody who could solve problems of large-scale data and so on. What's, what's, uh... how do you build a team like that through the interviewing process for them?

   20. PNPeter Norvig39:03

       Yeah, and I, and I think, uh, as a company grows, uh, you get more, uh, expansive in the types of people you're looking for, right? So, uh, I think, you know, in the early days, we'd interview people and the question we were trying to ask is, uh, how close are they to Jeff Dean?

   21. LFLex Fridman39:23

       (laughs)

   22. PNPeter Norvig39:25

       (laughs) And most people-

   23. LFLex Fridman39:26

       Sure.

   24. PNPeter Norvig39:26

       ... were pretty far away.

   25. LFLex Fridman39:26

       Yeah.

   26. PNPeter Norvig39:26

       But we'd take the ones that were, you know, not that far away.

   27. LFLex Fridman39:28

       Yeah.

   28. PNPeter Norvig39:29

       And so we got kind of a homogeneous group of people who were really great programmers.

   29. LFLex Fridman39:33

       Yeah.

   30. PNPeter Norvig39:34

       Uh, then as the company grows, you say, "Well, we don't want everybody to be the same, to have the same skillset." And so now, we're, uh, hiring, uh, biologists in our health areas and we're hiring physicists and we're hiring, uh, mechanical engineers and we're hiring, uh, you know, uh, social scientists and ethnographers and people with different backgrounds, uh, who bring different skills.

10. 40:01 – 41:17

    Code review

    1. LFLex Fridman40:01

       Mm-hmm. So you, uh, mentioned that you, uh, still may partake in code reviews. Given that you have a wealth of experience, as you've also mentioned (laughs) , uh, what errors do you often see and tend to highlight in the code of junior developers, of people coming up now, uh, given your background from LISP-

    2. PNPeter Norvig40:23

       Oh.

    3. LFLex Fridman40:23

       ... to, uh, a couple of decades of programming?

    4. PNPeter Norvig40:26

       Yeah, that's a great question. You know, sometimes I try to look at the flexibility of the design, of, yes, you know, this API solves this problem, but, uh, where is it going to go in the future? Who else is gonna wanna call this?

    5. LFLex Fridman40:41

       Mm-hmm.

    6. PNPeter Norvig40:42

       And, uh, you know, are, are you making it easier for them to do that?

    7. LFLex Fridman40:47

       Is this a matter of, uh, design? Is it documentation? Is it, is it, uh, sort of an amorphous thing you can't really-

    8. PNPeter Norvig40:54

       Yeah.

    9. LFLex Fridman40:54

       ... put into words? It's just how it feels? If you put yourself in the shoes of a developer, would you use this kind of thing?

    10. PNPeter Norvig40:59

        I think it is how you feel, right?

    11. LFLex Fridman41:01

        Right.

    12. PNPeter Norvig41:01

        And so, yeah, documentation is good, uh, but it's, but it's more a design question, right? If you get the design right, then people will figure it out, whether the documentation's good or not, and if-

    13. LFLex Fridman41:12

        Right.

    14. PNPeter Norvig41:12

        ... and if the design's wrong, uh, then it'll be harder to use.

    15. LFLex Fridman41:16

        How have,

11. 41:17 – 43:05

    How have you changed as a programmer

    1. LFLex Fridman41:17

       uh, you, yourself, changed as a programmer-

    2. PNPeter Norvig41:20

       (laughs)

    3. LFLex Fridman41:20

       ... over the years, as... in, in the way... you've already started to say sort of, you want to read the manual, you want to understand the core of the-

    4. PNPeter Norvig41:30

       Yeah.

    5. LFLex Fridman41:30

       ... from the syntax to the how the language is supposed to be used and so on. But, uh, what's the evolution been like from the '80s, '90s, to today?

    6. PNPeter Norvig41:40

       I guess one thing is, you don't have to worry about, uh, the small details of efficiencies as much-

    7. LFLex Fridman41:46

       Exactly.

    8. NANarrator41:46

       So true.

    9. PNPeter Norvig41:46

       ... as you used to, right? So, like, I remember, uh, I did my, uh, LISP book, uh, in the '90s, and one of the things I wanted to do was say, uh, here's how you do an object system, and, uh, basically, uh, we're gonna make it so each object is a hash table, and you look up the methods, and here's how it works. And then I said, "Of course, the real common LISP object system is much more complicated. It's got all these efficiency-type issues-"

    10. LFLex Fridman42:14

        Mm-hmm.

    11. PNPeter Norvig42:15

        "... and this is just a toy. Nobody would do this in real life." And it turns out, Python pretty much did exactly- (laughs)

    12. LFLex Fridman42:22

        (laughs)

    13. PNPeter Norvig42:22

        ... what I said-

    14. LFLex Fridman42:23

        Yeah.

    15. PNPeter Norvig42:24

        ... and said, uh, objects are, uh, just dictionaries. And yeah, they have a few little, uh, tricks as well. But mostly, you know, the thing that would've been 100 times too slow in the '80s is now plenty fast for most everything today.

    16. LFLex Fridman42:39

        So you had to, as a programmer, let go of...... perhaps an obsession that I remember coming up with of trying to write efficient code.

    17. PNPeter Norvig42:48

        Yeah, yeah. To say, you know, what really matters is the total time, uh, it takes to get the project done.

    18. LFLex Fridman42:55

        (laughs) Yeah.

    19. PNPeter Norvig42:56

        And most of that's gonna be the programmer time.

    20. LFLex Fridman42:58

        Yep.

    21. PNPeter Norvig42:58

        Uh, so if you're a little bit less efficient, but it makes it easier to understand and modify, then that's the right trade-off.

12. 43:05 – 47:41

    LISP

    1. PNPeter Norvig43:05

    2. LFLex Fridman43:05

       So, you've written quite a bit about Lisp. Your book on programming is in Lisp. You, you, you have a lot of code out there that's in Lisp. So, um, uh, m- myself and people who don't know what Lisp is should look it up.

    3. PNPeter Norvig43:17

       Yeah.

    4. LFLex Fridman43:18

       It's my favorite language. For many AI researchers, it is a favorite language. The favorite language they never use-

    5. PNPeter Norvig43:24

       (laughs)

    6. LFLex Fridman43:24

       ... uh, these days. (laughs) Uh, so what part of Lisp do you find most beautiful and powerful?

    7. PNPeter Norvig43:29

       So, I think the beautiful part is the simplicity. T- that in half a page, you can define the whole language. And o- other languages don't have that, so you feel like you can hold everything in your head. And then, you know, a lot of people say, "Well, then that's too simple, you know. Here's all these things I wanna do, and uh, you know, my Java or Python or whatever has 100 or 200 or 300 different syntax rules, and don't I need all those?"

    8. LFLex Fridman44:00

       Mm-hmm.

    9. PNPeter Norvig44:00

       And Lisp's answer was, "No, we're only gonna give you eight or so s- syntax rules, but we're gonna allow you to define your own."

    10. LFLex Fridman44:08

        Mm-hmm.

    11. PNPeter Norvig44:08

        Uh, and so that was a very powerful idea. And I think this idea of saying, uh, "I can start with my problem and with my data, and then I can build the language I want, uh, for that problem and for that data, and then I can make Lisp define that language." So, you, uh, you're sort of, uh, mixing levels and saying, "I'm simultaneously a, a programmer in a language and a language designer," and that allows a better match between your problem and your eventual code. And I think Lisp had, uh, done that better than other languages.

    12. LFLex Fridman44:47

        Yeah, it's a very elegant implementation of functional programming, but why do you think Lisp has not had the mass adoption and success-

    13. PNPeter Norvig44:55

        Yeah.

    14. LFLex Fridman44:56

        ... of languages like Python? Is it the parentheses?

    15. PNPeter Norvig44:58

        (laughs)

    16. LFLex Fridman44:59

        Is it all the parentheses?

    17. PNPeter Norvig45:02

        Yeah. (laughs) Yeah. So, I think a couple things. So, one was, I think it was designed for a single programmer or a small team, and a skilled programmer who had the good taste to say, "Well, I'm, I am doing language design and I have to make g- good choices." And if you make good choices, that's great. If you make bad choices, y- uh, you can hurt yourself.

    18. LFLex Fridman45:27

        Mm-hmm.

    19. PNPeter Norvig45:28

        And it can be hard for other people on the team to understand it.

    20. LFLex Fridman45:31

        Right.

    21. PNPeter Norvig45:31

        So, I think there was a, a limit to the scale of the size of a project in terms of number of people that Lisp was good for. And as an industry, we kind of grew, uh, beyond that. I think it is in part the parentheses. You know, one of the jokes is the acronym for Lisp is, uh, Lots Of Irritating Silly Parentheses.

    22. LFLex Fridman45:52

        (laughs)

    23. PNPeter Norvig45:54

        Uh, my acronym was, uh, Lisp Is Syntactically Pure, saying all you need is parentheses and atoms. But I remember, you know, so we had the, the AI textbook and, uh, because we did it in the '90s, we had, uh, we had pseudocode in the book, but then we said, "Well, we'll have Lisp online," 'cause that's the language of AI at the time.

    24. LFLex Fridman46:15

        Mm-hmm.

    25. PNPeter Norvig46:16

        And I remember some of the students complaining 'cause they hadn't had Lisp before and they didn't quite understand what was going on. And, and I remember one student complained, "I don't understand how this pseudocode corresponds to this Lisp." And there was a one-to-one correspondence between the (laughs) the, uh, symbols in the code and the pseudocode, and the only thing difference was the parentheses.

    26. LFLex Fridman46:37

        (laughs)

    27. PNPeter Norvig46:38

        (laughs) So, I said, "It must be that for some people, a certain number of left parentheses shuts off their brain."

    28. LFLex Fridman46:44

        (laughs) Yeah, it's very, it's very possible in that sense, and Python just goes the other way.

    29. PNPeter Norvig46:48

        Yeah.

    30. LFLex Fridman46:48

        It just-
13. 47:41 – 48:32

    Python

    1. LFLex Fridman47:41

       So, what's the story in Python behind pyTudes? Your GitHub repository-

    2. PNPeter Norvig47:46

       Yeah.

    3. LFLex Fridman47:46

       ... with puzzles and exercises in Python is pretty fun.

    4. PNPeter Norvig47:49

       Yeah, just, uh, uh, it seemed like fun. Uh, you, you know, I like, uh, doing puzzles and I like, uh, being an educator. I, I did a class with Udacity, uh, Udacity 212, I think it was, that was basically problem-solving, uh, uh, using Python and looking at different problems. And-

    5. LFLex Fridman48:09

       Does pyTudes feed that class, uh, in terms of the exercises? I was wondering what the-

    6. PNPeter Norvig48:12

       Uh, yeah, so the class, the class came first.

    7. LFLex Fridman48:14

       Yeah.

    8. PNPeter Norvig48:15

       Some of the stuff that's in pyTudes was write-ups of what was in the class, and then some of it was just continuing to, uh, to, uh, work on new problems.

    9. LFLex Fridman48:24

       So, what's the organizing madness of pyTudes? Is it just a-

    10. PNPeter Norvig48:27

        Yeah, it's just-

    11. LFLex Fridman48:27

        ... collec- a, a collection of cool exercises?

    12. PNPeter Norvig48:30

        Just whatever I thought was fun.

    13. LFLex Fridman48:31

        Okay, awesome.

14. 48:32 – 53:24

    Early days of Google Search

    1. LFLex Fridman48:32

       So, you were the director of search quality at Google from-

    2. PNPeter Norvig48:36

       Yeah.

    3. LFLex Fridman48:36

       ... 2001, '02, to 2005. In the early days, uh, when there was just a few employees and when the-

    4. PNPeter Norvig48:43

       Yeah.

    5. LFLex Fridman48:43

       ... when the company was growing like crazy, right? So-Th- I mean, Google revolutionized the way we discover, and share, and aggregate knowledge, so just, this is, uh, this is one of the fundamental aspects of civilization, right, is information being shared, and there's different mechanisms throughout history, but Google has just 10X improved that, right? And you're a part of that, right? People discovering that information. So, what, what were some of the challenges on a philosophical or the technical level in those early days?

    6. PNPeter Norvig49:18

       It definitely was an exciting time, and as you say, we were doubling in size every year. And the challenges were, we wanted to get the right answers, (laughs) right? And, uh, we had to figure out what that meant. We had to implement that, and we had to make it all, uh, efficient and, uh... We had to keep on testing and seeing if we were delivering good answers. Uh-

    7. LFLex Fridman49:44

       And now when you say good answers, it means whatever people are typing in, in terms of keywords, in terms of that kind of thing, that the- that the results they get are ordered by the desirability for them of those results. Like, they're like... The first thing they click on will likely be the thing that they were actually looking for.

    8. PNPeter Norvig50:01

       Right. One of the metrics we had was focused on the first thing. Uh, some of it wa- fo- was focused on the whole page. Some of it was focused on, you know, the top three or so. So, we looked at a lot of different metrics for- for how well we were doing, and we broke it down into subclasses of, you know, maybe here's a type of, uh- of, uh, query that we're not doing well on. Then we try to fix that. Uh, early on, we started to realize that we were in an adversarial position, right? So, we started thinking, uh, "Well, we're kind of like the card catalog in the library."

    9. LFLex Fridman50:35

       Mm-hmm.

    10. PNPeter Norvig50:36

        Right? So, the books are here, and we're off to the side and- and we're just, uh, reflecting what's there. And then we realized every time we make a change, the webmasters make a change.

    11. LFLex Fridman50:47

        (laughs)

    12. PNPeter Norvig50:48

        And it's, uh, game theoretic. And so we had to think not only of, is this the right move for us to make now? But also, if we make this move, what's the countermove gonna be? Is that gonna get us into a work- worse place? In which case we won't make that move, we'll make a different move.

    13. LFLex Fridman51:05

        And did you find... I mean, I assume with the popularity and the growth of the internet, that people were creating new content. So, you're almost helping guide the creation of new content.

    14. PNPeter Norvig51:14

        Yeah. So, that's certainly true, right? So, we- we kn- we definitely changed, uh, the structure of the network, right? So, if you think back, you know, in the- in the very early days, uh, uh, Larry and Sergey had the PageRank paper, and Jon Kleinberg had this, uh, hubs and authorities model, which says the web is made out of these, uh, hubs, which will be my page of cool links about dogs or whatever-

    15. LFLex Fridman51:44

        Mm-hmm.

    16. PNPeter Norvig51:44

        ... and people would just list links. Uh, and then there'd be authorities, which were the ones, uh, that- page about dogs that most people link to. That doesn't happen anymore. People don't bother to say, "My page of cool links-"

    17. LFLex Fridman51:57

        Mm-hmm.

    18. PNPeter Norvig51:57

        ... uh, 'cause we took over that function, right? So- so, uh, we changed th- the way that worked.

    19. LFLex Fridman52:03

        Did you imagine back then that the internet would be as massively vibrant as it is today? I mean, it was already growing quickly, but it's just another... I- I don't know if you've ever-

    20. PNPeter Norvig52:14

        You know-

    21. LFLex Fridman52:14

        ... if you- today, if you sit back and (laughs) just look at the internet with wonder, the amount of content that's just constantly being created, constantly being shared-

    22. PNPeter Norvig52:23

        Yeah.

    23. LFLex Fridman52:23

        ... and deployed.

    24. PNPeter Norvig52:24

        Yeah. It- it's, uh, it's always been surprising to me.

    25. LFLex Fridman52:27

        (laughs)

    26. PNPeter Norvig52:27

        I- I guess I'm not very good at- at, uh-

    27. LFLex Fridman52:29

        Predicting the future?

    28. PNPeter Norvig52:29

        ... predicting the future.

    29. LFLex Fridman52:30

        Okay. (laughs)

    30. PNPeter Norvig52:31

        Uh, and I remember, you know, being a graduate student in- in 1980 or so, and, uh, you know, we had the ARPANET, and then there was this, uh, proposal to, uh, commercialize it-
15. 53:24 – 55:14

    What does it take to build human-level intelligence

    1. LFLex Fridman53:24

       the future, what do you think it takes to build a system that approaches human-level intelligence? Y- you've talked about, of course, that w- you know, we shouldn't be so obsessed about creating human-level intelligence, just cr- create systems that are very useful for humans. But what do you think it takes-

    2. PNPeter Norvig53:41

       Yeah.

    3. LFLex Fridman53:42

       ... to, uh, to- uh, to, uh, yeah, approach that level?

    4. PNPeter Norvig53:45

       Right. So, certainly, I- I don't think human-level intelligence is one thing, right? So, I think-

    5. LFLex Fridman53:50

       Right.

    6. PNPeter Norvig53:50

       ... there's lots of different tasks, lots of different capabilities. I also don't think, uh, that should be the goal, right? So, I, you know, I wouldn't want to create a, uh, calculator that could do multiplication at human level, right?

    7. LFLex Fridman54:03

       Right.

    8. PNPeter Norvig54:04

       That would- that would be a step backwards. And so for many things, we should be aiming far beyond human level. Uh, for other things, uh, maybe human level is a good level to aim at. Uh, and for others we'd say, "Well, let's not bother doing this 'cause we al- we already have humans who can take on those tasks." So, as you say, I like to focus on, uh, what- what's a useful tool?

    9. LFLex Fridman54:26

       Right.

    10. PNPeter Norvig54:26

        And- and in some cases, being at human level is an important part of crossing that threshold to- to make the tool useful. So, we see in- in things like these, uh, uh, personal assistants now that you get either on your phone or on a- a speaker that sits on the table, uh, you wanna be able to have a conversation with those.

    11. LFLex Fridman54:46

        Mm-hmm.

    12. PNPeter Norvig54:47

        And- and I think as an industry, we haven't quite figured out what the right model is for what these things can do.

    13. LFLex Fridman54:54

        Right.

    14. PNPeter Norvig54:54

        Uh, and we're aiming towards, well, you just have a conversation with them the way you can with a person.

    15. LFLex Fridman54:59

        Right.

    16. PNPeter Norvig54:59

        Uh, but we haven't delivered on that model yet, right? So, you can ask it, "What's the weather?" You can ask it-... play some nice songs, uh, and, uh, you know, five or six other things and then you run out of stuff that it can do.

16. 55:14 – 57:00

    Her

    1. LFLex Fridman55:14

       In terms of, uh, deep meaningful connection, so you've mentioned the movie Her as one of your favorite AI movies. Do you think it's possible for a human being to fall in love with an AI system, AI assistant as you mentioned? So taking this big leap from, uh, "What's the weather?" to, you know-

    2. PNPeter Norvig55:29

       Yeah.

    3. LFLex Fridman55:29

       ... having a, a deep connection.

    4. PNPeter Norvig55:31

       Yeah. I, I think, uh, as people, that's what we love to do.

    5. LFLex Fridman55:35

       Yeah.

    6. PNPeter Norvig55:36

       And, uh, I was at a, uh, a showing of Her where we had a panel discussion and, and somebody asked me, uh, "What other movie do you think Her is similar to?" And my answer was, uh, "Life of Brian." Which, which is not a science fiction movie.

    7. LFLex Fridman55:52

       Mm-hmm.

    8. PNPeter Norvig55:53

       Uh, but both movies are about wanting to believe in something that's not necessarily real.

    9. LFLex Fridman55:59

       (laughs) Yeah. By the way, for people who don't know, it's Monty Python. Yeah.

    10. PNPeter Norvig56:03

        Yeah.

    11. LFLex Fridman56:03

        (laughs) That's been brilliantly put.

    12. PNPeter Norvig56:05

        Right?

    13. LFLex Fridman56:05

        But, so-

    14. PNPeter Norvig56:05

        So, uh, I mean, I think that's just the way we are. We, we want to trust, we want to believe, we want to fall in love and, uh, it doesn't necessarily take that much, right? So, uh, you know, my kids, uh, fell in love with their teddy bear.

    15. LFLex Fridman56:20

        Right.

    16. PNPeter Norvig56:20

        And the teddy bear was not very interactive, right? (laughs)

    17. LFLex Fridman56:23

        (laughs)

    18. PNPeter Norvig56:23

        So that's all us.

    19. LFLex Fridman56:24

        Yeah.

    20. PNPeter Norvig56:25

        Pushing our feelings onto our devices and our things, and, and I think that that's what we like to do, so we'll continue to do that.

    21. LFLex Fridman56:33

        So, yeah, as human beings we long for that connection and just AI has to, uh, do a little bit of work to, uh, to c- catch us-

    22. PNPeter Norvig56:41

        Yeah.

    23. LFLex Fridman56:41

        ... on the other end.

    24. PNPeter Norvig56:41

        Yeah. And certainly, you know, if you can get to, uh, dog level-

    25. LFLex Fridman56:46

        (laughs)

    26. PNPeter Norvig56:46

        ... a lot of people have invested a lot of, uh, love in their pets.

    27. LFLex Fridman56:49

        In their pets. Some, some people, uh, as I've been told in working with autonomous vehicles, have invested a lot of love into their inanimate cars.

    28. PNPeter Norvig56:57

        Yeah. (laughs)

    29. LFLex Fridman56:58

        So it, it really doesn't take much.

17. 57:00 – 58:41

    Test of intelligence

    1. LFLex Fridman57:00

    2. PNPeter Norvig57:00

       Yeah.

    3. LFLex Fridman57:01

       So what is a good test... to linger on the topic that, uh, may be silly or a little bit philosophical, what is a good test of intelligence in your view? Is natural conversation like in the Turing test a good, a good test? Put another way, what would impress you-

    4. PNPeter Norvig57:19

       Yeah.

    5. LFLex Fridman57:20

       ... if you saw a computer do it these days?

    6. PNPeter Norvig57:22

       Yeah, I mean, I get impressed all the time, right?

    7. LFLex Fridman57:24

       (laughs) Okay.

    8. PNPeter Norvig57:26

       Right? So, uh-

    9. LFLex Fridman57:26

       But like really impress you.

    10. PNPeter Norvig57:28

        You know, Go playing, uh, StarCraft playing, uh, those are all pretty cool. You know, and I think, uh, sure, conversation is important. I think, uh, you know, we sometimes have these tests where it's easy to fool the system, where you can have a chatbot that can have a conversation, but you never, uh, it never gets into a situation where it has to be deep enough that, uh, it really reveals itself as, as being in- intelligent or not. I think, uh, you know, Turing suggested that, uh, but I think if he were alive, he'd say, "You know, I didn't really mean that seriously."

    11. LFLex Fridman58:10

        (laughs)

    12. PNPeter Norvig58:10

        Right?

    13. LFLex Fridman58:11

        Yeah. Yeah.

    14. PNPeter Norvig58:11

        And, and I think, uh, and you know, this is just my opinion, but, but I think Turing's point was not that, uh, this test of conversation is a good test. I think his point was having a test is the right thing.

    15. LFLex Fridman58:25

        Mm-hmm.

    16. PNPeter Norvig58:25

        So rather than having the philosopher say, "Oh, no, AI is impossible," you should say, "Well, we'll just have a test and then the result of that will, will tell us the answer." And it doesn't necessarily have to be a conversation test.

    17. LFLex Fridman58:37

        That's right. And coming up a new better test as the technology evolves is probably the right

18. 58:41 – 1:00:58

    Future threats from AI

    1. LFLex Fridman58:41

       way. Do you worry as a lot of the general public does about... not a lot, but some vocal, uh, part of the general public about the existential threat of artificial intelligence? So, looking farther into the future, as you said, most of us are not able to predict much. So when shrouded in such mystery, there's a concern of, well, you think, start thinking about worst case.

    2. PNPeter Norvig59:05

       Mm-hmm. Yeah.

    3. LFLex Fridman59:05

       Is that something that occupies your mind space much?

    4. PNPeter Norvig59:09

       So I certainly think about, uh, threats, I think about, uh, dangers, uh, and I think, uh, any new technology, uh, has positives and negatives, and if it's a powerful technology, it can be used for bad as well as for good. So I'm certainly not worried about, uh, the robot apocalypse, uh-

    5. LFLex Fridman59:29

       Mm-hmm.

    6. PNPeter Norvig59:29

       ... and the Terminator type scenarios. I am worried about change in employment and, uh, are we gonna be re- able to react fast enough to deal with that? I think we're, you know, we're already seeing it today where a lot of people are, are disgruntled about, uh, uh, the way income inequality is, is working and, uh, and automation could help accelerate those kinds of, uh, problems. I see powerful technologies can always be used as weapons, uh, whether they're robots or drones or whatever. Uh, some of that, uh, we're seeing due to AI, a lot of it, uh, you don't need AI.

    7. LFLex Fridman1:00:08

       Mm-hmm.

    8. PNPeter Norvig1:00:08

       Uh, and I don't know what's a, what's a worse threat if it's a autonomous drone or, uh, it's, uh, CRISPR technology becoming available or... we have lots of, uh, threats to face and some of them involve AI and some of them don't.

    9. LFLex Fridman1:00:24

       So the threats that technology presents, are you, for the most part, optimistic about technology also alleviating those threats or creating new opportunities or protecting us from the more detrimental effects of these new technologies?

    10. PNPeter Norvig1:00:38

        Yeah, I don't know. It, it... again, it's hard to predict the future and, uh-

    11. LFLex Fridman1:00:42

        (laughs) Yes.

    12. PNPeter Norvig1:00:43

        ... as a succ- society so far we've survived, uh, nuclear bombs-

    13. LFLex Fridman1:00:48

        Somehow, yeah.

    14. PNPeter Norvig1:00:49

        ... and, and other things. Of course, uh, only societies that have survived are having this conversation.

    15. LFLex Fridman1:00:54

        (laughs)

    16. PNPeter Norvig1:00:54

        So, uh, uh, maybe that's a survivorship bias

19. 1:00:58 – 1:02:57

    Exciting open problems in AI

    1. PNPeter Norvig1:00:58

       there.

    2. LFLex Fridman1:00:58

       Yeah. What problem stands out to you as exciting, challenging, impactful to work on in the near future for yourself, for the community and, and broadly?

    3. PNPeter Norvig1:01:09

       So, I, you know, we talked about these, uh, assistants in conversation. I, I think that's a great area. I think, uh, combining, uh, common sense reasoning, uh, with, uh, the power of data i- is a, a great area.

    4. LFLex Fridman1:01:26

       In which application? In, in conversation relations or just broadly speaking?

    5. PNPeter Norvig1:01:29

       Just in, in general, yeah. As a programmer, I'm interested in, uh, programming tools, both in terms of, uh, you know, the current systems we have today with, with TensorFlow and so on. Can we make them much easier to use for a broader-

    6. LFLex Fridman1:01:44

       Mm-hmm.

    7. PNPeter Norvig1:01:44

       ... uh, class of people? And also, can we apply, uh, machine learning to, uh, the more traditional type of programming, right? So, you know, when you go to Google and you, uh, type in a query and you spell something wrong, it says, "Did you mean..."

    8. LFLex Fridman1:02:00

       Mm-hmm.

    9. PNPeter Norvig1:02:00

       And the reason we're able to do that is 'cause lots of other people made a similar error and then they corrected it. Uh, we should be able to go into our code bases and our bug fix bases and, uh, when I type a line of code, it should be able to say, "Did you mean such and such?" If you type this today, you're probably gonna fi- type in this bug fix, uh, tomorrow.

    10. LFLex Fridman1:02:20

        Yeah, that's a really exciting application of, uh, almost a, a- an assistant for the coding programming experience-

    11. PNPeter Norvig1:02:27

        Yeah.

    12. LFLex Fridman1:02:27

        ... at every level. So, I think I could s- safely speak for the entire AI community (laughs) -

    13. PNPeter Norvig1:02:35

        Mm-hmm.

    14. LFLex Fridman1:02:35

        ... first of all for, uh, thanking you for the amazing work you've done. Uh, certainly for the amazing work you've done with, uh, AI: A Modern Approach book.

    15. PNPeter Norvig1:02:43

        Yeah, thank you.

    16. LFLex Fridman1:02:43

        I think we're all looking forward very much for the fourth edition and then the fifth edition and so on.

    17. PNPeter Norvig1:02:49

        Yeah, yeah.

    18. LFLex Fridman1:02:49

        So, uh, Peter, thank you so much for talking today.

    19. PNPeter Norvig1:02:51

        Yeah, thank you. A pleasure.

Episode duration: 1:03:12