Charles Isbell and Michael Littman: Machine Learning and Education | Lex Fridman Podcast #148

1. 0:00 – 2:27

   Introduction

   1. LFLex Fridman0:00

      The following is a conversation with Charles Isbell and Michael Littman. Charles is the Dean of the College of Computing at Georgia Tech, and Michael is a computer science professor at Brown University. I've spoken with each of them individually on this podcast, and since they are good friends in real life, we all thought it would be fun to have a conversation together. Quick mention of each sponsor, followed by some thoughts related to the episode. Thank you to Athletic Greens, the all-in-one drink that I start every day with to cover all my nutritional bases, Eight Sleep, a mattress that cools itself and gives me yet another reason to enjoy sleep, Masterclass, online courses from some of the most amazing humans in history, and Cash App, the app I use to send money to friends. Please check out these sponsors in the description to get a discount and to support this podcast. As a side note, let me say that, uh, having two guests on the podcast is an experiment that I've been meaning to do for a while, in particular because, uh, down the road, I would like to occasionally be a kind of moderator for debates between people that may disagree in some interesting ways. If you have suggestions for who you would like to see debate on this podcast, let me know. As with all experiments of this kind, it is a learning process. Both the video and the audio might need improvement. I realized, I think, I should probably do three or more cameras next time as opposed to just two, and also try different ways to mount the microphone for the third person. Also, after recording this intro, I'm going to have to go figure out the, uh, thumbnail for the video version of the podcast since I usually put the guest's head on the thumbnail and, uh, now there's two heads and two names to try to fit into the thumbnail. It's a kind of a bin packing problem which in, uh, theoretical computer science happens to be an NP-hard problem. Whatever I come up with, if you have better ideas for the thumbnail, let me know as well. And in general, I always welcome ideas how this thing can be improved. If you enjoy it, subscribe on YouTube, review it with five stars on Apple Podcast, follow on Spotify, support on Patreon, or connect with me on Twitter @lexfridman. And now, here's my conversation with Charles Isbell and Michael Littman.

2. 2:27 – 6:49

   Is machine learning just statistics?

   1. LFLex Fridman2:27

      You'll probably disagree about this question, but what is your biggest, would you say, disagreement about either something, uh, profound and very important or something completely not important at all?

   2. CICharles Isbell2:39

      I don't think we have any disagreements at all.

   3. MLMichael Littman2:41

      Ah, I'm not sure that's true.

   4. CICharles Isbell2:43

      (laughs) We walked into that one, didn't we?

   5. LFLex Fridman2:45

      (laughs) Yeah. That's, that's pretty good.

   6. MLMichael Littman2:46

      So, so one thing that you sometimes mention is that... and we did this one on air too, as it were, whether or not machine learning is computational statistics.

   7. CICharles Isbell2:55

      It's not.

   8. MLMichael Littman2:57

      But it is.

   9. CICharles Isbell2:57

      Well, it's not. And in particular, and more importantly, it is not just computational statistics.

   10. LFLex Fridman3:02

       So what's missing in the picture? What-

   11. CICharles Isbell3:04

       All the rest of it.

   12. LFLex Fridman3:05

       (laughs)

   13. MLMichael Littman3:06

       (laughs)

   14. LFLex Fridman3:06

       (laughs)

   15. CICharles Isbell3:06

       (laughs)

   16. MLMichael Littman3:06

       What's missing? That which is missing. Oh, yes.

   17. CICharles Isbell3:08

       Because it's-

   18. MLMichael Littman3:09

       Well, you can't be wrong now.

   19. CICharles Isbell3:10

       Well, it's not just the statistics. He doesn't even believe this. We've had this conversation before. If it were just the statistics then we would be happy with where we are. But it's not just the statistics.

   20. MLMichael Littman3:19

       That's why it's computational statistics.

   21. CICharles Isbell3:21

       Or if it were just the computational ... kind of statistics.

   22. MLMichael Littman3:22

       I agree that machine learning is not just statistics.

   23. CICharles Isbell3:24

       It is not just statistics, nor is it-

   24. MLMichael Littman3:25

       We can agree on that.

   25. CICharles Isbell3:26

       ... nor is it just computational statistics.

   26. MLMichael Littman3:28

       It's computational statistics.

   27. CICharles Isbell3:29

       It is computational. It's not just a...

   28. LFLex Fridman3:30

       What is the computational in computational statistics? Does this take us into the realm of computing?

   29. CICharles Isbell3:35

       It does, but I think perhaps the way I can get him to admit that, uh, he's wrong-

   30. MLMichael Littman3:39

       That he's wrong. Yeah. (laughs)
3. 6:49 – 9:05

   NeurIPS vs ICML

   1. CICharles Isbell6:49

      ago. What would you say is the difference between, say, the early 2000s, ICML and what we used to call NIPS, NeurIPS?

   2. MLMichael Littman6:57

      Mm.

   3. CICharles Isbell6:57

      Is there a difference? A lot of the... Particularly in the machine learning that was done there?

   4. MLMichael Littman7:00

      Mm.

   5. LFLex Fridman7:01

      ICML was around that long?

   6. CICharles Isbell7:03

      Oh, yeah.

   7. LFLex Fridman7:03

      So, ICLR is the new conference, newish?

   8. CICharles Isbell7:06

      Uh, yeah, I guess so.

   9. LFLex Fridman7:07

      And ICML was around in 2000?

   10. CICharles Isbell7:10

       Oh, ICML predates that.

   11. MLMichael Littman7:11

       I, um, well, I think my most cited ICML paper is from '94.

   12. CICharles Isbell7:15

       Yeah. Michael knows this better than me because, of course, he's significantly older than I am. But the point is-

   13. LFLex Fridman7:18

       Yeah.

   14. CICharles Isbell7:19

       ... what is the difference-

   15. LFLex Fridman7:20

       Years.

   16. CICharles Isbell7:20

       ... what was the difference between ICML and NeurIPS in the late '90s, early 2000s?

   17. MLMichael Littman7:24

       I don't know what everyone else's perspective would be, but I had a particular perspective at that time.

   18. CICharles Isbell7:29

       Which was?

   19. MLMichael Littman7:29

       Which is, I felt like ICML was more of a g- of a computer science place-

   20. CICharles Isbell7:33

       Mm-hmm.

   21. MLMichael Littman7:33

       ... and that N- NIPS/NeurIPS was more of an engineering place, like the kind of math that happened at the two places.

   22. CICharles Isbell7:40

       Oh, interesting.

   23. MLMichael Littman7:40

       As a computer scientist, I felt more comfortable with the ICML math, and the NeurIPS people would say that that's because I'm dumb.

   24. CICharles Isbell7:48

       Mm-hmm.

   25. MLMichael Littman7:48

       And that's such an engineering thing to say. So.

   26. CICharles Isbell7:51

       I agree with that part of it. I, so I do a little differently. We actually had a nice conversation with Tom Dietterich about this i- in public.

   27. MLMichael Littman7:57

       On Twitter.

   28. CICharles Isbell7:57

       On Twitter, just a couple of days ago. I'd put it a little differently, which is that ICML was machine learning done by, uh, computer scientists, and, uh, NeurIPS was, uh, machine learning done by computer scientists trying to impress statisticians.

   29. LFLex Fridman8:11

       (laughs)

   30. MLMichael Littman8:12

       (laughs)
4. 9:05 – 14:49

   Data is more important than algorithm

   1. MLMichael Littman9:05

      We're sorry.

   2. LFLex Fridman9:06

      How does neural networks change this, just to even linger on this topic, change this idea of w- statistics, th- how big s- of a pie statistics is within the machine learning thing? Like, 'cause it sounds like hyperparameters and also just the role of data, you know, there's pe- people are starting to use this terminology of software 2.0, which is like the act of programming as a, as a... Like, you're a designer in the hyperparameter space of neural networks, and you're also the collector and the organizer and the cleaner, uh, of the data. And that's part of the programming. Uh, how, so how did... On the NeurIPS versus ICML topic, what's the role of neural networks in redefining the size and the role of machine learning? Which-

   3. CICharles Isbell9:57

      Well, I can't, I can't wait to, to, to hear what Michael thinks about this, but, um, I would add one other thing.

   4. MLMichael Littman10:01

      But you will.

   5. CICharles Isbell10:01

      (laughs) But I can't (laughs) ... That's true, I will. I'll force myself to. I think the-

   6. LFLex Fridman10:05

      (laughs)

   7. CICharles Isbell10:05

      ... the... There's one other thing I would add to your description, which is the kind of software engineering part of, what does it mean to debug, for example?

   8. LFLex Fridman10:11

      Oh, yeah.

   9. CICharles Isbell10:11

      But this is a difference between, uh, the kind of computational statistics view of machine learning and the c- computational view of machine learning, uh, which is, I think, one is worried about the equation as it were. And by the way, this is not a value judgment. I just think it's about perspective. But the kinda questions you would ask when you start asking yourself, "Well, what does it mean to program and develop and build the system?" is a very computer sciencey view of the problem. I mean, when... If you get on, uh, data science Twitter and econ Twitter, you actually hear this a lot with the, uh, you know, the economist and the data scientist complaining about the machine learning people. "Well, it's d- you know, it's just statistics, and I don't know why they don't, don't see this." But they're not even asking the same questions. They're not thinking about it as a kind of programming problem, and I think that that really matters, just asking this question. I actually think it's a little different from, uh, programming in hyperparameter space and, and sort of collecting the data. I, but I do think that that immersion really matters. So I'll give you a quick y- a quick example of the way I think about this. So, I teach machine learning. Michael and I have co-taught a machine learning class which has now reached, I don't know, 10,000 people at least, over the last several years, or somewhere there'sabouts. And my machine learning assignments are of this form. So, the super f- the first one is something like, "Implement these five algorithms," you know, K-NN, and S... You know, SVMs and boosting and decision trees and neural networks, and maybe that's it, I can't remember. And when I say implement, I mean steal the code. I am completely uninterested. You get zero points for getting the thing to work.

   10. MLMichael Littman11:38

       Mm-hmm.

   11. CICharles Isbell11:38

       I don't want you spending your time worrying about, uh, getting the corner case right of, you know, what happens when you are trying to normalize distances and the points on the thing, and so you divide it by z- I'm not interested in that, right?

   12. MLMichael Littman11:49

       Mm-hmm.

   13. CICharles Isbell11:50

       Steal the code. However, you're going to run those algorithms on two datasets. The datasets have to be interesting. What does it mean to be interesting? Well, a dataset's interesting if it reveals differences between algorithms, which presumably are all the same, because they can represent whatever they can represent. And two datasets are interesting together if they show different differences, as it were. And you have to analyze them. You have to justify their interestingness, and you have to analyze them in a whole bunch of ways. But all I care about is the data in your analysis, not the programming. And I occasionally end up in these long discussions with students... Well, I don't really. I copy and paste the things that I've said-

   14. MLMichael Littman12:24

       (laughs)

   15. CICharles Isbell12:24

       ... the other 15,000 times it's come out, which is... They go, "But the only way to learn-"... really understand is to code them up-

   16. MLMichael Littman12:31

       Yeah.

   17. CICharles Isbell12:31

       ... which is a very programmer software engineering view of the world. If you don't program it, you don't understand it. Which is, I, by the way, I think is wrong in a very specific way, but it is a way that you come to understand because then you have to wrestle with the algorithm.

   18. MLMichael Littman12:44

       Mm-hmm.

   19. CICharles Isbell12:44

       But the thing about machine learning is it's not just sorting numbers, where in some sense the data doesn't matter. What matters is, well, does the algorithm work on these abstract things-

   20. MLMichael Littman12:51

       Mm-hmm.

   21. CICharles Isbell12:51

       ... if one's less than the other? In machine learning, the data matters. It da-

   22. MLMichael Littman12:55

       Mm-hmm.

   23. CICharles Isbell12:55

       ... it matters more than almost anything.

   24. MLMichael Littman12:56

       Mm-hmm.

   25. CICharles Isbell12:57

       And s- not everything, but almost anything. And so as a result, you have to live with the data and don't get distracted by the algorithm per se. And I think that that focus on the data and what it can tell you and what question it's actually answering for you, as opposed to the question you thought you were asking, is a key and important thing about machine learning and is a way that computationalists as opposed to statisticians bring a particular view about how to think about the process. The statisticians, by contrast, bring, I- I think I'd be willing to say, a better view about the kind of formal math that's behind it-

   26. MLMichael Littman13:31

       Mm-hmm.

   27. CICharles Isbell13:31

       ... and what an actual number ultimately is saying about the data. And those are both important, but they're also different.

   28. MLMichael Littman13:38

       I didn't really think of it this way is to build intuition about the role of data, the different characteristics of data by having two data sets that are different and then reveal the differences in the differences.

   29. CICharles Isbell13:50

       Yeah.

   30. MLMichael Littman13:50

       That's- that's a really fasc- that's a really interesting educational approach. The- the students love it, but not right away.
5. 14:49 – 23:33

   The role of hardship in education

   1. CICharles Isbell14:49

      of.

   2. MLMichael Littman14:50

      What's your view... let me put on my Russian hat, which believes that life is suffering- I like Russian hats, by the way. (laughs) If you have one, I would like this. Those are ridiculous, yes.

   3. CICharles Isbell14:59

      (laughs)

   4. MLMichael Littman15:00

      (laughs)

   5. CICharles Isbell15:00

      (laughs)

   6. MLMichael Littman15:00

      But in a delightful way. But sure, of course. Uh- (laughs) what do you think is the role of, uh, we talked about balance a little bit.

   7. CICharles Isbell15:08

      Mm-hmm.

   8. MLMichael Littman15:08

      What do you think is the role of hardship in education? Like, I think the biggest things I've learned, like, the w- what made me fall in love with math, for example, is by being bad at it until I got good at it. So like, like, struggling with a problem, which increased the level of joy I felt when I finally figured it out. And it- it always felt with me with teachers, especially modern discussions of education, how can we make education more fun, more engaging, more all those things? Well, from my perspective, it's like you're maybe missing the point that education, that life is suffering. (laughs) Education is supposed to be hard and that actually what increases the joy you feel when you actually learn something. Is that r- ridiculous? (laughs)

   9. CICharles Isbell16:02

      Oh, I get it.

   10. MLMichael Littman16:02

       Do you like to see your students suffer?

   11. CICharles Isbell16:04

       (laughs)

   12. MLMichael Littman16:04

       Okay. So th- this may be a point where we differ.

   13. CICharles Isbell16:07

       I suspect not.

   14. MLMichael Littman16:08

       Okay.

   15. CICharles Isbell16:08

       But do go on.

   16. MLMichael Littman16:10

       Well, what would your answer be?

   17. CICharles Isbell16:11

       I wanna hear you first.

   18. MLMichael Littman16:12

       Okay. Well, I w- I was gonna not answer the question. (laughs)

   19. CICharles Isbell16:15

       (laughs) 'Cause you don't want the students to know you enjoy them suffering?

   20. MLMichael Littman16:18

       I was gonna... No, no, no, no, no. I was- I was gonna say that there's... I think there's a d- a distinction that you can make in the kind of suffering, right? So, I think you can be in a mode where you're s- you're suffering in a hopeless way versus you're suffering in a hopeful way, right? Where you're like, you can see that if you- that you still have... you can still imagine getting to the end, right? And as long as people are in that mindset where they're struggling, but it's not a hopeless kind of struggling, that's p- that's productive. I think that's really helpful. But if struggling, like, if you- you break their will-

   21. CICharles Isbell16:53

       (laughs)

   22. MLMichael Littman16:53

       ... if you leave them hopeless, no, that don't... I- sure, some people are gonna, whatever, lift themselves up by their bootstraps, but like, mostly you give up and certainly it takes the joy out of it, and you're not gonna spend a lot of time on something that brings you no joy. So it's a- it's- it is a bit of a delicate balance, right? You have to thwart people in a way that they still believe that there's a way through.

   23. CICharles Isbell17:17

       Right. So that's a- that, uh, we strongly agree actually. So I think s- well, first off, struggling and suffering aren't the same thing.

   24. MLMichael Littman17:23

       Mm-hmm.

   25. CICharles Isbell17:23

       Right? One can-

   26. MLMichael Littman17:24

       He's being poetic. (laughs)

   27. CICharles Isbell17:25

       Oh, no, no. I- I actually appreciate the poetry. And- and I- one of the reasons I appreciate it is that they are often the same thing and often quite different, right? So you can struggle without suffering. You can certainly suffer- (laughs) suffer- suffer pretty easily. You don't necessarily have to struggle to suffer. So I think that you want people to struggle, but that hope matters. The- you have to- they have to understand that they're gonna get through it on the other side. And it's very easy to confuse the two. Um, I actually think Brown University has a very just philosophically has a very different take on the relationship with their students, particularly undergrads, from say a place like Georgia Tech-

   28. MLMichael Littman17:59

       Okay.

   29. CICharles Isbell17:59

       ... which is, uh-

   30. MLMichael Littman17:59

       Which university's better?
6. 23:33 – 28:05

   How Charles and Michael met

   1. CICharles Isbell23:33

      Well, let's rewind the clock back to the '50s and '60s when you guys met. (laughs) How did you... I'm just kidding. I don't... Uh, but what, can you tell the story of, of how you met? So you've-

   2. MLMichael Littman23:44

      Mm-hmm.

   3. CICharles Isbell23:44

      ... so like the internet and the world kind of knows you as, as, as, as connected in some ways in terms of education, of teaching the world. That's, that's like the public facing thing. But how did you as human beings and as collaborators, uh, meet? I think there's two stories. One is how we met, and the other is how we-

   4. MLMichael Littman24:05

      Fell in love.

   5. CICharles Isbell24:05

      ... got to know each other.

   6. MLMichael Littman24:06

      (laughs)

   7. CICharles Isbell24:07

      I'm not gonna say fell, I'm not gonna say fell in love.

   8. MLMichael Littman24:08

      (laughs)

   9. CICharles Isbell24:08

      I'm gonna say that we came to understand that we-

   10. MLMichael Littman24:11

       Had some common-

   11. CICharles Isbell24:12

       Something.

   12. MLMichael Littman24:13

       ... something. Yeah.

   13. CICharles Isbell24:14

       Yeah, there you go.

   14. MLMichael Littman24:14

       It's funny, 'cause on the surface I think we're, we're different in a lot of ways. But there's something-

   15. CICharles Isbell24:18

       Yeah. I mean, now we complete each other's-

   16. MLMichael Littman24:19

       ... just constant.

   17. CICharles Isbell24:21

       There you go.

   18. MLMichael Littman24:22

       Afternoons.

   19. CICharles Isbell24:23

       (laughs)

   20. MLMichael Littman24:23

       (laughs)

   21. CICharles Isbell24:23

       So, uh, I will tell the story of how we met and I'll let Michael tell the story of how we met.

   22. MLMichael Littman24:28

       Okay. All right.

   23. CICharles Isbell24:28

       Okay. So here's how we met. Um, I was already at, at that point it was AT&T Labs. There's a long interesting story there. But anyway, I was there and, uh, Michael was coming to interview. He was a professor at Duke at the time but decided for reasons that he wanted to be in New Jersey.Uh, and so that would mean, uh, Bell Labs/A T&T Labs. Uh, and we were doing interview, interviews were very much like academic interviews. Uh, and so I had to be there. Uh, we all had to meet with him afterwards and so on, one-on-one. Uh, but it was obvious to me that he was gonna be hired. Like, no matter what, because everyone loved him. They were just talking about all the great stuff he did. "Oh, he did this great thing." And you had just won something at Triple AI, I think, or maybe you got 18 papers in Triple AI that year.

   24. MLMichael Littman25:08

       But I got, I got the best paper award at Triple AI, for the crossword stuff.

   25. CICharles Isbell25:11

       For the crossword.

   26. LFLex Fridman25:11

       Right, exactly.

   27. MLMichael Littman25:11

       Yeah.

   28. CICharles Isbell25:12

       So that had all happened and everyone was going on and on and on about it. Actually, Satinder was saying incredibly nice things about you.

   29. MLMichael Littman25:16

       Really?

   30. CICharles Isbell25:17

       Yes. So-
7. 28:05 – 31:23

   Key to success: never be satisfied

   1. MLMichael Littman28:05

      in the group.

   2. LFLex Fridman28:05

      Can we take a slight tangent on that-

   3. MLMichael Littman28:07

      Sure.

   4. LFLex Fridman28:07

      ... on this topic of... It sounds like, uh, maybe you could speak to the bigger picture. It sounds like you're quite self-critical.

   5. MLMichael Littman28:15

      Who, Charles?

   6. LFLex Fridman28:15

      No, you.

   7. MLMichael Littman28:16

      Oh.

   8. LFLex Fridman28:17

      Okay, so-

   9. MLMichael Littman28:17

      I think I can, I can do better. I can do better. I'll, I'll... Tr- try me again. I'll, I'll, I'll do better.

   10. LFLex Fridman28:21

       (laughs)

   11. CICharles Isbell28:21

       (laughs)

   12. LFLex Fridman28:21

       (laughs)

   13. MLMichael Littman28:21

       Be so self-critical. I won't, I won't, I won't.

   14. LFLex Fridman28:23

       Yeah, that, that was like a, like a three out of 10 response, so-

   15. MLMichael Littman28:26

       (laughs)

   16. LFLex Fridman28:27

       Uh, so let's, let's try to work it up to five and six. Uh, you know, I remember, uh, Marvin Minsky said, uh, on, uh, on a video interview something that the key to success in academic research is to hate everything you do.

   17. CICharles Isbell28:40

       Hmm.

   18. MLMichael Littman28:41

       Oh.

   19. LFLex Fridman28:42

       Uh, (laughs) for some reason-

   20. MLMichael Littman28:44

       I think I followed that because I hate everything he's done.

   21. LFLex Fridman28:47

       (laughs)

   22. CICharles Isbell28:48

       (laughs)

   23. LFLex Fridman28:48

       Uh, that's a good line. That's a s-

   24. CICharles Isbell28:50

       (laughs)

   25. LFLex Fridman28:51

       (laughs) ... that's a

   26. (laughs)

   27. MLMichael Littman28:52

       Maybe that's a keeper. But, um-

   28. LFLex Fridman28:54

       (laughs) But do you, do you, do you find that resonates with you at all in, in how you think about talks and so on?

   29. CICharles Isbell28:59

       I would say a different length. It's not that-

   30. MLMichael Littman29:01

       I... No, not really. I don't-
8. 31:23 – 42:50

   Bell Labs

   1. MLMichael Littman31:23

   2. LFLex Fridman31:23

      Uh, so how did you actually meet, meet?

   3. CICharles Isbell31:26

      Yeah, Mike.

   4. MLMichael Littman31:26

      So my, the way I think about it is, 'cause we didn't do much research together...

   5. CICharles Isbell31:31

      At AT&T.

   6. MLMichael Littman31:32

      ... at AT&T.

   7. CICharles Isbell31:32

      No.

   8. MLMichael Littman31:32

      But, um, but then we all got laid off, so, so that was, that-

   9. LFLex Fridman31:36

      By the way-

   10. MLMichael Littman31:36

       ... su-

   11. LFLex Fridman31:37

       ... sorry to interrupt, but that was like one of the most magical places, historically speaking-

   12. CICharles Isbell31:41

       Yes.

   13. LFLex Fridman31:41

       ... uh...

   14. CICharles Isbell31:42

       They did not appreciate what they had.

   15. LFLex Fridman31:45

       And how do we, uh... (laughs) I feel like there's a profound lesson in there too. Uh, how do we get it, like what was, why was it so magical? Was it just a coincidence of history or is there something special about-

   16. MLMichael Littman31:56

       There were some really good managers and people who really believed in machine learning as, this is gonna be important. Um, let's get the, the people who are thinking about this in creative and, and insightful ways and put them in one place and stir.

   17. CICharles Isbell32:10

       Yeah, but even beyond that, right, it's, it was, it was Bell Labs at its heyday, and even when we were there, which I think was past its heyday.

   18. MLMichael Littman32:17

       And to be clear, he's gotten to be at Bell Labs. I never got to be at Bell Labs.

   19. CICharles Isbell32:21

       Yeah, I was-

   20. MLMichael Littman32:21

       I joined after that.

   21. CICharles Isbell32:22

       Yeah, I showed up in '91 as a grad student, so I was there for a long time. Um, every summer except for two-

   22. MLMichael Littman32:28

       So twice I worked for companies that had just stopped being Bell Labs. (laughs)

   23. CICharles Isbell32:31

       Right, Bell-

   24. MLMichael Littman32:32

       Bellcore and then AT&T Labs.

   25. CICharles Isbell32:33

       Right.

   26. LFLex Fridman32:33

       So Bell Labs was several locations or for the, for the research or is it one, like does that-

   27. MLMichael Littman32:38

       Definitely se-

   28. CICharles Isbell32:38

       ... 'cause I don't know if Jersey's- Oh, yeah.

   29. MLMichael Littman32:39

       ... involved somehow.

   30. CICharles Isbell32:40

       Oh, they're, they're all-
9. 42:50 – 53:01

   Teaching machine learning

   1. MLMichael Littman42:50

      when they were starting their online master's program, he knew that I was really excited about MOOCs and online teaching and he's like, "I have a plan." And I'm like, "Tell me your plan." He's like, "I can't tell you the plan yet," 'cause they were deep in, in negotiations between Georgia Tech and Udacity to make this happen and they didn't want it to leak. So, Charles would... kept teasing me about it but wouldn't tell me what was actually going on. And eventually it was announced and he said, "I would like you to teach the machine learning course with me." I'm like, "That can't possibly work." Um, but it was a great idea and it was, it was super fun. It was a lot of work to put together but it was, it was really great and...

   2. LFLex Fridman43:24

      Was that the first time you thought about... First of all, was it the first time you got seriously into teaching?

   3. MLMichael Littman43:30

      I mean, you know, I was a professor-

   4. LFLex Fridman43:30

      ... trying to get the timing right.

   5. MLMichael Littman43:32

      (laughs)

   6. LFLex Fridman43:32

      Oh so you, this was already, this was already-

   7. MLMichael Littman43:33

      And I had done-

   8. LFLex Fridman43:34

      ... after you jumped to, so like-

   9. MLMichael Littman43:36

      Yeah.

   10. LFLex Fridman43:36

       ... there's a little bit of jumping around in time.

   11. MLMichael Littman43:38

       Yeah, sorry about that.

   12. CICharles Isbell43:39

       That's a pretty big jump in timing.

   13. LFLex Fridman43:40

       So like, the MOOCs thing is, is, is less-

   14. MLMichael Littman43:42

       So Charles got to Georgia Tech and he... I mean, maybe Char- maybe this is a Charles story.

   15. CICharles Isbell43:45

       I think this was like, 2002.

   16. MLMichael Littman43:47

       He got to Georgia Tech in 2002.

   17. CICharles Isbell43:49

       Yeah.

   18. MLMichael Littman43:49

       And, um, but then, and, and worked on things like revamping the curriculum, the undergraduate curriculum so that it had some kind of semblance of modular structure because computer science was, at the time, moving from a fairly narrow specific set of topics to touching a lot of other parts of, uh, of, of intellectual life. And the curriculum was supposed to reflect that.

   19. CICharles Isbell44:10

       Mm-hmm.

   20. MLMichael Littman44:10

       And so, um, Charles played a big role in, in kind of redesigning that. And then the-

   21. CICharles Isbell44:16

       And for my, and for my, my labors I ended up the, uh, associate dean.

   22. MLMichael Littman44:20

       Right, he got to-

   23. CICharles Isbell44:21

       Somehow.

   24. MLMichael Littman44:21

       ... become associate dean of, in charge of educational stuff.

   25. CICharles Isbell44:24

       Well, it was under-

   26. LFLex Fridman44:25

       This should be a valuable lesson, if you're good at something-

   27. MLMichael Littman44:28

       (laughs)

   28. LFLex Fridman44:29

       ... uh, they will give you responsibility to do more of that thing.

   29. CICharles Isbell44:33

       Mm-hmm.

   30. LFLex Fridman44:33

       Well, until you-
10. 53:01 – 1:01:00

    Westworld and Ex Machina

    1. LFLex Fridman53:01

       What, what do you think about Westworld?

    2. MLMichael Littman53:02

       Two episodes in.

    3. LFLex Fridman53:03

       Did you?

    4. MLMichael Littman53:03

       So I could tell you-

    5. LFLex Fridman53:04

       Okay, well, yeah.

    6. MLMichael Littman53:04

       ... so far, I'm just guessing what's gonna happen next. It seems like bad things are gonna happen with the robots uprising. There's a lot of s-

    7. LFLex Fridman53:11

       Spoiler alert. (laughs)

    8. CICharles Isbell53:12

       So I, I have not, I have not s-

    9. MLMichael Littman53:13

       (laughs)

    10. CICharles Isbell53:13

        I mean, you know, I vaguely remember a movie existing so I assume it's, it's related to that. But-

    11. MLMichael Littman53:18

        That was more my time than your time, Charles.

    12. CICharles Isbell53:20

        That's right 'cause you're much older than I am. I think the important thing here is that, uh, it's narrative, right? It's all about telling a story, that's the whole driving thing. But the idea that they would give these reveries, that they would make pe- they would make them-

    13. MLMichael Littman53:31

        Let them remember-

    14. CICharles Isbell53:32

        ... remember the awful things that happened.

    15. MLMichael Littman53:35

        ... terrible things that happened.

    16. CICharles Isbell53:35

        Who could possibly think that was gonna... I, I get a, uh... I mean, I don't know. I have only seen the first two episodes or maybe the third one. I think I've only seen the first two.

    17. MLMichael Littman53:40

        You know what it was? Do you know what the problem is?

    18. CICharles Isbell53:42

        What?

    19. MLMichael Littman53:42

        That the robots were actually designed by Hannibal Lecter.

    20. CICharles Isbell53:45

        (laughs) That's true. They, they were. (laughs)

    21. MLMichael Littman53:47

        So like, wh- what do you think is gonna happen?

    22. CICharles Isbell53:49

        Anyways.

    23. MLMichael Littman53:49

        Bad things.

    24. CICharles Isbell53:50

        It's clear that things are happening and characters are being introduced and we don't yet know anything but, uh, still I was just struck by how it's all driven by narrative and story. And there's all these implied things like programming hap- the programming interface is talking to them about what's going on in their heads which is both, I mean, artistically it's probably useful to film it that way. But think about how it would work in real life, that just seems very cra- but there was... we, we saw on the second episode, there's a screen you could see things-

    25. MLMichael Littman54:15

        They were wearing like Google Glass.

    26. CICharles Isbell54:16

        ... that sort of state in the world. It was quite interesting to just kind of ask this question so far. I mean, I assume it veers off into never-never land at some point. But, uh-

    27. MLMichael Littman54:24

        So we don't know, we can't answer that question.

    28. LFLex Fridman54:25

        I'm also a f- a fan of a guy named Alex Garland. He's the director of Ex Machina.

    29. MLMichael Littman54:30

        Mm-hmm.

    30. CICharles Isbell54:30

        Mm-hmm.
11. 1:01:00 – 1:07:49

    Simulation

    1. CICharles Isbell1:01:00

    2. LFLex Fridman1:01:00

       You mentioned the Matrix.

    3. MLMichael Littman1:01:01

       Mm-hmm.

    4. LFLex Fridman1:01:02

       Do you think we're living in a simulation?

    5. MLMichael Littman1:01:04

       It does f- feel like a thought game more than a real scientific question.

    6. LFLex Fridman1:01:10

       Well, I'll tell you why, like, I think it's an interesting thought experiment, see what you think.

    7. MLMichael Littman1:01:14

       Okay.

    8. LFLex Fridman1:01:14

       From a computer science perspective, it's a good experiment of, how difficult would it be to create a sufficiently realistic world that us humans would enjoy being in? It, it, that, that's almost like a competition.

    9. MLMichael Littman1:01:27

       I mean, if we're living in a simulation, then I don't believe that we were put in the simulation. I believe that it, it's just physics playing out and we came out of that. Like, I don't, I don't, I don't think-

    10. LFLex Fridman1:01:39

        So, you think you have to build the universe and all the phonomena-

    11. MLMichael Littman1:01:41

        I think the, the universe itself, we can think of that as a simulation. And in fact, what... I try, sometimes I try to think about... To understand what it's like for a computer to-... start to think about the world, I try to think about (laughs) the world. Um, things like quantum mechanics where it doesn't feel very natural to me at all. Um, and it really strikes me as, I don't understand this thing that we're living in. It, it has... there's weird things happening in it that don't feel natural to me at all. Now, if you want to call that a s- the result of a simulator, okay, I'm fine with that. But like I don't-

    12. LFLex Fridman1:02:14

        But those are the bugs in the simulation.

    13. CICharles Isbell1:02:17

        There's the bugs. I mean, the interesting thing about-

    14. MLMichael Littman1:02:19

        (laughs)

    15. CICharles Isbell1:02:19

        ... the simulation is that it, it might have bugs. I mean, that, that's the thing that I... the, the-

    16. MLMichael Littman1:02:23

        But there wouldn't be bugs for the people in the simulation, they're just... that's just reality.

    17. CICharles Isbell1:02:27

        Unless you were-

    18. MLMichael Littman1:02:27

        They're not bugs.

    19. CICharles Isbell1:02:27

        ... aware enough to know that there was a bug. But I, I, I think-

    20. LFLex Fridman1:02:30

        Back to The Matrix.

    21. CICharles Isbell1:02:31

        Yeah. The way you put the question though-

    22. MLMichael Littman1:02:32

        I see. I don't think that we live in a, in a simulation created for us. I... okay, I would say that.

    23. CICharles Isbell1:02:36

        I think that's interesting, I've actually never thought about it that way. I mean, you... the way you asked the question though, could you create a world that is enough for us humans? It's an interestingly sort of self-referential question because the beings that created the simulation probably have not created a simulation that's realistic for them. But we're in the simulation, and so it's realistic for us. So we could create a simulation that is fine for the people in the simulation, as it were.

    24. MLMichael Littman1:03:02

        Right.

    25. CICharles Isbell1:03:02

        That would not necessarily be fine for us as the creators of the simulation.

    26. LFLex Fridman1:03:05

        But... well, you can, you can forget. I mean, when you go into the... if you play video games in virtual reality, you can... if... with some suspension of disbelief-

    27. MLMichael Littman1:03:14

        Yeah.

    28. LFLex Fridman1:03:14

        ... or, or whatever.

    29. CICharles Isbell1:03:15

        Yeah.

    30. LFLex Fridman1:03:15

        Uh-

Episode duration: 1:57:46