Charles Isbell and Michael Littman: Machine Learning and Education | Lex Fridman Podcast #148

Lex Fridman 0:00 The following is a conversation with Charles Isbell and Michael Littman. Charles is the Dean of the College of Computing at Georgia Tech, and Michael is a computer science professor at Brown University. I've spoken with each of them individually on this podcast, and since they are good friends in real life, we all thought it would be fun to have a conversation together. Quick mention of each sponsor, followed by some thoughts related to the episode. Thank you to Athletic Greens, the all-in-one drink that I start every day with to cover all my nutritional bases, Eight Sleep, a mattress that cools itself and gives me yet another reason to enjoy sleep, Masterclass, online courses from some of the most amazing humans in history, and Cash App, the app I use to send money to friends. Please check out these sponsors in the description to get a discount and to support this podcast. As a side note, let me say that, uh, having two guests on the podcast is an experiment that I've been meaning to do for a while, in particular because, uh, down the road, I would like to occasionally be a kind of moderator for debates between people that may disagree in some interesting ways. If you have suggestions for who you would like to see debate on this podcast, let me know. As with all experiments of this kind, it is a learning process. Both the video and the audio might need improvement. I realized, I think, I should probably do three or more cameras next time as opposed to just two, and also try different ways to mount the microphone for the third person. Also, after recording this intro, I'm going to have to go figure out the, uh, thumbnail for the video version of the podcast since I usually put the guest's head on the thumbnail and, uh, now there's two heads and two names to try to fit into the thumbnail. It's a kind of a bin packing problem which in, uh, theoretical computer science happens to be an NP-hard problem. Whatever I come up with, if you have better ideas for the thumbnail, let me know as well. And in general, I always welcome ideas how this thing can be improved. If you enjoy it, subscribe on YouTube, review it with five stars on Apple Podcast, follow on Spotify, support on Patreon, or connect with me on Twitter @lexfridman. And now, here's my conversation with Charles Isbell and Michael Littman.

Lex Fridman 2:27 You'll probably disagree about this question, but what is your biggest, would you say, disagreement about either something, uh, profound and very important or something completely not important at all?

Charles Isbell 2:39 I don't think we have any disagreements at all.

Michael Littman 2:41 Ah, I'm not sure that's true.

Charles Isbell 2:43 (laughs) We walked into that one, didn't we?

Lex Fridman 2:45 (laughs) Yeah. That's, that's pretty good.

Michael Littman 2:46 So, so one thing that you sometimes mention is that... and we did this one on air too, as it were, whether or not machine learning is computational statistics.

Charles Isbell 2:55 It's not.

Michael Littman 2:57 But it is.

Charles Isbell 2:57 Well, it's not. And in particular, and more importantly, it is not just computational statistics.

Lex Fridman 3:02 So what's missing in the picture? What-

Charles Isbell 3:04 All the rest of it.

Lex Fridman 3:05 (laughs)

Michael Littman 3:06 (laughs)

Lex Fridman 3:06 (laughs)

Charles Isbell 3:06 (laughs)

Michael Littman 3:06 What's missing? That which is missing. Oh, yes.

Charles Isbell 3:08 Because it's-

Michael Littman 3:09 Well, you can't be wrong now.

Charles Isbell 3:10 Well, it's not just the statistics. He doesn't even believe this. We've had this conversation before. If it were just the statistics then we would be happy with where we are. But it's not just the statistics.

Michael Littman 3:19 That's why it's computational statistics.

Charles Isbell 3:21 Or if it were just the computational ... kind of statistics.

Michael Littman 3:22 I agree that machine learning is not just statistics.

Charles Isbell 3:24 It is not just statistics, nor is it-

Michael Littman 3:25 We can agree on that.

Charles Isbell 3:26 ... nor is it just computational statistics.

Michael Littman 3:28 It's computational statistics.

Charles Isbell 3:29 It is computational. It's not just a...

Lex Fridman 3:30 What is the computational in computational statistics? Does this take us into the realm of computing?

Charles Isbell 3:35 It does, but I think perhaps the way I can get him to admit that, uh, he's wrong-

Michael Littman 3:39 That he's wrong. Yeah. (laughs)

Charles Isbell 3:40 ... uh, is that it's about rules.

Michael Littman 3:43 It's about rules.

Charles Isbell 3:44 It's about symbols. It's about all these other things ... that they add up to something.

Michael Littman 3:45 But the statistics is not about rules? I'm gonna say statistics is about rules.

Charles Isbell 3:48 Mm, but it's not just the statistics, right? It's not just a random variable that you choose and you have a probability distribution on it.

Michael Littman 3:52 I think you have a narrow view of statistics.

Charles Isbell 3:54 Okay. Well, then what would be the broad view of statistics that would still allow it to be statistics and not, say, history, uh, that would-

Michael Littman 4:01 (laughs)

Charles Isbell 4:01 ... make computational statistics okay?

Michael Littman 4:03 Well, okay. So I, I had my, my first sort of research mentor, a guy named Tom Landauer, taught me to do some statistics, right?

Charles Isbell 4:12 Sure.

Michael Littman 4:12 And, and I was annoyed all the time because the statistics would say that what I was doing was not statistically (laughs) significant.

Charles Isbell 4:18 Mm-hmm.

Michael Littman 4:19 And I was like, "But, but, but..." And basically what he said to me is, "Statistics is how you're gonna keep from lying to yourself." Which I thought was really deep.

Charles Isbell 4:29 It is a way to keep yourself honest in a particular way. I agree with that.

Michael Littman 4:34 Yeah. And so you're trying to find rules.

Charles Isbell 4:37 Mm...

Michael Littman 4:37 I'm just kinda bringing it back to rules.

Lex Fridman 4:38 Wait, wait, wait. C- c- uh, could you possibly try to define rules? This-

Michael Littman 4:44 Even regular statisticians, non-computational statisticians do spend some of their time evaluating rules, right? Applying statistics to try to understand does this, you know, is this... does this rule capture this, does this not capture that? Like-

Lex Fridman 4:55 You mean like hypothesis testing kinda thing?

Michael Littman 4:57 Sure. Yeah.

Lex Fridman 4:57 Or, like, uh, confidence intervals? Like, like h- like-

Michael Littman 5:00 I think more like hypothesis. Like I feel like the word statistic literally means like a summary, like a number that summarizes other numbers.

Lex Fridman 5:06 Right.

Michael Littman 5:06 But I think the field of statistics actually applies that idea to, to things like rules, to understand whether or not a rule is valid.

Charles Isbell 5:15 So is software engineering statistics?

Michael Littman 5:18 No.

Charles Isbell 5:18 Programming languages statistics?

Michael Littman 5:20 No.

Charles Isbell 5:21 Because I think there's a very... it's useful to think about a lot of what AI and machine learning is or certainly should be as software engineering, uh, as programming languages. Just the... uh, if... to put it in language that you might understand, uh, the hyperparameters beyond the problem itself, right?

Michael Littman 5:35 See, hyperparameters has too many syllables for me to understand.

Charles Isbell 5:37 The hyperparameters of, uh...

Michael Littman 5:39 (laughs) That's better.

Charles Isbell 5:40 ... that goes around it, right? It's the decisions you choose to make. It's the metrics you choose to use. It's the loss function you choose to focus on.

Michael Littman 5:45 So you, you wanna say the practice of machine learning is different than the practice of statistics. Like the things you have to worry about and how you worry about them are different, therefore they're different.

Charles Isbell 5:54 Right. At a very little. Uh, I mean, at the very least, it's... that, that much is true.

Michael Littman 5:59 Okay.

Charles Isbell 5:59 It doesn't mean that statistics, computational or otherwise, aren't important. I think they are. I mean, I do a lot of that, for example. But I think it goes beyond that. I think that we could think about game theory in terms of statistics, but I don't think it's very... as useful to do. I mean, the way I would think about it or a way I would think about it is this way...... chemistry is just physics.

Michael Littman 6:19 Mm.

Charles Isbell 6:20 But I don't think it's as useful to think about chemistry as being just physics. It's useful to think about it as chemistry. The level of obstruction really matters here.

Michael Littman 6:27 So, I think it is u- there are contexts in which it is useful-

Charles Isbell 6:30 Yes.

Michael Littman 6:30 ... to think of it that way, right?

Charles Isbell 6:31 No, that is absolutely-

Michael Littman 6:31 And so finding that connection is actually helpful. And I think that's when I, when I emphasize the computational statistics thing. I think, I think I want to befriend statistics and not absorb them.

Charles Isbell 6:41 Here's the, here's the, a way to think about it beyond what I just said, right? So, what would you say... And I want you to think back to a conversation we had a very long time

Charles Isbell 6:49 ago. What would you say is the difference between, say, the early 2000s, ICML and what we used to call NIPS, NeurIPS?

Michael Littman 6:57 Mm.

Charles Isbell 6:57 Is there a difference? A lot of the... Particularly in the machine learning that was done there?

Michael Littman 7:00 Mm.

Lex Fridman 7:01 ICML was around that long?

Charles Isbell 7:03 Oh, yeah.

Lex Fridman 7:03 So, ICLR is the new conference, newish?

Charles Isbell 7:06 Uh, yeah, I guess so.

Lex Fridman 7:07 And ICML was around in 2000?

Charles Isbell 7:10 Oh, ICML predates that.

Michael Littman 7:11 I, um, well, I think my most cited ICML paper is from '94.

Charles Isbell 7:15 Yeah. Michael knows this better than me because, of course, he's significantly older than I am. But the point is-

Lex Fridman 7:18 Yeah.

Charles Isbell 7:19 ... what is the difference-

Lex Fridman 7:20 Years.

Charles Isbell 7:20 ... what was the difference between ICML and NeurIPS in the late '90s, early 2000s?

Michael Littman 7:24 I don't know what everyone else's perspective would be, but I had a particular perspective at that time.

Charles Isbell 7:29 Which was?

Michael Littman 7:29 Which is, I felt like ICML was more of a g- of a computer science place-

Charles Isbell 7:33 Mm-hmm.

Michael Littman 7:33 ... and that N- NIPS/NeurIPS was more of an engineering place, like the kind of math that happened at the two places.

Charles Isbell 7:40 Oh, interesting.

Michael Littman 7:40 As a computer scientist, I felt more comfortable with the ICML math, and the NeurIPS people would say that that's because I'm dumb.

Charles Isbell 7:48 Mm-hmm.

Michael Littman 7:48 And that's such an engineering thing to say. So.

Charles Isbell 7:51 I agree with that part of it. I, so I do a little differently. We actually had a nice conversation with Tom Dietterich about this i- in public.

Michael Littman 7:57 On Twitter.

Charles Isbell 7:57 On Twitter, just a couple of days ago. I'd put it a little differently, which is that ICML was machine learning done by, uh, computer scientists, and, uh, NeurIPS was, uh, machine learning done by computer scientists trying to impress statisticians.

Lex Fridman 8:11 (laughs)

Michael Littman 8:12 (laughs)

Charles Isbell 8:13 Which was weird because it was the same people, at least by the time I started paying attention, but it just felt very, very different. And I think that that perspective of whether you're trying to impress the statisticians or you're trying to impress the programmers is actually very different and has real impact on, on what-

Michael Littman 8:27 Indications on, yeah, on the focus.

Charles Isbell 8:28 ... what you choose to worry about and what kind of, uh, outcomes you come to. So I think it really matters. I think computational statistics is a means to an end, it is not an end in some sense. Um, and I think that really matters here, in the same way that I don't think computer science is just engineering or just science or just math or whatever, but-

Michael Littman 8:43 Okay, so I'd have to now agree that now we agree on everything.

Charles Isbell 8:46 Yes, yes. The important thing here is that-

Michael Littman 8:49 (laughs)

Charles Isbell 8:49 ... you know, (laughs) my opinions may have changed, but not the fact that I am right, I think is what, what we just came to.

Michael Littman 8:54 Right. And my opinions may have changed and not the fact that I'm wrong.

Charles Isbell 8:57 That's right.

Lex Fridman 8:58 Uh, you lost me. I'm not even-

Charles Isbell 9:00 I think I lost myself there too. But anyway, so-

Michael Littman 9:02 All right, but we're back.

Charles Isbell 9:03 We're back.

Lex Fridman 9:03 (laughs) Uh-

Michael Littman 9:03 We... Ha- this happens to us sometimes.

Michael Littman 9:05 We're sorry.

Lex Fridman 9:06 How does neural networks change this, just to even linger on this topic, change this idea of w- statistics, th- how big s- of a pie statistics is within the machine learning thing? Like, 'cause it sounds like hyperparameters and also just the role of data, you know, there's pe- people are starting to use this terminology of software 2.0, which is like the act of programming as a, as a... Like, you're a designer in the hyperparameter space of neural networks, and you're also the collector and the organizer and the cleaner, uh, of the data. And that's part of the programming. Uh, how, so how did... On the NeurIPS versus ICML topic, what's the role of neural networks in redefining the size and the role of machine learning? Which-

Charles Isbell 9:57 Well, I can't, I can't wait to, to, to hear what Michael thinks about this, but, um, I would add one other thing.

Michael Littman 10:01 But you will.

Charles Isbell 10:01 (laughs) But I can't (laughs) ... That's true, I will. I'll force myself to. I think the-

Lex Fridman 10:05 (laughs)

Charles Isbell 10:05 ... the... There's one other thing I would add to your description, which is the kind of software engineering part of, what does it mean to debug, for example?

Lex Fridman 10:11 Oh, yeah.

Charles Isbell 10:11 But this is a difference between, uh, the kind of computational statistics view of machine learning and the c- computational view of machine learning, uh, which is, I think, one is worried about the equation as it were. And by the way, this is not a value judgment. I just think it's about perspective. But the kinda questions you would ask when you start asking yourself, "Well, what does it mean to program and develop and build the system?" is a very computer sciencey view of the problem. I mean, when... If you get on, uh, data science Twitter and econ Twitter, you actually hear this a lot with the, uh, you know, the economist and the data scientist complaining about the machine learning people. "Well, it's d- you know, it's just statistics, and I don't know why they don't, don't see this." But they're not even asking the same questions. They're not thinking about it as a kind of programming problem, and I think that that really matters, just asking this question. I actually think it's a little different from, uh, programming in hyperparameter space and, and sort of collecting the data. I, but I do think that that immersion really matters. So I'll give you a quick y- a quick example of the way I think about this. So, I teach machine learning. Michael and I have co-taught a machine learning class which has now reached, I don't know, 10,000 people at least, over the last several years, or somewhere there'sabouts. And my machine learning assignments are of this form. So, the super f- the first one is something like, "Implement these five algorithms," you know, K-NN, and S... You know, SVMs and boosting and decision trees and neural networks, and maybe that's it, I can't remember. And when I say implement, I mean steal the code. I am completely uninterested. You get zero points for getting the thing to work.

Michael Littman 11:38 Mm-hmm.

Charles Isbell 11:38 I don't want you spending your time worrying about, uh, getting the corner case right of, you know, what happens when you are trying to normalize distances and the points on the thing, and so you divide it by z- I'm not interested in that, right?

Michael Littman 11:49 Mm-hmm.

Charles Isbell 11:50 Steal the code. However, you're going to run those algorithms on two datasets. The datasets have to be interesting. What does it mean to be interesting? Well, a dataset's interesting if it reveals differences between algorithms, which presumably are all the same, because they can represent whatever they can represent. And two datasets are interesting together if they show different differences, as it were. And you have to analyze them. You have to justify their interestingness, and you have to analyze them in a whole bunch of ways. But all I care about is the data in your analysis, not the programming. And I occasionally end up in these long discussions with students... Well, I don't really. I copy and paste the things that I've said-

Michael Littman 12:24 (laughs)

Charles Isbell 12:24 ... the other 15,000 times it's come out, which is... They go, "But the only way to learn-"... really understand is to code them up-

Michael Littman 12:31 Yeah.

Charles Isbell 12:31 ... which is a very programmer software engineering view of the world. If you don't program it, you don't understand it. Which is, I, by the way, I think is wrong in a very specific way, but it is a way that you come to understand because then you have to wrestle with the algorithm.

Michael Littman 12:44 Mm-hmm.

Charles Isbell 12:44 But the thing about machine learning is it's not just sorting numbers, where in some sense the data doesn't matter. What matters is, well, does the algorithm work on these abstract things-

Michael Littman 12:51 Mm-hmm.

Charles Isbell 12:51 ... if one's less than the other? In machine learning, the data matters. It da-

Michael Littman 12:55 Mm-hmm.

Charles Isbell 12:55 ... it matters more than almost anything.

Michael Littman 12:56 Mm-hmm.

Charles Isbell 12:57 And s- not everything, but almost anything. And so as a result, you have to live with the data and don't get distracted by the algorithm per se. And I think that that focus on the data and what it can tell you and what question it's actually answering for you, as opposed to the question you thought you were asking, is a key and important thing about machine learning and is a way that computationalists as opposed to statisticians bring a particular view about how to think about the process. The statisticians, by contrast, bring, I- I think I'd be willing to say, a better view about the kind of formal math that's behind it-

Michael Littman 13:31 Mm-hmm.

Charles Isbell 13:31 ... and what an actual number ultimately is saying about the data. And those are both important, but they're also different.

Michael Littman 13:38 I didn't really think of it this way is to build intuition about the role of data, the different characteristics of data by having two data sets that are different and then reveal the differences in the differences.

Charles Isbell 13:50 Yeah.

Michael Littman 13:50 That's- that's a really fasc- that's a really interesting educational approach. The- the students love it, but not right away.

Charles Isbell 13:57 No, they love it at the end.

Michael Littman 13:57 They love it later. (laughs)

Charles Isbell 13:58 They love it at the end. Not at the beginning. (laughs)

Michael Littman 14:01 (laughs)

Charles Isbell 14:02 Well-

Michael Littman 14:02 Not even- not even immediately after.

Charles Isbell 14:04 (laughs)

Michael Littman 14:04 It takes a little while. I feel like there's a deep profound lesson about education there.

Charles Isbell 14:07 Yeah.

Michael Littman 14:08 That, uh, you can't listen to students about whether what you're doing is the right or the wrong thing.

Charles Isbell 14:16 Yeah. Well, as- as a wise, uh, Michael Littman once said to me about children, which I think applies to teaching, is you have to give them what they need without bending to their will. And students are like that. You have to figure out what they need. You're a curator. Your whole job is to curate and to present, because on their own, they're not gonna necessarily know where to search so you're providing pushes in some direction and learn space. Uh, and you have to give them what they need in a way that keeps them engaged enough so that they eventually discover what they want and they get the tools they need to go and learn other things off

Charles Isbell 14:49 of.

Michael Littman 14:50 What's your view... let me put on my Russian hat, which believes that life is suffering- I like Russian hats, by the way. (laughs) If you have one, I would like this. Those are ridiculous, yes.

Charles Isbell 14:59 (laughs)

Michael Littman 15:00 (laughs)

Charles Isbell 15:00 (laughs)

Michael Littman 15:00 But in a delightful way. But sure, of course. Uh- (laughs) what do you think is the role of, uh, we talked about balance a little bit.

Charles Isbell 15:08 Mm-hmm.

Michael Littman 15:08 What do you think is the role of hardship in education? Like, I think the biggest things I've learned, like, the w- what made me fall in love with math, for example, is by being bad at it until I got good at it. So like, like, struggling with a problem, which increased the level of joy I felt when I finally figured it out. And it- it always felt with me with teachers, especially modern discussions of education, how can we make education more fun, more engaging, more all those things? Well, from my perspective, it's like you're maybe missing the point that education, that life is suffering. (laughs) Education is supposed to be hard and that actually what increases the joy you feel when you actually learn something. Is that r- ridiculous? (laughs)

Charles Isbell 16:02 Oh, I get it.

Michael Littman 16:02 Do you like to see your students suffer?

Charles Isbell 16:04 (laughs)

Michael Littman 16:04 Okay. So th- this may be a point where we differ.

Charles Isbell 16:07 I suspect not.

Michael Littman 16:08 Okay.

Charles Isbell 16:08 But do go on.

Michael Littman 16:10 Well, what would your answer be?

Charles Isbell 16:11 I wanna hear you first.

Michael Littman 16:12 Okay. Well, I w- I was gonna not answer the question. (laughs)

Charles Isbell 16:15 (laughs) 'Cause you don't want the students to know you enjoy them suffering?

Michael Littman 16:18 I was gonna... No, no, no, no, no. I was- I was gonna say that there's... I think there's a d- a distinction that you can make in the kind of suffering, right? So, I think you can be in a mode where you're s- you're suffering in a hopeless way versus you're suffering in a hopeful way, right? Where you're like, you can see that if you- that you still have... you can still imagine getting to the end, right? And as long as people are in that mindset where they're struggling, but it's not a hopeless kind of struggling, that's p- that's productive. I think that's really helpful. But if struggling, like, if you- you break their will-

Charles Isbell 16:53 (laughs)

Michael Littman 16:53 ... if you leave them hopeless, no, that don't... I- sure, some people are gonna, whatever, lift themselves up by their bootstraps, but like, mostly you give up and certainly it takes the joy out of it, and you're not gonna spend a lot of time on something that brings you no joy. So it's a- it's- it is a bit of a delicate balance, right? You have to thwart people in a way that they still believe that there's a way through.

Charles Isbell 17:17 Right. So that's a- that, uh, we strongly agree actually. So I think s- well, first off, struggling and suffering aren't the same thing.

Michael Littman 17:23 Mm-hmm.

Charles Isbell 17:23 Right? One can-

Michael Littman 17:24 He's being poetic. (laughs)

Charles Isbell 17:25 Oh, no, no. I- I actually appreciate the poetry. And- and I- one of the reasons I appreciate it is that they are often the same thing and often quite different, right? So you can struggle without suffering. You can certainly suffer- (laughs) suffer- suffer pretty easily. You don't necessarily have to struggle to suffer. So I think that you want people to struggle, but that hope matters. The- you have to- they have to understand that they're gonna get through it on the other side. And it's very easy to confuse the two. Um, I actually think Brown University has a very just philosophically has a very different take on the relationship with their students, particularly undergrads, from say a place like Georgia Tech-

Michael Littman 17:59 Okay.

Charles Isbell 17:59 ... which is, uh-

Michael Littman 17:59 Which university's better?

Charles Isbell 18:01 Uh, well, I have my opinions on that.

Michael Littman 18:03 I mean, remember, Charles said, "It doesn't matter what the facts are. I'm always right."

Charles Isbell 18:07 The correct answer-

Michael Littman 18:08 Yeah, 'cause he's always right.

Charles Isbell 18:08 ... is that it doesn't matter. They're different. Um, but they're clearly-

Michael Littman 18:12 (laughs)

Charles Isbell 18:12 ... clearly answer to that.

Michael Littman 18:13 (laughs) He- he's, well, he was- he- he went to a school like h- the school where he is as an undergrad.

Charles Isbell 18:18 Yeah.

Michael Littman 18:18 I went to a school, specifically the same school, though it was- it changed a bit in the- in the intervening years.

Charles Isbell 18:23 What, Brown or Georgia Tech?

Michael Littman 18:24 No, I was talking about Georgia Tech. And I went-

Charles Isbell 18:25 Great exchange.

Michael Littman 18:26 Yeah. And I went to an undergrad place that's a lot like the place where I work now, and so it does seem like we're more familiar with these models. There's a similarity between Brown and Yale?

18:35 ... yeah.

Charles Isbell 18:35 There's a-

Michael Littman 18:36 I, I think that they're, I think they're quite similar, yeah.

Charles Isbell 18:38 And Duke.

Michael Littman 18:39 Duke has some similarities too, but it's got a little southern drawl.

Charles Isbell 18:42 You've kind of worked your w- you've sort of worked at universities that are like the places where you learned.

Michael Littman 18:49 Mm-hmm.

Charles Isbell 18:50 And the same would be true for me.

Michael Littman 18:52 Mm-hmm.

Charles Isbell 18:52 Are you uncomfortable, uh, venturing outside the box? Is that what you're saying?

Michael Littman 18:57 Mm-hmm.

Charles Isbell 18:57 Journeying out? Not what I'm saying, but it- (laughs)

Michael Littman 18:58 Yeah, Charles is definitely... He only goes to places that have institute in the name, right?

Charles Isbell 19:03 It has worked out that way. Well, academic places anyway. Well, no, I was a visiting scientist at UPenn. Or visiting, visiting something at UPenn. Oh, wow. I just, I just understood your joke.

Michael Littman 19:14 Which one? (laughs)

Charles Isbell 19:16 (laughs) Five minutes later.

Michael Littman 19:18 I like to s- (laughs) I like to set these sort of time bombs.

Charles Isbell 19:20 The institute is in the, uh, uh, that Charles only goes to places that have institute in the name.

Michael Littman 19:25 (laughs)

Charles Isbell 19:25 Yeah, so I guess Georgia, I, I forget that Georgia Tech is Georgia Institute of Technology.

Michael Littman 19:30 Institute of Technology, yeah.

Charles Isbell 19:30 The number of people-

Michael Littman 19:31 Massachusetts.

Charles Isbell 19:31 ... who refer to it as Georgia Tech University is large and incredibly irritating.

Michael Littman 19:36 (laughs)

Charles Isbell 19:36 It's one of the few things that genuinely gets under my skin.

Michael Littman 19:39 But, like, schools like Georgia Tech and MIT-

Charles Isbell 19:41 (laughs)

Michael Littman 19:41 ... have as part of the ethos, like there is... I wanna say there's a, there's an abbreviation that someone taught me, like IHTFP, something like that. Like, there's a, there's an, there's an expression which is basically, "I hate being here."

Charles Isbell 19:53 Mm-hmm.

Michael Littman 19:53 Which they say so proudly.

Charles Isbell 19:55 Yeah.

Michael Littman 19:55 And that is definitely not the ethos at Brown. Like Brown is, there's a little more pampering and empowerment and stuff.

Charles Isbell 20:02 Yes.

Michael Littman 20:02 And it's not like, "We're gonna crush you and you're gonna love it." So yeah, I think there's a, I think the ethoses are different.

Charles Isbell 20:08 Mm-hmm. That's interesting, yeah. We had drown proofing. What's that? Drown proof-

Michael Littman 20:12 Drown proof.

Charles Isbell 20:12 In order to graduate from Georgia Tech, this is a true thing, feel free to look it up.

Michael Littman 20:16 (laughs)

Charles Isbell 20:16 Uh, if you were to-

Michael Littman 20:17 A lot of schools have this, by the way.

Charles Isbell 20:19 No.

Michael Littman 20:19 Yeah.

Charles Isbell 20:19 Actually Georgia Tech was apparently the first.

Michael Littman 20:20 Brandeis has it.

Charles Isbell 20:22 Had it.

Michael Littman 20:23 Bandeis had it.

Charles Isbell 20:23 I feel like Georgia Tech was the first in a lot-

Michael Littman 20:25 Georg-

Charles Isbell 20:25 ... in a lot of, in a lot of ways.

Michael Littman 20:26 ... Georgia Tech was the first in a lot of things. It was, it was the first in a lot of things. Um, had the first master's degree in computer- First bumblebee mascot.

Charles Isbell 20:30 Stop that.

Michael Littman 20:31 (laughs)

Charles Isbell 20:32 Uh, first, uh, master's in computer science actually. It was pretty s-

Michael Littman 20:34 Right, online master's.

Charles Isbell 20:35 Well that too-

Michael Littman 20:36 Oh, okay.

Charles Isbell 20:36 ... but way back in the '60s. Um-

Michael Littman 20:37 Oh, really?

Charles Isbell 20:37 ... NSF grant. Yeah, yeah.

Michael Littman 20:38 Nice.

Charles Isbell 20:38 We had the first information and computer science master's degree in the country. Um-

Michael Littman 20:42 Wow.

Charles Isbell 20:42 ... but the, uh, Georgia Tech, it used to be the case in order to graduate from Georgia Tech, uh, you had to take a drown proofing class, where effectively they threw you in the water (laughs) and tied you up. And if you didn't drown you got to graduate. In fact-

Michael Littman 20:55 Tied you up?

Charles Isbell 20:55 I believe so.

Michael Littman 20:56 No.

Charles Isbell 20:56 You had, you basically had, there were certainly versions of it. But, I mean, luckily they ended it just before I had to graduate because otherwise I would never have graduated. (laughs)

Michael Littman 21:03 (laughs)

Charles Isbell 21:03 It wasn't gonna happen. Uh, I want to say '84, '83, somewhere around then they, they, they ended it. But, uh, yeah, used to have to prove you could tread water for some ridiculous amount of time or you couldn't-

Michael Littman 21:14 Two minutes.

Charles Isbell 21:14 ... you couldn't graduate. No, it was more than two minutes.

Michael Littman 21:15 I bet it was two minutes.

Charles Isbell 21:16 Okay, well, we'll look into- And it was in a bathtub. (laughs)

Michael Littman 21:18 (laughs) Yeah, right.

Charles Isbell 21:19 No, it was, it was in a pool.

Michael Littman 21:20 Yeah.

Charles Isbell 21:20 But it was a real thing. But that idea that, you know, push you-

Michael Littman 21:23 Fully clothed.

Charles Isbell 21:24 Uh, yeah, fully clothed, and you couldn't drown-

Michael Littman 21:25 I don't think, I bet, I bet it was that and not tied up. Because, like, who needs to learn how to swim when you're tied? Nobody.

Charles Isbell 21:30 No one.

Michael Littman 21:30 But who needs to learn when, to swim when you're actually falling into the water dressed? That's a real thing.

Charles Isbell 21:35 I think your facts are getting in the way with a good story.

Michael Littman 21:37 Oh, that's fair, that's fair. I didn't even-

Charles Isbell 21:39 (laughs) Thank you.

Michael Littman 21:39 All right. So they tie you up.

Charles Isbell 21:40 Sometimes the, the narrative matters more.

Michael Littman 21:41 (laughs)

Charles Isbell 21:42 But whatever it was, you had to, it was called drown proofing for a reason. The point of the story- Yeah. ... Michael, uh, is- Struggle. ... that it, it's, well no, but that's good. It does bring it back-

Michael Littman 21:51 Oh. That was, yeah, that's you're right.

Charles Isbell 21:52 ... it does bring it back to story. That's a part of what Georgia Tech has always been. And we struggle with that, by the way, uh, about what we want to be-

Michael Littman 21:58 And they let you.

Charles Isbell 21:58 ... particularly as, as things go. But you, you sort of, how much can you be pushed without breaking? And you come out of the other end stronger, right? There, there's this saying we used to have when I was an undergrad there which was, "Georgia Tech, building tomorrow the night before."

Michael Littman 22:13 (laughs)

Charles Isbell 22:14 Right?

Michael Littman 22:14 (laughs)

Charles Isbell 22:14 And it was just this kinda, kind of idea that-

Michael Littman 22:17 That's a good one.

Charles Isbell 22:17 ... you know, give me something impossible to do and I'll do it in a couple of days because that's what I just spent the last four or five or six, uh, years doing.

Michael Littman 22:24 This, that ethos definitely stuck to you. Having now done a number of projects with you, you definitely will do it the night before.

Charles Isbell 22:30 That's not entirely true. There is nothing wrong with waiting until the last minute. The secret is knowing when the last minute is. Right. That's brilli- that's brilliantly put.

Michael Littman 22:38 Yeah. That-

Charles Isbell 22:38 Yeah.

Michael Littman 22:39 ... that's, that is a definite Charles statement that I am trying not to embrace.

Charles Isbell 22:43 (laughs)

Michael Littman 22:44 (laughs)

Charles Isbell 22:44 And I appreciate that because you helped move my last minute up. (laughs)

Michael Littman 22:47 (laughs)

Charles Isbell 22:47 That's a social construct the way you converge together what the definition of last minute is.

Michael Littman 22:52 Right.

Charles Isbell 22:52 And we, we, we figure that out all together. In fact, MIT, you know, I'm, I'm sure a lot of universities have this, but MIT has, like, MIT time that-

Michael Littman 23:00 Yeah.

Charles Isbell 23:01 ... everyone is always agreed together that there's, there is such a concept and everyone just keeps showing up, like, 10 to 15 to 20, depending on the department, late to everything. So there's like a weird drift that happens. It's kind of fascinating. Yeah, we're five minutes, 10 to 15 minutes. We're five minutes. In fact, the classes will say, you know... Well, this is no longer true actually, but it used to be a class would start at 8:00, but actually it started at 8:05. Yeah. It ends at 9:00, actually it ends at 8:55. Yeah. Uh, everything's five minutes off and nobody expects anything to start until five minutes after the half hour, whatever it is.

Michael Littman 23:31 (laughs)

Charles Isbell 23:31 Um, it still exists.

Michael Littman 23:32 It hurts my head.

Charles Isbell 23:33 Well, let's rewind the clock back to the '50s and '60s when you guys met. (laughs) How did you... I'm just kidding. I don't... Uh, but what, can you tell the story of, of how you met? So you've-

Michael Littman 23:44 Mm-hmm.

Charles Isbell 23:44 ... so like the internet and the world kind of knows you as, as, as, as connected in some ways in terms of education, of teaching the world. That's, that's like the public facing thing. But how did you as human beings and as collaborators, uh, meet? I think there's two stories. One is how we met, and the other is how we-

Michael Littman 24:05 Fell in love.

Charles Isbell 24:05 ... got to know each other.

Michael Littman 24:06 (laughs)

Charles Isbell 24:07 I'm not gonna say fell, I'm not gonna say fell in love.

Michael Littman 24:08 (laughs)

Charles Isbell 24:08 I'm gonna say that we came to understand that we-

Michael Littman 24:11 Had some common-

Charles Isbell 24:12 Something.

Michael Littman 24:13 ... something. Yeah.

Charles Isbell 24:14 Yeah, there you go.

Michael Littman 24:14 It's funny, 'cause on the surface I think we're, we're different in a lot of ways. But there's something-

Charles Isbell 24:18 Yeah. I mean, now we complete each other's-

Michael Littman 24:19 ... just constant.

Charles Isbell 24:21 There you go.

Michael Littman 24:22 Afternoons.

Charles Isbell 24:23 (laughs)

Michael Littman 24:23 (laughs)

Charles Isbell 24:23 So, uh, I will tell the story of how we met and I'll let Michael tell the story of how we met.

Michael Littman 24:28 Okay. All right.

Charles Isbell 24:28 Okay. So here's how we met. Um, I was already at, at that point it was AT&T Labs. There's a long interesting story there. But anyway, I was there and, uh, Michael was coming to interview. He was a professor at Duke at the time but decided for reasons that he wanted to be in New Jersey.Uh, and so that would mean, uh, Bell Labs/A T&T Labs. Uh, and we were doing interview, interviews were very much like academic interviews. Uh, and so I had to be there. Uh, we all had to meet with him afterwards and so on, one-on-one. Uh, but it was obvious to me that he was gonna be hired. Like, no matter what, because everyone loved him. They were just talking about all the great stuff he did. "Oh, he did this great thing." And you had just won something at Triple AI, I think, or maybe you got 18 papers in Triple AI that year.

Michael Littman 25:08 But I got, I got the best paper award at Triple AI, for the crossword stuff.

Charles Isbell 25:11 For the crossword.

Lex Fridman 25:11 Right, exactly.

Michael Littman 25:11 Yeah.

Charles Isbell 25:12 So that had all happened and everyone was going on and on and on about it. Actually, Satinder was saying incredibly nice things about you.

Michael Littman 25:16 Really?

Charles Isbell 25:17 Yes. So-

Michael Littman 25:17 He can be very grumpy.

Charles Isbell 25:19 Yes. But he-

Michael Littman 25:20 So that's very... That's nice to hear.

Charles Isbell 25:21 He was grumpily saying very nice things about you.

Michael Littman 25:22 Oh, that's... That makes sense.

Charles Isbell 25:23 And... Yeah, it does make sense. So, you know, so he was gonna come, so why were we... Why was I meeting him? I had something else I had to do. I can't remember what it was.

Lex Fridman 25:29 Yeah.

Michael Littman 25:30 (laughs)

Charles Isbell 25:30 It probably involved comic books, but-

Michael Littman 25:31 So he remembers meeting me as inconveniencing his afternoon.

Charles Isbell 25:34 So he came... So eventually, he came to my office. I was in the middle of trying to do something, I can't remember what, and he came and he sat down. And for reasons that are purely accidental, despite what Michael thinks, uh-

Michael Littman 25:42 Yeah.

Charles Isbell 25:42 ... my desk at the time was set up in such a way that it had sort of an L-shape, and the chair on the outside was always lower than the, the chair that I was in. And, you know, the, the kind of point was to-

Michael Littman 25:52 The only reason I think that that was on purpose is because you told me it was on purpose.

Charles Isbell 25:56 I don't remember that. Anyway, the thing is that, you know, it kind of came-

Michael Littman 25:59 He had... His guest chair was really low so that he could-

Lex Fridman 26:01 Yeah.

Michael Littman 26:01 ... he could look down at everybody.

Charles Isbell 26:02 No, the idea was just to simply create a nice environment, that you were asking for a mortgage and I was gonna say no.

Lex Fridman 26:07 (laughs)

Michael Littman 26:07 (laughs)

Charles Isbell 26:07 That was the point. It was a very simple idea here.

Michael Littman 26:09 Yeah.

Charles Isbell 26:09 Anyway, so w- we sat there and we just talked for a little while, and I think he got the impression that I didn't like him.

Michael Littman 26:14 I-

Charles Isbell 26:14 Which wasn't true.

Michael Littman 26:15 ... strongly got that impression.

Charles Isbell 26:15 The talk was really good and he was full of enthusiasm.

Michael Littman 26:17 The talk, by the way, was terrible. And after, right after the talk, I said to my host, Michael Kearns, who ultimately was my boss-

Lex Fridman 26:23 I'm a huge fan. I'm a friend and a huge fan of Michael, yeah.

Michael Littman 26:25 Yeah. He is a remarkable person. Um, I ca-... I- I... After my talk, I went into the-

Charles Isbell 26:31 He has a table and he plays basketball.

Michael Littman 26:31 Um, I went-

Lex Fridman 26:32 Racquetball. He's good at everything.

Charles Isbell 26:33 No, basketball.

Lex Fridman 26:34 No, but basketball, racquetball too.

Charles Isbell 26:36 Squash. He's good at squash.

Lex Fridman 26:37 Squash, squash, squash. Not racquetball.

Charles Isbell 26:38 Yeah, squash, which is not... Racquetball, yes. Squash, no. And I hope you, you hear that, Michael.

Lex Fridman 26:44 (laughs) You mean in terms of-

Michael Littman 26:46 Of Michael Kearns.

Lex Fridman 26:46 ... as a game, not his skill level, because I'm pretty sure he's-

Charles Isbell 26:50 Both.

Michael Littman 26:50 (laughs) All right. The- there's some competitiveness there. But the point is that, that it was like the middle of the day. I had full day of interviews. Like, I met with people, but then in the middle of the day, I gave a job talk and then, um, and then there was going to be more interviews, but I, I pulled Michael aside and I said, "I think it's in both of our best interest if I just leave now." Because-

Lex Fridman 27:09 (laughs)

Michael Littman 27:09 ... that was so bad that it'd just be embarrassing if I have to talk to any more people. Like, you'll look bad for having invited me. Like, it's just... Let's just forget this ever happened.

Lex Fridman 27:19 (laughs)

Michael Littman 27:19 So I don't think the talk went well.

Charles Isbell 27:21 That's one of the most Michael Littman set of sentences I think I've ever heard. He did great. Or at least everyone knew he was great, so maybe it didn't matter. I was there. I remember the talk and I remember him being very much the way I remember him now, uh, on any given week. So it was good. And we met and we talked about stuff. He thinks I didn't like him, but-

Michael Littman 27:37 Because he was so grumpy.

Lex Fridman 27:39 Must've been the chair thing.

Michael Littman 27:40 The chair thing and the low voice, I think. The... Like, he obviously wasn't happy with you.

Lex Fridman 27:43 And that, like, that, like, slight, like, skeptical look.

Michael Littman 27:47 Yes.

Lex Fridman 27:47 Like, "Are you sure?"

Michael Littman 27:48 "I have no idea what you're talking about."

Lex Fridman 27:50 (laughs)

Charles Isbell 27:50 Well, I probably didn't have any idea what you were talking about.

Lex Fridman 27:52 (laughs)

Charles Isbell 27:53 Anyway, I liked him. He was-

Lex Fridman 27:54 Yeah.

Michael Littman 27:54 He asked me questions, I answered questions. I felt bad about myself. It was a normal day.

Charles Isbell 27:59 It was.

Lex Fridman 27:59 (laughs)

Charles Isbell 27:59 It (laughs) was a normal day.

Michael Littman 28:00 (laughs)

Charles Isbell 28:00 And then he left.

Lex Fridman 28:01 And then he left and that's how you-

Michael Littman 28:02 But that's how we met.

Lex Fridman 28:03 Can we take a slight-

Michael Littman 28:03 And then I got hired and I was

Michael Littman 28:05 in the group.

Lex Fridman 28:05 Can we take a slight tangent on that-

Michael Littman 28:07 Sure.

Lex Fridman 28:07 ... on this topic of... It sounds like, uh, maybe you could speak to the bigger picture. It sounds like you're quite self-critical.

Michael Littman 28:15 Who, Charles?

Lex Fridman 28:15 No, you.

Michael Littman 28:16 Oh.

Lex Fridman 28:17 Okay, so-

Michael Littman 28:17 I think I can, I can do better. I can do better. I'll, I'll... Tr- try me again. I'll, I'll, I'll do better.

Lex Fridman 28:21 (laughs)

Charles Isbell 28:21 (laughs)

Lex Fridman 28:21 (laughs)

Michael Littman 28:21 Be so self-critical. I won't, I won't, I won't.

Lex Fridman 28:23 Yeah, that, that was like a, like a three out of 10 response, so-

Michael Littman 28:26 (laughs)

Lex Fridman 28:27 Uh, so let's, let's try to work it up to five and six. Uh, you know, I remember, uh, Marvin Minsky said, uh, on, uh, on a video interview something that the key to success in academic research is to hate everything you do.

Charles Isbell 28:40 Hmm.

Michael Littman 28:41 Oh.

Lex Fridman 28:42 Uh, (laughs) for some reason-

Michael Littman 28:44 I think I followed that because I hate everything he's done.

Lex Fridman 28:47 (laughs)

Charles Isbell 28:48 (laughs)

Lex Fridman 28:48 Uh, that's a good line. That's a s-

Charles Isbell 28:50 (laughs)

Lex Fridman 28:51 (laughs) ... that's a

28:52 (laughs)

Michael Littman 28:52 Maybe that's a keeper. But, um-

Lex Fridman 28:54 (laughs) But do you, do you, do you find that resonates with you at all in, in how you think about talks and so on?

Charles Isbell 28:59 I would say a different length. It's not that-

Michael Littman 29:01 I... No, not really. I don't-

Charles Isbell 29:02 That's such an MIT view of the world though.

Lex Fridman 29:04 (laughs)

Charles Isbell 29:04 So I remember, I, I remember talking about this when, uh... As a student, you know, you were basically told, uh, I wo- I will clean it up for the purposes of the podcast.

Lex Fridman 29:12 (laughs)

Michael Littman 29:12 (laughs)

Charles Isbell 29:12 Um, uh, "My work is crap, my work is crap, my work is crap, my work is crap." Then you, like, go to a conference or something and like, "Everybody else's work is crap, everybody else's work is crap." And you feel better and better about it-

Lex Fridman 29:21 Yeah.

Charles Isbell 29:21 ... uh, relatively speaking.

Lex Fridman 29:23 Yeah.

Charles Isbell 29:23 And then you sort of keep working on it. I, I don't hate my work. I've n-

Michael Littman 29:26 That resonates with me.

Charles Isbell 29:27 Yes. I've never hated my work, but I have... I have been dissatisfied with it. And I think being dissatisfied... Being okay with the fact that you've taken a positive step, the derivative's positive-

Lex Fridman 29:39 Mm-hmm.

Charles Isbell 29:40 ... maybe even the second derivative's positive. That's important because that's a part of the, the hope, right? But you have to... But, but I haven't gotten there yet. If, if that's not there, that I haven't gotten there yet, then, you know, it's hard to, it's hard to move forward, I think. So I buy that, which is a little different from hating everything that you do.

Michael Littman 29:56 Yeah. I mean, there's, there's things that I've done that I like better than I like myself. So it's a separating me from the work, essentially. So I think I d- I am very critical of myself, but sometimes the work I'm really excited about, and sometimes I think it, it, it's kind of good.

Charles Isbell 30:10 It doesn't happen right away. So I, I found the work that I've liked that I've done, most of it, I liked it in retrospect more when I was far away-

Michael Littman 30:19 Oh, interesting.

Charles Isbell 30:20 ... from it in time.

Michael Littman 30:21 I have to be fairly excited about it to get done.

Charles Isbell 30:24 No, excited at the time, but then happy with the result or what... But years later, or even I might look back, "You know what? That actually turned out to matter."

Michael Littman 30:30 That wasn't terrible. Yeah. Yeah.

Charles Isbell 30:31 "That turned out to matter." Or, "Oh gosh, it turns out I've been thinking about that... It's actually influenced all the work that I've done since, without realizing it."

Michael Littman 30:37 "Boy, that guy was smart."

Charles Isbell 30:39 Yeah. "That guy had a future."

Michael Littman 30:41 (laughs)

Charles Isbell 30:41 Yeah. I... Yeah. I think there's a-

Michael Littman 30:43 "He's going places."

Lex Fridman 30:44 (laughs)

Charles Isbell 30:44 (laughs) I think there... So, yeah. So I, I think there's something to it. I think there's something to the idea you've got to, you know, hate what you do. But it's not quite hate, it's just being unsatisfied.

Michael Littman 30:52 Mm-hmm.

Charles Isbell 30:52 And different people motivate themselves differently. I don't happen to motivate myself with self-loathing. I happen to motivate myself-

Michael Littman 30:57 He really doesn't.

Charles Isbell 30:57 ... with something else.

Lex Fridman 30:58 So you're able to sit back and be proud-... of, in retrospect of the work you've done.

Charles Isbell 31:04 Well, and it's easier when you can connect it with other people, because then you can be proud of them.

Michael Littman 31:08 Mm-hmm.

Lex Fridman 31:09 Proud of the people, yeah.

Charles Isbell 31:10 And then the question-

Lex Fridman 31:10 And then you can still safely hate yourself-

Charles Isbell 31:12 Yeah, that's right. (laughs)

Lex Fridman 31:12 ... privately. (laughs)

Michael Littman 31:13 (laughs)

Charles Isbell 31:13 It's win-win, Michael. Or at least win-lose, which is what you're looking for.

Michael Littman 31:18 (laughs)

Lex Fridman 31:18 Oh, wow. There's so many-

Michael Littman 31:20 Yeah, there's-

Lex Fridman 31:20 ... brilliant minds in this. (laughs)

Michael Littman 31:22 (laughs) There's levels.

Lex Fridman 31:23 Uh, so how did you actually meet, meet?

Charles Isbell 31:26 Yeah, Mike.

Michael Littman 31:26 So my, the way I think about it is, 'cause we didn't do much research together...

Charles Isbell 31:31 At AT&T.

Michael Littman 31:32 ... at AT&T.

Charles Isbell 31:32 No.

Michael Littman 31:32 But, um, but then we all got laid off, so, so that was, that-

Lex Fridman 31:36 By the way-

Michael Littman 31:36 ... su-

Lex Fridman 31:37 ... sorry to interrupt, but that was like one of the most magical places, historically speaking-

Charles Isbell 31:41 Yes.

Lex Fridman 31:41 ... uh...

Charles Isbell 31:42 They did not appreciate what they had.

Lex Fridman 31:45 And how do we, uh... (laughs) I feel like there's a profound lesson in there too. Uh, how do we get it, like what was, why was it so magical? Was it just a coincidence of history or is there something special about-

Michael Littman 31:56 There were some really good managers and people who really believed in machine learning as, this is gonna be important. Um, let's get the, the people who are thinking about this in creative and, and insightful ways and put them in one place and stir.

Charles Isbell 32:10 Yeah, but even beyond that, right, it's, it was, it was Bell Labs at its heyday, and even when we were there, which I think was past its heyday.

Michael Littman 32:17 And to be clear, he's gotten to be at Bell Labs. I never got to be at Bell Labs.

Charles Isbell 32:21 Yeah, I was-

Michael Littman 32:21 I joined after that.

Charles Isbell 32:22 Yeah, I showed up in '91 as a grad student, so I was there for a long time. Um, every summer except for two-

Michael Littman 32:28 So twice I worked for companies that had just stopped being Bell Labs. (laughs)

Charles Isbell 32:31 Right, Bell-

Michael Littman 32:32 Bellcore and then AT&T Labs.

Charles Isbell 32:33 Right.

Lex Fridman 32:33 So Bell Labs was several locations or for the, for the research or is it one, like does that-

Michael Littman 32:38 Definitely se-

Charles Isbell 32:38 ... 'cause I don't know if Jersey's- Oh, yeah.

Michael Littman 32:39 ... involved somehow.

Charles Isbell 32:40 Oh, they're, they're all-

Michael Littman 32:41 They're all in Jersey.

Charles Isbell 32:41 Yeah. They're all over the place, but-

Michael Littman 32:42 But they were in a couple places in Jersey.

Charles Isbell 32:43 Uh, Murray Hill was-

Michael Littman 32:45 The big one.

Charles Isbell 32:45 ... the Bell Labs place. Um-

Michael Littman 32:47 So you, you had, you had an office in Murray Hill at one point in your career?

Charles Isbell 32:51 Yeah, I, and I-

Michael Littman 32:52 And I never did.

Charles Isbell 32:52 ... I, I played ultimate frisbee on the cricket pitch at Bell Labs at Murray Hill. Uh, and then it became AT&T Labs when it split off with Luce- during what we called tri-vestiture as opposed to di-

Lex Fridman 33:01 Are you better than Michael Kearns at ultimate frisbee?

Michael Littman 33:03 Yeah.

Charles Isbell 33:03 Oh, yeah.

Lex Fridman 33:04 Okay.

Michael Littman 33:05 But I think that one's not boasting. I think that, I think Charles plays a lot of ultimate and I don't think Mike, Mike does.

Charles Isbell 33:10 No. I was... Yes, but, but that wasn't the point. The point is yes.

Michael Littman 33:12 Sorry. (laughs)

Charles Isbell 33:13 I'm finally better-

Michael Littman 33:13 Oh, yeah, yes, yeah, so-

Charles Isbell 33:13 ... at the end of the-

Michael Littman 33:13 ... sorry. Char- okay, I have played on a w- a championship winning ultimate frisbee team or whatever, ultimate team with Charles, so I know how good he is. He's really good.

Charles Isbell 33:23 How good I was anyway when I was younger. But the thing is-

Michael Littman 33:25 I know how young he was when he was younger.

Charles Isbell 33:26 (laughs) That's true. That is true, but-

Michael Littman 33:27 So much younger than now, you hit-

Lex Fridman 33:28 He's older now.

Michael Littman 33:29 He's older now.

Charles Isbell 33:29 Yes, I am older. Michael is a much, was a much better basketball player than I was.

Michael Littman 33:33 Michael Kearns?

Charles Isbell 33:34 Yes. No, not Michael. (laughs) I'm just-

Michael Littman 33:36 I've never-

Charles Isbell 33:36 Let's be very clear about that.

Michael Littman 33:37 To be clear, I have not played basketball with you.

Charles Isbell 33:38 This is true.

Michael Littman 33:38 So you don't know how terrible I am, but you have a probably pretty good guess.

Charles Isbell 33:42 That you're not as good as Michael Kearns.

Michael Littman 33:44 He's tall and I don't know-

Charles Isbell 33:45 And he cared about it.

Michael Littman 33:46 ... athletic.

Charles Isbell 33:46 He's very athletic. He's very good. It's, I-

Michael Littman 33:48 And probably competitive.

Charles Isbell 33:48 I love hanging out, I love hanging out with Michael.

Michael Littman 33:50 Okay.

Charles Isbell 33:50 Anyway, but we were talking about something else, although I no longer remember what it was. What were we talking about?

Lex Fridman 33:53 Calling you Matt, Matt-

Charles Isbell 33:54 Oh, Bell Labs.

Lex Fridman 33:54 Oh, Bell Labs.

Charles Isbell 33:55 But also labs. So, so, uh, this was kind of cool about what was magical about it. The first thing you have to know is that Bell Labs was an arm of the government, right? Because AT&T was an arm of the government. It was a monopoly, uh, and, you know, every month you paid a, a little thing on your phone bill, which turned out was a tax for like all the research (laughs) that Bell Labs was doing. And, you know, they invented transistors and the laser and whatever else it is they did.

Michael Littman 34:17 The Big Bang or whatever the ac-

Charles Isbell 34:18 Yeah.

Michael Littman 34:18 ... the ra- cosmic background radiation.

Charles Isbell 34:20 Yeah, they did all that stuff. They had some amazing stuff with directional microphones, by the way. I got to go in this room, um, where they, they, they had all these panels and everything, and we would talk at one another... and he'd move some panels around and then he would have me step two steps to the left, and I couldn't hear a thing he was saying because nothing was bouncing off the walls.

Michael Littman 34:37 Mm-hmm.

Charles Isbell 34:37 And then he would shut it all down and you could hear your heartbeat.

Michael Littman 34:40 Yeah.

Charles Isbell 34:40 Which is deeply disturbing to hear your heartbeat. You can feel it. I mean, you can feel it now, there's just so much all this sort of noise around. Anyway, Bell Labs was about pure research. It was a university, in some sense the purest sense of a university, but without students. So it was all the faculty working with one another, and students would come in to learn. They would come in for three or four months, you know, during the summer and they would go away. But it was just this kind of wonderful experience. I could walk out my door, in fact I would often have to walk out my door, and deal with Rich Sutton and Michael Kearns yelling at each other about whatever it is they were yelling about, um, the proper way to prove something or another. And I could just do that. And Dave McAlister and even R- and Peter Stone, and, and all of these other people-

Michael Littman 35:19 Satinder.

Charles Isbell 35:19 ... including S- Satinder and then e- eventually Michael, and it was just a place where you could think thoughts and it was okay because so long as once every 25 years or so, somebody invented a transistor, it paid for everything else. You could afford to take the risk. And then when that all went away, it became harder and harder and harder to justify it as far as the folks who were very far away were concerned, and there was such a fast turnaround among middle management, um, on the AT&T side, that you never had a chance to really build a relationship. At least people like us didn't have a chance to, to, to build relationships. So when the diaspora happened, um, it was amazing, right?

Michael Littman 35:55 (laughs) Yeah.

Charles Isbell 35:55 Everybody left and I, I think everybody ended up at a, at a great place and made, made a huge, made a, continue to do really good work with, with machine learning. But it was a wonderful place and people will ask me, you know, "What's the best job you, you've ever had?" And as a professor anyway, the answer that I, I would give is, um, well, probably Bell Labs in some very real sense and I will never have a job like that again because Bell Labs doesn't exist anymore. And, you know, Microsoft Research is great and Google does good stuff and you can pick IBM or you could tell who you want to, but Bell Labs was magical. It was around for... it was an important time and it represents a, a, a high water mark in, in basic research in the US.

Lex Fridman 36:32 Is there something you could say about the physical proximity and the chance collisions? Like-

Charles Isbell 36:36 Oh.

Lex Fridman 36:37 ... we live in this time of the pandemic where everyone is maybe trying to see the silver lining and accepting the remote nature of things. Is, is there... one of the things that people like faculty, uh, that, that I talk to miss is the, the procrastination.

Charles Isbell 36:55 Mm-hmm.

Lex Fridman 36:55 (laughs) Like the chance to... like everything is about meetings that are supposed to be... like there's not a chance to just, uh, you know, talk about comic book or whatever, like go into discussion that's totally pointless.

Charles Isbell 37:07 So it's funny you say this because that's how we met, Matt.It was exactly that. So, I'll let Michael say that, but I'll just add one thing, which is just that, uh, you know, research is a social process.

Lex Fridman 37:16 Yeah.

Charles Isbell 37:16 And it helps to have random social interactions, even if they don't feel social at the time. That's how you get things done. Having... One of the great things about, um, the AI Lab when I was there, I don't quite know what it looks like now once they moved buildings, but we had entire walls that were whiteboards and people would just get up there and they would just write and people would walk up and you'd have arguments and you'd explain things to one another, and you got so much out of the freedom to do that. You had to be okay with people challenging every fricking word you said, which I would-

Lex Fridman 37:45 Mm-hmm.

Charles Isbell 37:45 ... sometimes find deeply irritating. But most of the time it was, it was quite useful. But the sort of pointlessness and the interaction was, in some sense, the point, at least for me.

Lex Fridman 37:54 Yeah, I mean, you... I, I think offline yesterday I mentioned Josh Tenenbaum and he's very much... He pu- he's, oh man, he's such an inspiration in the, in the childlike way that he-

Charles Isbell 38:06 Mm.

Lex Fridman 38:06 ... pulls you in on any topic. It doesn't even have to be about machine learning. It could... or, or the, the brain. He'll just pull you in to a closest writable surface-

Charles Isbell 38:15 (laughs)

Lex Fridman 38:16 ... which is, uh, still you can find whiteboards at MIT everywhere.

Charles Isbell 38:19 Mm-hmm.

Lex Fridman 38:19 And, and just like, uh, like basically cancel all meetings and talk for a couple of hours about some, some, uh, aimless thing and it, it feels like the whole world, the time-space continuum kind of warps and that becomes the most important thing. And then it just-

Charles Isbell 38:33 It's so true.

Lex Fridman 38:34 ... it, it's amaz- it's definitely something worth missing in this, in this world where everything's remote. There's some magic to the physical presence.

Charles Isbell 38:43 Whenever I wonder myself whether MIT really is as great as I remember it, I just go talk to Josh.

Lex Fridman 38:48 Yeah. You know, that's funny is there's a few people in this world that carry the best of what particular institutions stand for, right? Mm.

Charles Isbell 38:56 And it's, uh-

Lex Fridman 38:57 There's Josh. I mean, I, I don't... I... My guess is he's unaware of this. That's the point.

Charles Isbell 39:02 Yeah.

Lex Fridman 39:02 That the masters are not aware of their mastery. So-

Charles Isbell 39:07 How did you meet?

Lex Fridman 39:09 (laughs)

Michael Littman 39:09 (laughs)

Lex Fridman 39:09 Yes. But, but first a tangent. No.

Charles Isbell 39:12 (laughs)

Lex Fridman 39:13 (laughs) The how did you meet me?

Michael Littman 39:14 So, I'm not sure what you were thinking of, but I... My-

Lex Fridman 39:17 (laughs)

Michael Littman 39:17 When, when it started to dawn on me that maybe we had a longer term bond was after we all got laid off and you had decided at that point that there... That we were... We were still paid. We were given an opportunity to, like, do a job search and kind of make a transition. But it was clear that we were done. And I would go to my office to work and you would go to my office to keep me from working.

Charles Isbell 39:40 Yeah.

Michael Littman 39:41 That was, that was my recollection of it. Um, you had decided that there was no, really no point in working for the company 'cause the com- our relationship with the company was, was done.

Charles Isbell 39:49 Yeah, but remember I felt that way beforehand.

Michael Littman 39:51 (laughs)

Charles Isbell 39:51 It wasn't about the company, it was about the set of people there doing really cool things and it always, always been that way. But we were working on something together.

Michael Littman 39:57 Oh, yeah, yeah, yeah. That's right. So, at the very end we all got laid off, but then our boss came, our boss's boss came to us.

Charles Isbell 40:04 Our boss's boss.

Michael Littman 40:04 'Cause our boss was Michael Kearns and he had jumped ship brilliantly, like perfect timing, like things... Like, right before the ship was about to sink he was like, "Gotta go!" (laughs) And, and, and landed perfectly because Michael Kearns.

Charles Isbell 40:18 Because Michael Kearns.

Michael Littman 40:18 And, um, the... le- leaving the rest of us to go like, "This is fine." And then it was clear that it wasn't fine and, and we were all toast. So, we had this sort of long period of time, but th- then our boss figured out, "Okay, wait. Maybe we can save a couple of these people if we can have them do something really useful."

Lex Fridman 40:36 Mm.

Michael Littman 40:37 And, uh, the useful thing was we were gonna make a... basically an automated assistant that could help you with your calendar. You could, like, tell it things and it would, it would respond appropriately. It would just kind of integrate across all sorts of your personal information. And so me and Charles and Peter Stone-

Lex Fridman 40:56 Mm-hmm.

Michael Littman 40:56 ... um, were this... were set up as the crack team to actually solve this problem. Uh, other people maybe were too theoretical that they thought and, and... but we could actually get something done. So, we sat down to get something done and there wasn't time and it wouldn't have saved us anyway.

Lex Fridman 41:09 Mm-hmm.

Michael Littman 41:10 And so it all kind of went downhill. But the interesting, I think, coda to that is that our boss's boss is a guy named Ron Brockman and he... when he left AT&T 'cause we were all laid off, he went to DARPA, uh, started up a program there that became Halo which is the program from which Siri sprung, which is a digital assistant that helps you with your calendar and a bunch of other things. Um, it really, you know, in some ways got its start with me and Charles and Peter trying to implement this vision that Ron Brockman had that he ultimately got implemented through his role at DARPA. So, when I'm trying to feel less bad about having been laid off from what is possibly the greatest job of all time, um, I think about, "Well, we kind of helped birth Siri." That's something.

Charles Isbell 42:01 Mm. And then he did other things too. But the... We got to spend a lot of time in his office and-

Michael Littman 42:07 (laughs)

Charles Isbell 42:07 ... talk about

Lex Fridman 42:07 lots of things.

Michael Littman 42:07 We got to spend a lot of time in my office. Yeah.

Charles Isbell 42:10 Yeah.

Michael Littman 42:10 Yeah. Yeah. And so, uh, so then we went on our merry way. Everyone went to different places, um, Charles landed at Georgia Tech which was, uh, what he always dreamed he would do and so, um, that worked out well.

Charles Isbell 42:22 Yeah.

Michael Littman 42:23 Um, I came up with a saying at the time which is, "Luck favors the Charles."

Charles Isbell 42:27 (laughs)

Michael Littman 42:28 It's kind of like, "Luck favors the prepared."

Charles Isbell 42:30 Yeah, yeah.

Michael Littman 42:30 Um, but Charles, like, like he'd wish something and then it would basically happen just the way he wanted. It was, it was inspirational to see things go that way.

Charles Isbell 42:38 Things worked out.

Michael Littman 42:39 And we stayed in touch and then, um... I think it really helped when you were working on... I mean, you'd kept me in the loop for things like Threads and-

Lex Fridman 42:48 Mm-hmm.

Michael Littman 42:48 ... and the work that you were doing at Georgia Tech.

Lex Fridman 42:49 Mm-hmm.

Michael Littman 42:49 But then

Michael Littman 42:50 when they were starting their online master's program, he knew that I was really excited about MOOCs and online teaching and he's like, "I have a plan." And I'm like, "Tell me your plan." He's like, "I can't tell you the plan yet," 'cause they were deep in, in negotiations between Georgia Tech and Udacity to make this happen and they didn't want it to leak. So, Charles would... kept teasing me about it but wouldn't tell me what was actually going on. And eventually it was announced and he said, "I would like you to teach the machine learning course with me." I'm like, "That can't possibly work." Um, but it was a great idea and it was, it was super fun. It was a lot of work to put together but it was, it was really great and...

Lex Fridman 43:24 Was that the first time you thought about... First of all, was it the first time you got seriously into teaching?

Michael Littman 43:30 I mean, you know, I was a professor-

Lex Fridman 43:30 ... trying to get the timing right.

Michael Littman 43:32 (laughs)

Lex Fridman 43:32 Oh so you, this was already, this was already-

Michael Littman 43:33 And I had done-

Lex Fridman 43:34 ... after you jumped to, so like-

Michael Littman 43:36 Yeah.

Lex Fridman 43:36 ... there's a little bit of jumping around in time.

Michael Littman 43:38 Yeah, sorry about that.

Charles Isbell 43:39 That's a pretty big jump in timing.

Lex Fridman 43:40 So like, the MOOCs thing is, is, is less-

Michael Littman 43:42 So Charles got to Georgia Tech and he... I mean, maybe Char- maybe this is a Charles story.

Charles Isbell 43:45 I think this was like, 2002.

Michael Littman 43:47 He got to Georgia Tech in 2002.

Charles Isbell 43:49 Yeah.

Michael Littman 43:49 And, um, but then, and, and worked on things like revamping the curriculum, the undergraduate curriculum so that it had some kind of semblance of modular structure because computer science was, at the time, moving from a fairly narrow specific set of topics to touching a lot of other parts of, uh, of, of intellectual life. And the curriculum was supposed to reflect that.

Charles Isbell 44:10 Mm-hmm.

Michael Littman 44:10 And so, um, Charles played a big role in, in kind of redesigning that. And then the-

Charles Isbell 44:16 And for my, and for my, my labors I ended up the, uh, associate dean.

Michael Littman 44:20 Right, he got to-

Charles Isbell 44:21 Somehow.

Michael Littman 44:21 ... become associate dean of, in charge of educational stuff.

Charles Isbell 44:24 Well, it was under-

Lex Fridman 44:25 This should be a valuable lesson, if you're good at something-

Michael Littman 44:28 (laughs)

Lex Fridman 44:29 ... uh, they will give you responsibility to do more of that thing.

Charles Isbell 44:33 Mm-hmm.

Lex Fridman 44:33 Well, until you-

Michael Littman 44:34 Don't show competence.

Lex Fridman 44:35 Don't show competence if you-

Charles Isbell 44:37 Well, you know what they say.

Lex Fridman 44:37 ... want responsibility. (laughs)

Charles Isbell 44:38 Here's what they say.

Lex Fridman 44:39 Yeah.

Charles Isbell 44:40 Uh, the reward for good work is more work.

Lex Fridman 44:42 Yeah.

Charles Isbell 44:43 The reward for bad work is less work, which I don't know, depending upon what you're trying to do that week. One of those is better than the other.

Lex Fridman 44:51 Well, one of the problems with the word "work," sorry to interrupt-

Charles Isbell 44:53 Sure.

Lex Fridman 44:53 ... is that it's... seems to be an antonym in this particular language we have, the opposite of happiness.

Michael Littman 45:01 Mm-hmm.

Lex Fridman 45:01 But it seems like there, there, like... T- that's one of the, you know, we talked about balance. It's, uh, it's always like work/life balance, they always rub me the wrong way. It, as a, as a terminology. I know it's just words.

Michael Littman 45:13 Right, the opposite of work is play.

Lex Fridman 45:15 Yeah.

Michael Littman 45:16 But ideally work is play.

Charles Isbell 45:17 Oh, I can't tell you how much time I, I'd spend, certainly when I was at Bell Labs, that, except for a few very key moments, uh, as a professor I would do this too. I would just, "I cannot believe they are paying me to do this."

Lex Fridman 45:27 (laughs)

Charles Isbell 45:27 Um, 'cause it's fun. It's something that I would, I would do for a hobby if I could anyway. Uh, so that's what I worked out to be.

Lex Fridman 45:35 You sure you wanna be saying that w- when this is being recorded? (laughs)

Charles Isbell 45:38 Oh no, as a dean, that is not true at all.

Lex Fridman 45:40 Yeah.

Charles Isbell 45:40 I need a raise.

Michael Littman 45:41 Yeah. (laughs)

Lex Fridman 45:41 (laughs)

Charles Isbell 45:41 But, but I think here with, with this, that even though a lot of time passed, you know, Michael and I talked almost every... Well, we texted almost every day during the period we were-

Michael Littman 45:49 (laughs) Charles at one point took me... There was... The ICML Conference, the Machine Learning Conference was in Atlanta. I was the chair, the general chair of the conference. Charles was my p- publicity chair or something like that, or-

Charles Isbell 46:04 No, fundraising chair.

Michael Littman 46:04 ... fundraising chair?

Charles Isbell 46:05 Yeah.

Michael Littman 46:06 Um, but he decided it'd be really funny if he didn't actually show up for the conference in his own home city, uh, so he didn't. But he did at one point pick me up at the conference in his Tesla, and drove me to the Atlanta Mall and forced me to buy an iPhone, because he didn't like how it was to text with me and thought it would be better for him if I had an iPhone. The texts would be somehow smoother.

Charles Isbell 46:30 And it was.

Michael Littman 46:31 And it was.

Charles Isbell 46:31 And it is. And his life is better.

Michael Littman 46:32 And my life is better. And so, yeah. But, but it was... Yeah, Charles forcing me to get an iPhone so that he could text me more efficiently. I thought that was an interesting moment.

Charles Isbell 46:41 It works for me. Anyway, so we kept talking the whole time and then eventually we did the, we did the teaching thing, and it was great. And there's a couple of reasons for that, by the way. One is, I really wanted to do something different. Like, you've got this medium here. People claim it can change things. What's a thing that you could do in this medium that you could not do otherwise? Uh, besides edit, right? I mean, what could you do? And, and being able to do something with another person was that kind of thing. It's very hard to... I mean, you can take turns, but teaching together, having conversations is very hard, right? So that was a cool thing. The second thing is it gave me an excuse to do more stuff with him.

Michael Littman 47:12 Yeah, I always thought... I, he makes it sound brilliant, um, and that it is, I guess. But it's... At the time, it really felt like, "I've got a lot to do," Charles is saying, "and it would be great if Michael could teach the course, and I could just-"

Charles Isbell 47:27 Hang out.

Michael Littman 47:27 "Yeah, just kind of coast on that."

Charles Isbell 47:29 Well, that's what the second class was more like then. Because the second class-

Michael Littman 47:32 Second class was explicit.

Charles Isbell 47:32 'Cause the second well-

Michael Littman 47:33 The first class, it was at least half-

Charles Isbell 47:36 Yeah, but I knew all the stuff.

Michael Littman 47:37 ... the structure, the structure that he came up-

Lex Fridman 47:37 I think you're once again letting the, the facts get in the way-

Charles Isbell 47:40 (laughs)

Michael Littman 47:40 (laughs) Of a good story.

Lex Fridman 47:41 ... of a good story.

Michael Littman 47:43 I should just let Charles talk.

Charles Isbell 47:44 But that's the, but that's the facts as he saw them. But it... So that was, that was kind of true for 70-

Lex Fridman 47:47 That's your facts. (laughs)

Michael Littman 47:48 (laughs)

Charles Isbell 47:48 Yeah, that was sort of true for 7642 which was the reinforcement learning class, because that was really his class.

Lex Fridman 47:52 You started with reinforcement learning?

Charles Isbell 47:53 No, we started with... I did the-

Lex Fridman 47:54 Machine learning?

Charles Isbell 47:55 ... intro, intro to machine learning, 7641, uh, which is supervised learning, unsupervised learning, and reinforcement learning and decision-making, cram all that in there, the kind of assignments that we talked about earlier. And then eventually, about a year later, we did a follow on, 7642, which was reinforcement learning and decision-making. The first class was based on something I'd been teaching at that point for well over a decade. And the second class was based on something Michael had been teaching. Actually I learned quite a bit teaching that class with him, uh, but he drove most of that. But the first one I drove most of it, it was all my material. Although I had stolen that material originally from slides I found online from Michael who had originally stolen that material from, I guess, slides he found online probably from Andrew Moore, because the jokes were the same anyway. At least some of the-

Michael Littman 48:35 (laughs)

Charles Isbell 48:35 ... at least when I found the slides, some of the stuff was there.

Michael Littman 48:37 Is that true? I just-

Charles Isbell 48:38 Yes, every machine learning class taught in the early 2000s stole from Andrew Moore.

Michael Littman 48:41 A particular joke or, or two?

Charles Isbell 48:43 The, at least the structure. Now I did, and he did actually, a lot more with reinforcement learning and such, and game theory and those kinds of things. But, you know, we all sort of built-

Lex Fridman 48:52 You mean in the research world?

Michael Littman 48:52 No, no, no. In that class.

Charles Isbell 48:53 No, I mean in the, in teaching that class.

Lex Fridman 48:54 Oh.

Michael Littman 48:54 The coverage was different than-

Charles Isbell 48:56 Than what other people were doing.

Michael Littman 48:57 ... what was started. Oh, okay.

Charles Isbell 48:57 Most people were just doing supervised learning and maybe a little bit of, you know, clustering and whatnot, but we took it all the way to machine learning.

Michael Littman 49:02 A lot of it is, just comes from Tom Mitchell's book.

Charles Isbell 49:04 Oh, no. Yeah, except... Well, half of it comes from Tom Mitchell's book, right? But the other half doesn't. This is wh- this is why it's all readings, right? 'Cause certain things weren't invented when Tom wrote this book.

Michael Littman 49:13 Yeah, okay, that's true.

Charles Isbell 49:14 Right? Uh, but it was, it was quite good. But there's a reason for that besides, you know, just, I wanted to do it. I wanted to do something new and I wanted to do something with him, which is a realization which is, despite what you might believe, he's an introvert and I'm an introvert. Or I'm on the edge of an in- of being an introvert anyway. But both of us, I think, um, enjoy the energy of the crowd-... right? There's something about talking to people and bringing them into whatever we find interesting that is empowering, energi- energizing or whatever. And I found the idea of staring alone, uh, at a computer screen and then talking off of materials less inspiring than I wanted it to be. Uh, to- to do that.

Michael Littman 49:55 And I had in fact done a MOOC for Udacity on algorithms and it was a week in a dark room talking at the screen-

Charles Isbell 50:04 Mm-hmm.

Michael Littman 50:05 ... uh, writing on the little pad, and I didn't know this was happening but they had watched the, the, the crew had watched some of the videos while, you know, like in the middle of this and they're like, "Something's wrong." (laughs) "You're, you're sort of shutting down." Um, and I think a lot of it was I would make jokes and no one would laugh.

Lex Fridman 50:23 Yeah. Yeah.

Michael Littman 50:24 And I felt like the crowd hated me.

Charles Isbell 50:26 Mm-hmm.

Lex Fridman 50:26 Mm-hmm.

Michael Littman 50:26 Now, of course, there was no crowd.

Lex Fridman 50:27 Right.

Michael Littman 50:27 So like it didn't... it wasn't rational.

Lex Fridman 50:29 Yeah.

Michael Littman 50:30 But it's little... each time I tried it and I got no reaction, it just was taking the, the, the energy out of my performance, out of my-

Charles Isbell 50:37 This is a-

Michael Littman 50:38 ... presentation.

Charles Isbell 50:38 ... such a fantastic metaphor for grad school.

Michael Littman 50:40 (laughs)

Charles Isbell 50:40 Anyway, by working together-

Lex Fridman 50:42 Yeah.

Michael Littman 50:42 (laughs)

Charles Isbell 50:42 ... we could play off of each other and have a, and, and have a-

Michael Littman 50:44 And keep the energy up because you can't, you, with, you can't let your guard down for a moment with Charles. He'll just, he'll just overpower you.

Charles Isbell 50:51 I have no idea what you're talking about. But we would work really well together, I thought, and we knew each other so I knew that we could, we could sort of make it work. Plus I was the Associate Dean so they had to (laughs) do what I told them to do so we had to do the... we had to make it work. And so it worked out very well, I thought. Um, well enough that we-

Michael Littman 51:04 With great power comes great power.

Charles Isbell 51:06 That's right. And we became Smoov and Curly and, uh, that's when we, we, we did the, the, um, the, uh, overfitting thriller video.

Michael Littman 51:15 Yeah, we took... yeah, yeah, that's a thing.

Charles Isbell 51:17 Yeah. Introduction.

Lex Fridman 51:17 So, what... okay, can we just like, vroom. Like, uh-

Michael Littman 51:20 (laughs)

Lex Fridman 51:20 ... Smoov and Curly, where were that come from?

Michael Littman 51:21 So, okay, so the... and then th- it happened, it was completely spontaneous, I think.

Lex Fridman 51:24 These are nicknames you go by-

Michael Littman 51:25 Yeah. So, so-

Lex Fridman 51:26 ... for, uh-

Charles Isbell 51:27 It's what the students call us.

Michael Littman 51:28 He was, he was lecturing... So the, the way that we structured the lectures is one of us is the lecturer and one of us is basically the student.

Lex Fridman 51:35 Mm-hmm.

Michael Littman 51:35 And so the... he was lecturing on-

Charles Isbell 51:37 The lecturer prepares all the materials-

Michael Littman 51:39 ... game the-

Charles Isbell 51:39 ... comes up with the quizzes and then the student comes in not knowing anything. So it was, you know, just like being on campus.

Lex Fridman 51:44 Yeah.

Charles Isbell 51:45 Uh, and I was doing game theory, in particular-

Michael Littman 51:47 Yeah.

Charles Isbell 51:47 ... uh, the prisoner's dilemma.

Michael Littman 51:48 Prisoner's dilemma. And so he needed to s- to set up a little prisoner's dilemma grid. So he drew it and I could see what he was drawing and the, the prisoner's dilemma consists of two players, two parties, so he decided he, he would make little cartoons of the two of us. And so there was, uh, two criminals, right? That were deciding whether or not to rat each other out. Um, one of them he drew as, you know, a circle with a smiley face and a kind of goatee thing.

Charles Isbell 52:12 Mm-hmm.

Michael Littman 52:13 Smooth head. And the other one with all sorts of curly hair. And he said, "This is Smoov and Curly." I said, "Smoov and Curly?" He said, "No, no, Smoov with a V. It's very important that it have a V."

Lex Fridman 52:23 Yeah. And that stuck.

Michael Littman 52:24 And, and-

Lex Fridman 52:24 I actually watched that video.

Michael Littman 52:24 ... then the students really, yeah, the students really took to that like they've rela- they found that relatable. (laughs)

Charles Isbell 52:29 He started singing Smooth Criminal by Michael Jackson.

Lex Fridman 52:31 Yeah. Yeah, yeah. And that-

Charles Isbell 52:32 Because it's

Michael Littman 52:32 (laughs)

Lex Fridman 52:32 ... those na- those names stuck.

Michael Littman 52:33 So that's... so we now have a video series, the... an episode, uh, kind of first actual episode should be coming out today, um, Smoov and Curly on video-

Charles Isbell 52:42 On video.

Michael Littman 52:43 ... where the two of us discuss, uh, West, episodes of Westworld where we watch Westworld and we're like, "Huh, what does this say about computer science- (laughs)

Charles Isbell 52:51 (laughs)

Michael Littman 52:51 ... and AI?"

Charles Isbell 52:52 And we've never... we, we did not watch it. I mean, I know it's on season three or whatever. We haven't-

Lex Fridman 52:55 Yeah.

Charles Isbell 52:55 As of this recording, it's on season three.

Lex Fridman 52:57 Yeah.

Charles Isbell 52:57 And, uh-

Michael Littman 52:58 We've watched now two episodes total.

Charles Isbell 53:00 Yeah. I think I've watched three.

Lex Fridman 53:01 What, what do you think about Westworld?

Michael Littman 53:02 Two episodes in.

Lex Fridman 53:03 Did you?

Michael Littman 53:03 So I could tell you-

Lex Fridman 53:04 Okay, well, yeah.

Michael Littman 53:04 ... so far, I'm just guessing what's gonna happen next. It seems like bad things are gonna happen with the robots uprising. There's a lot of s-

Lex Fridman 53:11 Spoiler alert. (laughs)

Charles Isbell 53:12 So I, I have not, I have not s-

Michael Littman 53:13 (laughs)

Charles Isbell 53:13 I mean, you know, I vaguely remember a movie existing so I assume it's, it's related to that. But-

Michael Littman 53:18 That was more my time than your time, Charles.

Charles Isbell 53:20 That's right 'cause you're much older than I am. I think the important thing here is that, uh, it's narrative, right? It's all about telling a story, that's the whole driving thing. But the idea that they would give these reveries, that they would make pe- they would make them-

Michael Littman 53:31 Let them remember-

Charles Isbell 53:32 ... remember the awful things that happened.

Michael Littman 53:35 ... terrible things that happened.

Charles Isbell 53:35 Who could possibly think that was gonna... I, I get a, uh... I mean, I don't know. I have only seen the first two episodes or maybe the third one. I think I've only seen the first two.

Michael Littman 53:40 You know what it was? Do you know what the problem is?

Charles Isbell 53:42 What?

Michael Littman 53:42 That the robots were actually designed by Hannibal Lecter.

Charles Isbell 53:45 (laughs) That's true. They, they were. (laughs)

Michael Littman 53:47 So like, wh- what do you think is gonna happen?

Charles Isbell 53:49 Anyways.

Michael Littman 53:49 Bad things.

Charles Isbell 53:50 It's clear that things are happening and characters are being introduced and we don't yet know anything but, uh, still I was just struck by how it's all driven by narrative and story. And there's all these implied things like programming hap- the programming interface is talking to them about what's going on in their heads which is both, I mean, artistically it's probably useful to film it that way. But think about how it would work in real life, that just seems very cra- but there was... we, we saw on the second episode, there's a screen you could see things-

Michael Littman 54:15 They were wearing like Google Glass.

Charles Isbell 54:16 ... that sort of state in the world. It was quite interesting to just kind of ask this question so far. I mean, I assume it veers off into never-never land at some point. But, uh-

Michael Littman 54:24 So we don't know, we can't answer that question.

Lex Fridman 54:25 I'm also a f- a fan of a guy named Alex Garland. He's the director of Ex Machina.

Michael Littman 54:30 Mm-hmm.

Charles Isbell 54:30 Mm-hmm.

Lex Fridman 54:31 And he is the first... I wonder if Kubrick was like this actually. Is he... like studies what would it take to program an AI system? Like he's, he's curious enough to go into that direction. Uh, on the Westworld side, I felt there was more emphasis on the narratives than like actually asking like computer science questions. (laughs)

Charles Isbell 54:52 Yeah.

Lex Fridman 54:52 Like, how would you build this? How would you, uh, and, and-

Charles Isbell 54:57 How would you debug it? I still think-

Michael Littman 54:58 (laughs)

Charles Isbell 54:58 ... to me that's the, the key issue.

Lex Fridman 55:00 Oh, that's brilliant.

Charles Isbell 55:01 They were terrible debuggers.

Lex Fridman 55:02 Yeah. And I think-

Michael Littman 55:03 Well, they said specifically, "So we make a change and we put it out in the world and that's bad because something terrible could happen." Like, if you're putting things out in the world and you're not sure whether something terrible is going to happen, your probably- your process is probably flawed.

Charles Isbell 55:13 I just feel like there should've been someone who's sole job it was was to walk around and poke his head in and say, "What could possibly go wrong?"

Michael Littman 55:19 (laughs)

Charles Isbell 55:19 Just over and over again.

Michael Littman 55:20 (laughs)

Lex Fridman 55:20 I would have loved if there was an, an... and I did watch a lot more and I'm not giving anything away. I would have loved it if there was like an episode where like, like the new intern is like debugging a new model or something.

Michael Littman 55:31 Mm-hmm.

Lex Fridman 55:31 And like it just keeps failing and, and they're like, "All right," and then it's... but more turns into like a s- uh, episode of Silicon Valley or something like that-

Charles Isbell 55:38 Yeah.

Lex Fridman 55:39 ... versus the, versus like o- this ominous AI systems that are constantly like threatening the, the fabric of this world that's been created.

Michael Littman 55:47 Yeah.

Charles Isbell 55:47 Yeah, and you know the other... the, this, this reminds me of something that-So, I agree with that. That'd actually be very cool at least, well, for (laughs) the small percentage of people who care about debugging systems. But the other thing is-

Michael Littman 55:57 (laughs) Debugging, the series.

Charles Isbell 55:59 Yeah.

Michael Littman 55:59 (laughs)

Charles Isbell 55:59 It falls into... And think of the sequels. Fear of the Debugging.

Michael Littman 56:02 Oh, my gosh.

Charles Isbell 56:03 And, anyway, so I, I-

Michael Littman 56:04 It's a nightmare show. It's a, it's a horror movie.

Charles Isbell 56:07 (laughs) I, I think that's where we lose people, by the way, early on, is that people who either decide... Either figure out debugging or think debugging is terrible. Th- this is part of the struggle-

Michael Littman 56:13 Oh, where we lose people in computer science.

Charles Isbell 56:14 This, this is part of the struggle-

Michael Littman 56:15 Yeah, yeah, yeah, yeah.

Charles Isbell 56:15 ... versus suffering, right? You, you get through it and you kind of get the skills of it or you're just like, "This is dumb. This is a dumb way to do anything," and I think that's when we lose people. But, um, I... Well, I'll leave it at that. But I think that-

Michael Littman 56:27 Mm-hmm.

Charles Isbell 56:27 I think that, that there's something really, really neat about framing it that way. But what I don't like about all of these pro- all of these things, and I love Ex Machina, by the way.

Michael Littman 56:39 Yeah.

Charles Isbell 56:39 Although the, the ending was very depressing. Um-

Lex Fridman 56:41 Well, okay, uh, one of the things after talking to Al- uh, A- Alex about, he says that the thing that nobody noticed-

Charles Isbell 56:48 Mm-hmm.

Lex Fridman 56:49 ... he put in is, uh, the, uh, uh, at the end, spoiler alert, the, the robot turns and looks at the camera and smiles-

Charles Isbell 56:59 Right.

Lex Fridman 56:59 ... briefly. And to him, he thought that his definition of passing the Turing, the general version of the Turing test or the consciousness test, is smiling for no one.

Charles Isbell 57:13 Hmm.

Michael Littman 57:13 Oh.

Lex Fridman 57:14 Like, like, not al- you know, it's, it's like the Chinese room kind of experiment. It's not always trying to act for others-

Charles Isbell 57:22 Right.

Lex Fridman 57:22 ... but just on your own, being able to have a relationship with the actual experience and just like take it in. I don't know. He said like nobody noticed-

Charles Isbell 57:31 I mean, I-

Lex Fridman 57:32 ... kind of the magic of it.

Charles Isbell 57:32 I have this vague feeling that I remember the smile, but now you, now you've just put the memory in my head, so-

Lex Fridman 57:37 Yeah.

Michael Littman 57:37 Yeah.

Charles Isbell 57:37 ... probably not. But I do think that that's interesting. Although, by looking at the camera, you are smiling for the audience, right? You're breaking-

Lex Fridman 57:44 Yeah.

Charles Isbell 57:44 ... the fourth wall.

Lex Fridman 57:44 Yeah.

Charles Isbell 57:45 It seems... I mean, well, that's a, that's a limitation-

Lex Fridman 57:47 That's a deep one.

Charles Isbell 57:47 ... of the medium, but I, I like that idea. But here's the problem I have-

Lex Fridman 57:50 Yeah.

Charles Isbell 57:50 ... with all of those movies, all of them, um, is that... And, but I know why it's this way, and, and I enjoy those movies, um, and Westworld, is, uh, it sets up the problem of AI as succeeding and then having something we cannot control? But it's, that's not the bad part of AI. The bad part of AI is the stuff we're living through now, right? It's the using the data to make decisions that are terrible. It's not the intelligence that's gonna go out there and surpass us and, you know, take over the world or, you know, lock us into a room to starve to death slowly, um, over multiple days. It's instead, uh, the, the tools that we're building that are allowing us to make the terrible decisions we would've less efficiently made before, right? You know, computers are very good at making us more efficient, including being more efficient at doing terrible things, and, and that's the part of the AI we have to worry about. It's not the, you know, true intelligence that we're gonna build sometime in the future, probably long after we're around. Um, but, you know, I, I, I, I, I just, I think that whole framing of it sort of misses the point, even though it is inspiring and I was inspired by those ideas, right? That I got into this in, in part 'cause I wanted to build something like that. Philosophical questions were interesting to me, but, but, you know, that's not where the terror comes from. The terror comes from the every day.

Lex Fridman 59:06 And you can construct situa- it's in, in the subtlety of the interaction between AI and the human, like with the, uh, with the social networks, uh, all the stuff you're doing with, uh, interactive artificial intelligence.

Charles Isbell 59:17 Mm-hmm.

Lex Fridman 59:17 But, you know, I, I feel like HAL 9000 came a little bit closer to that when, in s- in 2001: Space Odyssey, because it felt like, uh, a personal assistant, you know? It felt like closer to the AI systems we have today-

Charles Isbell 59:31 Mm-hmm.

Lex Fridman 59:31 ... and the, and the, and the real things we might actually encounter, which is over-relying, uh, on, in some fundamental way on our, like, dumb assistants or on social networks, like over- offloading too much of us onto, um, uh, you know, onto things that require internet and power and so on, and thereby becoming powerless as a, uh, standalone entity. And then when that thing starts to misbehave in some subtle way, it creates a lot of problems. And those problems are dramat- dramatized when you're in space because-

Charles Isbell 1:00:08 Mm-hmm.

Lex Fridman 1:00:09 ... you don't have a way to walk away, but-

Charles Isbell 1:00:11 Well, as the man said, um, "Once you, once we started making the decisions for you, it stopped being your world," right? That's the Matrix, Michael, in case you don't-

Michael Littman 1:00:18 I didn't catch it, thank you.

Charles Isbell 1:00:19 ... you don't, you don't remember. But on the other hand-

Michael Littman 1:00:21 Oh, yeah.

Charles Isbell 1:00:21 ... I could say no, because isn't that what we do with people anyway? You know, just kind of the shared intelligence that is humanity-

Michael Littman 1:00:28 Mm-hmm.

Charles Isbell 1:00:28 ... is relying on other people constantly to... I mean, we s- we hyper-specialize, right? As individuals. We're still generally intelligent. We make our own decisions in a lot of ways, but we leave most of this up to other people and that's perfectly fine. And by the way, everyone doesn't necessarily share our goals. Sometimes they seem to be quite against us. Sometimes we make decisions that others would see as against our own interests, and yet we somehow manage it, manage to survive. I'm not entirely sure why an AI would actually make that worse. Or even different, really.

Lex Fridman 1:01:00 You mentioned the Matrix.

Michael Littman 1:01:01 Mm-hmm.

Lex Fridman 1:01:02 Do you think we're living in a simulation?

Michael Littman 1:01:04 It does f- feel like a thought game more than a real scientific question.

Lex Fridman 1:01:10 Well, I'll tell you why, like, I think it's an interesting thought experiment, see what you think.

Michael Littman 1:01:14 Okay.

Lex Fridman 1:01:14 From a computer science perspective, it's a good experiment of, how difficult would it be to create a sufficiently realistic world that us humans would enjoy being in? It, it, that, that's almost like a competition.

Michael Littman 1:01:27 I mean, if we're living in a simulation, then I don't believe that we were put in the simulation. I believe that it, it's just physics playing out and we came out of that. Like, I don't, I don't, I don't think-

Lex Fridman 1:01:39 So, you think you have to build the universe and all the phonomena-

Michael Littman 1:01:41 I think the, the universe itself, we can think of that as a simulation. And in fact, what... I try, sometimes I try to think about... To understand what it's like for a computer to-... start to think about the world, I try to think about (laughs) the world. Um, things like quantum mechanics where it doesn't feel very natural to me at all. Um, and it really strikes me as, I don't understand this thing that we're living in. It, it has... there's weird things happening in it that don't feel natural to me at all. Now, if you want to call that a s- the result of a simulator, okay, I'm fine with that. But like I don't-

Lex Fridman 1:02:14 But those are the bugs in the simulation.

Charles Isbell 1:02:17 There's the bugs. I mean, the interesting thing about-

Michael Littman 1:02:19 (laughs)

Charles Isbell 1:02:19 ... the simulation is that it, it might have bugs. I mean, that, that's the thing that I... the, the-

Michael Littman 1:02:23 But there wouldn't be bugs for the people in the simulation, they're just... that's just reality.

Charles Isbell 1:02:27 Unless you were-

Michael Littman 1:02:27 They're not bugs.

Charles Isbell 1:02:27 ... aware enough to know that there was a bug. But I, I, I think-

Lex Fridman 1:02:30 Back to The Matrix.

Charles Isbell 1:02:31 Yeah. The way you put the question though-

Michael Littman 1:02:32 I see. I don't think that we live in a, in a simulation created for us. I... okay, I would say that.

Charles Isbell 1:02:36 I think that's interesting, I've actually never thought about it that way. I mean, you... the way you asked the question though, could you create a world that is enough for us humans? It's an interestingly sort of self-referential question because the beings that created the simulation probably have not created a simulation that's realistic for them. But we're in the simulation, and so it's realistic for us. So we could create a simulation that is fine for the people in the simulation, as it were.

Michael Littman 1:03:02 Right.

Charles Isbell 1:03:02 That would not necessarily be fine for us as the creators of the simulation.

Lex Fridman 1:03:05 But... well, you can, you can forget. I mean, when you go into the... if you play video games in virtual reality, you can... if... with some suspension of disbelief-

Michael Littman 1:03:14 Yeah.

Lex Fridman 1:03:14 ... or, or whatever.

Charles Isbell 1:03:15 Yeah.

Lex Fridman 1:03:15 Uh-

Michael Littman 1:03:16 It becomes the world, yeah.

Lex Fridman 1:03:17 It becomes the world e- even like in brief moments, you s- you forget that another world exists.

Michael Littman 1:03:22 Mm-hmm.

Lex Fridman 1:03:22 I mean that's what like good stories do, they pull you in.

Charles Isbell 1:03:25 Mm-hmm.

Lex Fridman 1:03:25 The question is, is it possible to pull... you know, our brains are limited, is it possible to pull the brain in to where we actually stay in that world longer and longer and longer and longer? And like not only that but we don't want to leave, and so especially this is the key thing about the dev- developing brain, is if we journey into that world early on in life often-

Charles Isbell 1:03:48 How would you even know? Yeah.

Lex Fridman 1:03:49 Yeah. So I, I... but like from a video game design perspective, from a West World perspective, it's... I think, I think it's an important thing for even, uh, computer scientists to think about, 'cause it's clear that video games are getting much better and virtual reality, although it's been ups and downs just like artificial intelligence, it feels like virtual reality will be here in a very impressive form if we were to fast-forward 100 years into the future in a way that might change society fundamentally. Like, if I were to... I'm very limited in, in predicting the future as all of us are.

Michael Littman 1:04:26 Mm-hmm.

Lex Fridman 1:04:26 But if I were to try to predict like in which way I'd be surprised to see the world 100 years from now, it'd be that... or impressed, it'd be that we're all no longer living in this physical world, that we're all living in a virtual world.

Charles Isbell 1:04:45 You really need to read Calculating God by Sawyer. It's a... you'll, you'll read it in a night. It's a very easy read but it's... a- as- assuming you're that kind of reader, but it's a, it's a good story and it's kind of about this but not in a way that it appears, and I, uh, really enjoyed the thought experiment. Um, I think it's... pretty sure it's Robert Sawyer. But anyway, he's, he's apparently Canadian's top science fiction writer, which is why the story mostly takes place in Toronto. Uh, but it's a, it's a very good, uh, it's a very good sort of story that, that sort of imagines this very different kind of simulation hypothesis sort of thing from say, um, The Egg, for example, you know. You know I'm talking about the short story, um, by the guy who did The Martian. Who wrote The Martian?

Michael Littman 1:05:35 Mm-hmm.

Charles Isbell 1:05:36 You know what I'm talking