Leslie Kaelbling: Reinforcement Learning, Planning, and Robotics | Lex Fridman Podcast #15

1. 0:00 – 15:00

   The following is a…

   1. LFLex Fridman0:00

      The following is a conversation with Leslie Kaelbling. She's a roboticist and professor at MIT. She's recognized for her work in reinforcement learning, planning, robot navigation, and several other topics in AI. She won the IJCAI Computers and Thought Award and was the editor-in-chief of the prestigious Journal of Machine Learning Research. This conversation is part of the Artificial Intelligence Podcast at MIT and beyond. If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter at lexfridman, spelled F-R-I-D. And now, here's my conversation with Leslie Kaelbling.

   2. LKLeslie Kaelbling0:42

      What made me get excited about AI, I can say that, is I read Godel, Escher, Bach when I was in high school. That was pretty formative for me because it exposed, uh, the interestingness of primitives and combination and how you can make complex things out of simple parts and ideas of AI and what kinds of programs might generate intelligent behavior. So...

   3. LFLex Fridman1:08

      So you first fell in love with AI reasoning logic versus robots?

   4. LKLeslie Kaelbling1:13

      Yeah, the robots came because, um, my first job... So I finished an undergraduate degree in philosophy at Stanford and was about to finish a master's in computer science, and I got hired at SRI, uh, in their AI lab, and they were building a robot that was a kind of a follow-on to Shakey, but all the Shakey people were not there anymore.

   5. LFLex Fridman1:33

      Mm-hmm.

   6. LKLeslie Kaelbling1:33

      And so my job was to try to get this robot to do stuff, and that's really kind of what got me interested in robots.

   7. LFLex Fridman1:40

      So maybe taking a small step back-

   8. LKLeslie Kaelbling1:41

      Yeah.

   9. LFLex Fridman1:41

      ...to your bachelor's in Stanford in philosophy-

   10. LKLeslie Kaelbling1:44

       Yeah.

   11. LFLex Fridman1:44

       ...did master's and PhD in computer science, but the bachelor's in philosophy. Uh, so what was that journey like? What elements of philosophy do you think-

   12. LKLeslie Kaelbling1:53

       Yeah.

   13. LFLex Fridman1:53

       ...you bring to your work in computer science?

   14. LKLeslie Kaelbling1:55

       So it's surprisingly relevant. So the... Part of the reason that I didn't do a computer science undergraduate degree was that there wasn't one at Stanford at the time, but that there's part of philosophy, and in fact, Stanford has a special sub-major in something called now symbolic systems which is logic model theory, formal semantics of natural language. And so that's actually a perfect preparation for work in AI and computer science.

   15. LFLex Fridman2:20

       That, that's kind of interesting. So if you were interested in artificial intelligence, what, what kind of majors were people even thinking about taking? Was it in neuroscience? Was... So besides philosophies, what, what were you supposed to do if you were fascinated by the idea of creating intelligence?

   16. LKLeslie Kaelbling2:37

       There weren't enough people who did that for that even to be a conversation.

   17. LFLex Fridman2:41

       Okay.

   18. LKLeslie Kaelbling2:42

       I mean, I think probably, probably philosophy. I mean, it's interesting, in my class, uh, my graduating class of undergraduate philosophers, probably, maybe slightly less than half went on in computer science-

   19. LFLex Fridman2:57

       Mm-hmm.

   20. LKLeslie Kaelbling2:58

       ...slightly less than half went on in law, and, like, one or two went on in philosophy. Uh, so it was a common kind of connection.

   21. LFLex Fridman3:05

       Do you think AI researchers have a role, be part-time philosophers? Or should they stick to the solid science and engineering without sort of taking the philosophizing tangents? I mean, you work with robots, you think about what it takes to create intelligent beings. Uh, aren't you the perfect person to think about the big picture philosophy of it all?

   22. LKLeslie Kaelbling3:25

       The parts of philosophy that are closest to AI, I think, or at least the closest to AI that I think about are stuff like belief and knowledge and denotation and that kind of stuff. And that's, uh, you know, it's quite formal and it's, like, just one step away from the kinds of computer science work that we do kind of routinely. I think that there are important questions still about what you can do with a machine and what you can't and so on, although at least m- my personal view is that I'm completely a materialist and I don't think that there's any reason why we can't make a robot be behaviorally indistinguishable from a human. And the question of whether it's in- distinguishable internally, whether it's a zombie or not, in philosophy terms, I actually don't... I don't know and I don't know if I care too much about that.

   23. LFLex Fridman4:17

       Right. But there, there is, uh, philosophical notions, they're mathematical and philosophical because we don't know so much, of how difficult it is, how difficult is the perception problem? How difficult is the planning problem? How difficult is it to operate in this world successfully? Because our robots are not currently as successful as human beings in many tasks. The, the question about the gap between current robots and human beings borders a little bit on philosophy. Uh, you know, the, the expanse of knowledge that's required to operate in this world and the ability to, uh, form common sense knowledge, the ability to reason about uncertainty, much of the work you've been doing-

   24. LKLeslie Kaelbling4:59

       Mm-hmm.

   25. LFLex Fridman4:59

       ...there's, there's open questions there that, uh, I, I don't know, require to activate a certain big-picture view.

   26. LKLeslie Kaelbling5:08

       To me, that doesn't seem like a philosophical gap at all.

   27. LFLex Fridman5:10

       I see.

   28. LKLeslie Kaelbling5:10

       That's just a t-... To me, it's a, it's a... There is a big technical gap.

   29. LFLex Fridman5:13

       Yes.

   30. LKLeslie Kaelbling5:13

       There's a huge technical gap. But I don't see any reason why it's more than a technical gap.
2. 15:00 – 30:00

   I think the idea…

   1. LFLex Fridman15:00

      guess the question is, what kinda things can you, uh, encode, uh, symbolically so you can reason about...

   2. LKLeslie Kaelbling15:08

      I think the idea about... And, and even symbolic, I don't even like that terminology-

   3. LFLex Fridman15:16

      (laughs)

   4. LKLeslie Kaelbling15:16

      ... 'cause I don't know what it means-

   5. LFLex Fridman15:17

      Right.

   6. LKLeslie Kaelbling15:17

      ... technically and formally. I do believe in abstractions. So abstractions are critical, right? It, you cannot reason at completely fine grain about everything in your life, right? You can't-

   7. LFLex Fridman15:29

      Right.

   8. LKLeslie Kaelbling15:29

      ... make a plan at the level of images and torques for getting a PhD.

   9. LFLex Fridman15:33

      Right.

   10. LKLeslie Kaelbling15:34

       So you have to reduce the size of the state space, and you have to reduce the horizon if you're gonna reason about getting a PhD or even buying the ingredients to make dinner. And so, so how can you reduce the spaces and the horizon of the reasoning you have to do? And the answer is abstraction. Spatial abstraction, temporal abstraction. I think abstraction along the lines of goals is also interesting, like you might... Or well, abstraction and decomposition.

   11. LFLex Fridman16:01

       Mm-hmm.

   12. LKLeslie Kaelbling16:01

       Like, goals is maybe more of a decomposition thing. So I think that's where these kinds of, if you wanna call it symbolic or discrete models come in. You, you talk about a room of your house instead of your pose.

   13. LFLex Fridman16:14

       Mm-hmm.

   14. LKLeslie Kaelbling16:14

       You talk about, uh, you know, doing something during the afternoon instead of at 2:54. And you do that because it makes your reasoning problem easier and also because y- you have, you don't have enough information to reason in high fidelity about your pose of your elbow at 2:35 this afternoon anyway.

   15. LFLex Fridman16:37

       Right. When you're trying to get a PhD. That, that, that's-

   16. LKLeslie Kaelbling16:40

       Right. Or when you're doing anything really.

   17. LFLex Fridman16:41

       Oh, re- yeah, okay. Uh-

   18. LKLeslie Kaelbling16:42

       Except for at that moment. At that moment, you do have to reason about the pose of your elbow maybe.

   19. LFLex Fridman16:46

       Right.

   20. LKLeslie Kaelbling16:46

       But then you, maybe you do that in some continuous joint space kinda model. And so I, again, I, m- my biggest point about all of this is that there should be... that dogma is not the thing, right? We shouldn't, it shouldn't be that I am in favor against symbolic reasoning, and you're in favor against neural networks. It should be that just, just computer science tells us what the right answer to all these questions is if we were smart enough to figure it out.

   21. LFLex Fridman17:13

       Well, yeah. When you try to actually solve the problem with computers, eh, the, the right answer comes out. But you mentioned abstractions.

   22. LKLeslie Kaelbling17:19

       Mm-hmm.

   23. LFLex Fridman17:19

       I mean, neural networks form abstractions, or, uh, rather, uh, there's a- there's automated ways to form abstractions.

   24. LKLeslie Kaelbling17:27

       Absolutely.

   25. LFLex Fridman17:27

       And there's expert-driven ways to form abstractions.

   26. LKLeslie Kaelbling17:30

       Mm-hmm.

   27. LFLex Fridman17:30

       And, uh, expert human-driven ways.

   28. LKLeslie Kaelbling17:32

       Mm-hmm.

   29. LFLex Fridman17:33

       And humans just seems to be way better at forming abstractions currently in certain problems. So when you're referring to 2:45 AM, uh, PM versus afternoon, how do we construct that taxonomy? Is there any room for automated construction of such abstractions?

   30. LKLeslie Kaelbling17:52

       Oh, I think eventually, yeah. I mean, I think when we get to be better and machine learning engineers, we'll build algorithms that build awesome abstractions.
3. 30:00 – 45:00

   Mm-hmm. …

   1. LKLeslie Kaelbling30:00

      the temporal hierarchy in parti... Well, p- there's spatial hierarchy, but let's talk about temporal hierarchy.

   2. LFLex Fridman30:04

      Mm-hmm.

   3. LKLeslie Kaelbling30:04

      So you might say, "Oh, I have this long, uh, execution I have to do, but I can divide it into some segments abstractly." Right? So maybe I have to get out of the house, I have to get in the car, I have to drive and so on. And so-You can plan, if you can build abstractions. So this, we started out by talking about abstractions and we're back to that now. If you can build abstractions in your state space, and abstractions, sort of temporal abstractions, then you can make plans at a high level and you can say, "I'm gonna go to town and then I'll have to get gas, and then I can go here and I can do this other thing." And you can reason about the dependencies and constraints among these actions, again, without thinking about the complete details. What we do in our hierarchical planning work is then say, "All right, I make a plan at a high level of abstraction." I have to have some reason to think that it's feasible without working it out in complete detail.

   4. LFLex Fridman31:06

      Mm-hmm.

   5. LKLeslie Kaelbling31:06

      And that's actually the interesting step. I always like to talk about walking through an airport, like-

   6. LFLex Fridman31:11

      Mm-hmm.

   7. LKLeslie Kaelbling31:12

      ... you can plan to go to New York and arrive at the airport and then find yourself in an office building later. You can't even tell me in advance what your plan is for walking through the airport.

   8. LFLex Fridman31:22

      Hm.

   9. LKLeslie Kaelbling31:22

      Partly because you're too lazy to think about it maybe, but partly also because you just don't have the information. You don't know what gate you're landing in or what people are gonna be in front of you or anything. So, there is no point in planning in detail.

   10. LFLex Fridman31:35

       Mm-hmm.

   11. LKLeslie Kaelbling31:36

       But you have to have ... You have to make a leap of faith that you can figure it out once you get there. And it's really interesting to me how you arrive at that. How do you ... So you have learned over your lifetime to be able to make some kinds of predictions about how hard it is to achieve some kinds of sub-goals.

   12. LFLex Fridman31:57

       Mm-hmm.

   13. LKLeslie Kaelbling31:57

       And that's critical. Like, you would never plan to fly somewhere if you couldn't, didn't have a model of how hard it was to do some of the intermediate steps. So one of the things we're thinking about now is how do you do this kind of very aggressive generalization, uh, I mean, to situations that you haven't been in and so on to predict how long will it take to walk through the Kuala Lumpur airport?

   14. LFLex Fridman32:18

       Mm-hmm.

   15. LKLeslie Kaelbling32:18

       Like, you, you could give me an estimate and it wouldn't be crazy. And you have to have an estimate of that in order to make plans that involve walking through the Kuala Lumpur airport, even if you don't need to know it in detail. So I'm really interested in these kinds of abstract models and how do we acquire them. But once we have them, we can use them to do hierarchical reasoning, which is, I think is very important.

   16. LFLex Fridman32:40

       Yeah, there's this notion of go- uh, goal regression and pre-image back chaining, this idea of starting at the goal-

   17. LKLeslie Kaelbling32:48

       Mm-hmm.

   18. LFLex Fridman32:48

       ... and just forming these big clouds of states that you, you get, I mean, it, it's almost like saying to the airport, you know, you, you know once you show up to the, uh, the airport that that's, you, you're like a few steps away from the goal. So like, thinking of it this way, uh, is kind of interesting. I don't know if you have sort of, uh, further comments on that-

   19. LKLeslie Kaelbling33:12

       Hm.

   20. LFLex Fridman33:12

       ... uh, uh, of starting at the goal, why that's Yeah.

   21. LKLeslie Kaelbling33:15

       cool. I mean, it's interesting that Simon, Herb Simon, back in the early days of AI did, talked a lot about means-ends reasoning and reasoning back from the goal. There's a kind of an intuition that people have that the number of the, with, so state space is big, the number of actions you could take is really big. So if you say, "Here I sit and I wanna search forward from where I am, what are all the things I could do?"

   22. LFLex Fridman33:40

       Right.

   23. LKLeslie Kaelbling33:40

       That's just overwhelming. If you say, if you can reason at this other level and say, "Here's what I'm hoping to achieve. What could I do to make that true?" That somehow the branching is smaller. Now, what's interesting is that, like in the AI planning community, that hasn't worked out. In the class of problems that they look at and the methods that they tend to use, it hasn't turned out that it's better to go backward. Um, it's still kind of my intuition that it is, but I can't prove that to you right now.

   24. LFLex Fridman34:08

       (laughs) Right. I share your intuition, at least for us mere humans.

   25. LKLeslie Kaelbling34:12

       Mm-hmm.

   26. LFLex Fridman34:14

       Speaking of which, uh, y- when you, uh, maybe now we take a, take a, take a little step into that philosophy circle.

   27. LKLeslie Kaelbling34:21

       Uh-oh.

   28. LFLex Fridman34:22

       Uh, how hard would it ... When you think about human life as you, so you give those as examples often, how hard do you think it is to formulate human life as a planning problem, or aspects of human life? So when you look at robots, you're often trying to think about object manipulation, uh, tasks, about moving a, a thing. When you, when you take a slight step outside the room, let the robot leave and go get lunch, uh, or maybe try to, um, pursue more fuzzy goals, how hard do you think is that problem? If you were to try to maybe, put another way, try to formulate human life as, as a planning problem.

   29. LKLeslie Kaelbling35:01

       W- well, that would be a mistake. I mean, it's not all a planning problem.

   30. LFLex Fridman35:05

       (laughs)
4. 45:00 – 1:00:00

   Hm. …

   1. LFLex Fridman45:00

      jumping topics a little bit, you started the Journal of Machine Learning Research and served as its editor in chief. Uh, how did the publication come about?

   2. LKLeslie Kaelbling45:10

      Hm.

   3. LFLex Fridman45:11

      And, uh, what do you think about the current publishing model space in, uh, machine learning-

   4. LKLeslie Kaelbling45:17

      Hmm.

   5. LFLex Fridman45:17

      ... artificial intelligence, and of course

   6. NANarrator45:18

      That's interesting.

   7. LKLeslie Kaelbling45:18

      Yeah. Okay, good. So it came about because there was a journal called Machine Learning, which still exists, which was owned by Kluwer. And there was, I was on the editorial board, and we used to have these meetings annually where we would complain to Kluwer that it was too expensive for the libraries and that people couldn't publish, and we would really like to have some kind of relief on those fronts. And they would always sympathize but not do anything. So, uh, we just decided to make a new journal. And, uh, there was the Journal of AI Research which has, was on the same model which had been in existence for maybe five years or so, and it was going along pretty well. So, uh, we just made a new journal. It wasn't, I mean, it, um, I don't know, I guess it was work, but it wasn't that hard. So basically, the editorial board, probably 75% of the editorial board of, uh, Machine Learning resigned and we founded then this new journal.

   8. LFLex Fridman46:17

      But it was s- sort of, it was more open.

   9. LKLeslie Kaelbling46:21

      Yeah, right. So it's completely open. It's open access. Actually, uh, uh, I had a postdoc, George Konidaris, who wanted to call these journals Free For All.

   10. LFLex Fridman46:32

       (laughs)

   11. LKLeslie Kaelbling46:32

       Uh, because there were... I mean, it both has no page charges and has no, uh, uh, access restrictions. And the reason... And so lots of people, I mean, for, there were, there were p- people who were mad about the existence of this journal who thought it was a fraud or something. It would be impossible, they said, to run a journal like this with basically...

   12. LFLex Fridman46:55

       Yeah.

   13. LKLeslie Kaelbling46:55

       I mean, for a long time I didn't even have a bank account. Uh, I paid for the lawyer to incorporate and the IP address and th- uh, it just, it cost a couple hundred dollars a year to run. It's a little bit more now, but not that much more. But, uh, that's because I think computer scientists are competent and autonomous in a way that many scientists in other fields aren't, I mean, uh, doing these kinds of things. We already typeset our own papers. We all have students and people who can hack a website together in an afternoon. So the infrastructure for us was like, pfft, not a problem. But for other people in other fields, it's a harder thing to do.

   14. LFLex Fridman47:34

       Yeah, and this kind of open access journal is nevertheless one of the most prestigious journals. So it's not like, uh, uh, prestige and it can be achieved without any of the puzzles.

   15. LKLeslie Kaelbling47:46

       Paper is not required-

   16. LFLex Fridman47:48

       Yeah.

   17. LKLeslie Kaelbling47:48

       ... for prestige-

   18. LFLex Fridman47:49

       Yeah.

   19. LKLeslie Kaelbling47:49

       ... it turns out. Yeah.

   20. LFLex Fridman47:50

       Uh, so on the review process side, I've actually, a long time ago, I don't, don't remember when, uh, I, I reviewed a paper where you were also a reviewer, and I remember reading your review and being influenced by it, and it was really well written. It influenced how I write feature reviews. Uh, you disagreed with me actually. Uh... (laughs)

   21. LKLeslie Kaelbling48:09

       (laughs) Oh.

   22. LFLex Fridman48:09

       And you made it, uh, uh, m- my review much better. So I, but nevertheless, the review process, you know, has its, uh, flaws.

   23. LKLeslie Kaelbling48:18

       Mm-hmm.

   24. LFLex Fridman48:19

       And how do you think, uh, what do you think works well? How c- how can it be improved?

   25. LKLeslie Kaelbling48:23

       So actually when I started JMLR, I wanted to do something completely different, and I didn't because it felt like we needed a traditional journal of record and so we just made JMLR be almost like a normal journal-

   26. LFLex Fridman48:38

       Mm-hmm.

   27. LKLeslie Kaelbling48:38

       ... except for the open access parts of it basically. Um-... increasingly, of course, publication is not even a sensible word. You can publish something by putting it in arXiv, so I can publish everything tomorrow. So, making stuff public is, there's no barrier. We still need curation and evaluation. I don't have time to read all of arXiv. And you could argue that kind of social thumbs-upping of articles suffices, right? You might say, "Oh, heck with this. We don't need journals at all. We'll put everything on arXiv, and people will up-vote and down-vote the articles," and then your CV will say, "Oh, man, they, he got a lot of up-votes, so, uh-"

   28. LFLex Fridman49:32

       (laughs)

   29. LKLeslie Kaelbling49:32

       "... that's good." Um, but I think there's still value in careful reading and commentary of things. And it's hard to tell when people are up-voting and down-voting or arguing about your paper on Twitter and Reddit whether they know what they're talking about, right? So then I have the second order problem of trying to decide whose opinions I should value and such. So, I don't know. I, th- w- what I would, if I had infinite time, which I don't, and I'm not gonna do this because I really wanna make robots work-

   30. LFLex Fridman50:07

       (laughs)
5. 1:00:00 – 1:01:08

   And when you say…

   1. LKLeslie Kaelbling1:00:00

      of learning and not learning. And what should that combination be, and what's the stuff we build in? So, to me that's the most compelling question.

   2. LFLex Fridman1:00:07

      And when you say engineer robots, you mean engineering systems that work in the real world.

   3. LKLeslie Kaelbling1:00:14

      Yeah.

   4. LFLex Fridman1:00:14

      Is that, that, that's the emphasis? Last question. Which robots or robot is your favorite from science fiction? So, you can go with Star Wars or R- R2-D2, or you can go with a more modern, uh, maybe HAL from (laughs) ...

   5. LKLeslie Kaelbling1:00:33

      Uh, I, I don't know, sir, I don't think I have a favorite robot from science fiction. (laughs)

   6. LFLex Fridman1:00:38

      (laughs) This is, this is back to, uh, you, you like to make robots work in the real world here, not, uh, not in...

   7. LKLeslie Kaelbling1:00:45

      I mean, I love the process. And I care more about the process (laughs) -

   8. LFLex Fridman1:00:50

      Oh... The engineering process.

   9. LKLeslie Kaelbling1:00:51

      ... than the end product. Yeah.

   10. LFLex Fridman1:00:51

       Yeah.

   11. LKLeslie Kaelbling1:00:52

       I mean, I do research because it's fun, not because I care about what we produce. (laughs)

   12. LFLex Fridman1:00:57

       (laughs) Well, that's a, that's a beautiful note actually-

   13. LKLeslie Kaelbling1:00:59

       Yeah.

   14. LFLex Fridman1:00:59

       ... to, end on. Leslie, thank you so much for talking today.

   15. LKLeslie Kaelbling1:01:02

       Sure, it's been fun.

   16. LFLex Fridman1:01:02

       All right.

Episode duration: 1:01:23