Chris Urmson: Self-Driving Cars at Aurora, Google, CMU, and DARPA | Lex Fridman Podcast #28

1. 0:00 – 15:00

   The following is a…

   1. LFLex Fridman0:00

      The following is a conversation with Chris Urmson. He was a CTO of the Google self-driving car team, a key engineer and leader behind the Carnegie Mellon University autonomous vehicle entries in the DARPA Grand Challenges, and the winner of the DARPA Urban Challenge. Today, he's the CEO of Aurora Innovation, an autonomous vehicle software company he started with Sterling Anderson, who was the former director of Tesla Autopilot, and Drew Bagnell, Uber's former autonomy and perception lead. Chris is one of the top roboticists and autonomous vehicle experts in the world, and a longtime voice of reason in a space that is shrouded in both mystery and hype. He both acknowledges the incredible challenges involved in solving the problem of autonomous driving and is working hard to solve it. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, give it five stars on iTunes, support it on Patreon, or simply connect with me on Twitter at Lex Fridman, spelled F-R-I-D-M-A-N. And now, here's my conversation with Chris Urmson. You were part of both the DARPA Grand Challenge and the DARPA Urban Challenge teams at CMU with, uh, Red Whittaker. What technical or philosophical things have you learned from these races?

   2. CUChris Urmson1:22

      I think the, the high order bit was that it could be done. I think that was the thing that was incredible about the, first, the, the grand challenges.

   3. LFLex Fridman1:34

      Mm-hmm.

   4. CUChris Urmson1:35

      That I remember, you know, I was a grad student at Carnegie Mellon and there we, it was kind of this dichotomy of it seemed really hard, so that would be cool and interesting. But, you know, at the time, we were the only robotics institute around. And so, you know, if we went into it and fell on our faces, that would, that would be-

   5. LFLex Fridman1:56

      Okay.

   6. CUChris Urmson1:56

      ... embarrassing. Uh, so I think, you know, just having the will to go do it, to try to do this thing that at the time was marked as, you know, darn near impossible. And, and then after a couple of tries, be able to actually make it happen, I think that was, you know, that was really exciting.

   7. LFLex Fridman2:12

      But at which point did you believe it was possible? Did you from the very beginning? Did you personally? 'Cause you were one of the lead engineer, you actually had to do a lot of the work.

   8. CUChris Urmson2:21

      Yeah, I was the technical director there and did a lot of the work (laughs) , uh, along with a bunch of other really good people. Did I believe it could be done? Yeah, of course. Right? Like, why would you go do something you thought was impossible, completely impossible? Uh, we thought it was going to be hard. We didn't know how we were gonna be able to do it. We didn't know if we'd be able to do it the first time. (laughs) Turns out we couldn't. That, yeah, I guess you have to. I th- I think there's a certain benefit to naivete, right? That if you don't know how hard something really is, you, you try different things and, you know, it gives you an opportunity that others who are, you know, wiser maybe don't, don't have.

   9. LFLex Fridman3:04

      Well, what were the biggest pain points? Mechanical, sensors, hardware, software, algorithms for mapping, localization, uh, just general perception control? What, like hardware/software, first of all.

   10. CUChris Urmson3:15

       Well, I, I, I think that's the joy of this field, is that it's all hard.

   11. LFLex Fridman3:19

       (laughs) Yeah.

   12. CUChris Urmson3:19

       Um, and that you have to be good at, at, at each part of it. So for the first, for the urban challenges, uh, if I look back at it from today, uh, it should be easy today. That, you know, it was a static world, there weren't other actors moving through it, th- is what that means. Uh, it was out in the desert, so you get really good GPS. You know, so that, that when, and, you know, we could map it roughly. And so in retrospect now, it's, you know, it's, it's within the realm of things we could do. Back then, just actually getting the vehicle and the, you know, there was a bunch of engineering work to get the vehicle so that we could control it and drive it.

   13. LFLex Fridman4:02

       Mm-hmm.

   14. CUChris Urmson4:04

       That's, you know, that's still a pain today, but it was even more so back then. Uh, and then the uncertainty of exactly what they wanted us to do was, was part of the challenge as well.

   15. LFLex Fridman4:17

       Right, you didn't actually know the track heading in. You, you knew approximately, but you know, didn't actually know the route, the route that was gonna be taken.

   16. CUChris Urmson4:23

       That's right. We didn't know the route. We didn't even r- really, the way the rules had been described, you had to kind of guess. So i- if you think back to that challenge, the idea was to, uh, that the, the government would give us, uh, DARPA would give us, uh, a set of waypoints and kind of the width, um, that you had to stay within between the line that went between, you know, each of those waypoints. And so the, the most devious thing they could have done is set, you know, a kilometer-wide corridor across, you know, a field of scrub brush, um, and rocks and said, you know, "Go figure it out." Uh, fortunately, it really, it turned into basically driving along a set of trails, which, you know, is much more relevant to, to the application they were looking for. But no, it was, it was a hell of a thing back in the day.

   17. LFLex Fridman5:12

       So, uh, the legend, Red, was, uh, kind of leading that effort-

   18. CUChris Urmson5:16

       Yeah.

   19. LFLex Fridman5:16

       ... uh, in terms of just broadly speaking. So you're a leader now. What have you learned from Red about leadership?

   20. CUChris Urmson5:25

       I, I think there's a couple things. One is, you know, go and try those really hard things. That, that's where there is a, an incredible opportunity. Uh, I think the other big one though is to see people for who they can be, not who they are. It, it's one of the things that I actually, one of the deepest lessons I, I learned from Red was that he would look at, um, you know, undergraduates or graduate students and empower them, uh, to be leaders-

   21. LFLex Fridman5:56

       Mm-hmm.

   22. CUChris Urmson5:56

       ... to, to, you know, have responsibility, to do great things, that I think...... another person might look at them and think, "Oh, well, that's just, you know, an undergraduate student. What, what could they know?" And so I think that, that, you know, kind of trust but verify, have confidence in what people can become, I think is, is a really powerful thing.

   23. LFLex Fridman6:16

       So through that, let's just, like, fast-forward through the history. Can you maybe talk through the technical evolution of autonomous vehicle systems from the first two Grand Challenges to the Urban Challenge to today?

   24. CUChris Urmson6:30

       Sure.

   25. LFLex Fridman6:31

       Are there major shifts in your mind or is it the same kind of technology, just made more robust?

   26. CUChris Urmson6:37

       I think there's been some big, big steps. So the, for the Grand Challenge, the real technology that unlocked that was HD mapping.

   27. LFLex Fridman6:51

       Mm-hmm.

   28. CUChris Urmson6:51

       Prior to that, a lot of the off-road robotics work had been done without any real prior model of what the vehicle was going to encounter. And so that innovation, that... The fact that we could get, you know, decimeter resolution models was really a, a big deal. And, and that allowed us to, to kind of bound the complexity of the driving problem the vehicle had and allowed it to operate at speed, because we could assume things about the environment that it was going to encounter. So that was a, that was one of the... That was the big step there. For the Urban Challenge, you know, one of the big technological innovations there was the multi-beam LIDAR, and being able to generate, um, high-resolution, you know, mid-to-long range 3D models of the world and use that for, you know, f- for understanding the world around the vehicle. And that was really a, you know, kind of a game-changing technology. In parallel with that, we saw a bunch of other technologies that had been kind of converging, half their, their day in the sun. So, uh, Bayesian estimation-

   29. LFLex Fridman8:11

       Mm-hmm.

   30. CUChris Urmson8:12

       ... uh, had been... You know, SLAM had been a big field i- in robotics. You know, you, you would go to a conference, you know, a couple of years before that and, you know, every paper would effectively have SLAM somewhere in it.
2. 15:00 – 30:00

   In terms of sensors,…

   1. CUChris Urmson15:00

      can get into vehicles and drive them, and, and so there's an existence proof that you can drive using, you know, passive vision, no doubt, can't argue with that.

   2. LFLex Fridman15:12

      In terms of sensors, yeah, so there's proof that-

   3. CUChris Urmson15:15

      Yeah, in terms of sensors, right? So like, there's, there's an example that, you know, we all go do it, uh, many of us every day. In terms of, uh, Lidar being a crutch, sure.

   4. LFLex Fridman15:28

      (laughs) .

   5. CUChris Urmson15:28

      But, but, you know, in the same way that, uh, you know, the combustion engine was a crutch on the path to an electric vehicle, in the same way that, you know, any technology ultimately gets replaced by some superior technology in the future. Uh, and really, what, the way that I look at this is that the way we get around on the ground, the way that we use transportation is broken, um-

   6. LFLex Fridman15:54

      Hmm.

   7. CUChris Urmson15:55

      ... and that we have, you know, this, this, you know, what was... I think the number I saw this morning, 37,000 Americans killed, uh, last year on our roads, and that's just not acceptable. And so te- any technology that we can bring to bear that accelerates the, this techno- you know, self-driving technology coming to market and saving lives is technology we should be using. And it feels just arbitrary to say, "Well, you know, I'm, I'm not okay with using lasers because that's whatever, but I am okay with using an eight megapixel camera or a 16 megapixel camera." You know, like, it's just, these are just bits of technology and we should be taking the best technology from the tool bin that allows us to go and, you know, and solve a problem.

   8. LFLex Fridman16:41

      The question I often talk to, uh, well, obviously you do as well, to, uh, the sort of automotive companies, and, you know, if, if there's one word that comes up more often than anything, it's cost and, and, uh-

   9. CUChris Urmson16:53

      Yeah.

   10. LFLex Fridman16:53

       ... tr- trying to drive cost down. So while it's, it's true that it's, um, it's a tragic number, the 37,000, the, the question is what... (laughs) And I'm not the one asking this question, 'cause I hate this question, but-

   11. CUChris Urmson17:06

       Yeah.

   12. LFLex Fridman17:06

       ... we, we ha- we want to find the cheapest sensor suite that, uh, that creates a safe vehicle.

   13. CUChris Urmson17:13

       Yep.

   14. LFLex Fridman17:13

       So in that, uh, uncomfortable trade-off, do you foresee Lidar, uh, coming down in cost in the future or do you see a day where level four autonomy is possible without Lidar?

   15. CUChris Urmson17:29

       I, I see both of those, but it's really a matter of time. And I, and I think really maybe the- I, I would talk to the question you asked about, you know, the cheapest sensor.

   16. LFLex Fridman17:37

       Mm-hmm.

   17. CUChris Urmson17:38

       I don't think that's actually what you want. What you want is a sensor suite that is economically viable, and then after that, everything is about margin and driving cost out of the system. Uh, what you also want is a sensor suite that works.

   18. LFLex Fridman17:54

       Right.

   19. CUChris Urmson17:55

       And so it's great to tell a story about, um, how, you know, how it'd be better to have a self-driving system with a $50 sensor instead of a, you know, a $500 sensor. But if the $500 sensor makes it work and the $50 sensor doesn't work-... you know, who cares? So long as you, you can actually, uh, you know, have an economic oppor- you know, there's an economic opportunity there. And the economic opportunity is important because that's how you actually have a, a sustainable business, and, and that's how you can actually see this come to scale and, and, and be out in the world. And so when I look at LiDAR, I see a technology that has no underlying fundamentally, you know, expense to it, fundamental expense to it. It's, it's going to be more expensive than, uh, an imager because, you know, CMOS processes are, or, you know, fab processes are, are dramatically more scalable than mechanical processes, but we still should be able to drive cost down substantially on that side. Uh, and then I also do think that, uh, with the right business model, you can absorb more, you know, certainly more cost on the bill of materials.

   20. LFLex Fridman19:09

       Yeah, if the sensor suite works, extra value is provided, thereby you don't need to drive cost down to zero. It's the basic economics. You've talked about your intuition that level two autonomy is problematic because of the human factor, uh, of vigilance, decrement, complacency, over-trust and so on, just us being human.

   21. CUChris Urmson19:29

       Yeah.

   22. LFLex Fridman19:29

       We over-trust the system, we start doing even more, so partaking in the secondary activities like smartphone and so on. Have your views evolved on this point in either direction? Can you, can you speak to it?

   23. CUChris Urmson19:44

       So... And I want to be really careful because sometimes this gets twisted in a way that's, that, that, that I certainly didn't intend. So active safety systems are a really important technology that we should be pursuing and integrating into vehicles. And there's an opportunity in the near term to reduce accidents, reduce fatalities and that's, uh, and we should be, we should be pushing on that. Level two systems are systems where the vehicle is controlling two axes, so, you know, breaking and s- breaking and throttle/steering. And I think there are variants of level two systems that are supporting the driver that absolutely, like, we should, we should encourage to be out there. Um, where I think there's a real challenge is in the, the human factors part around this and the misconception from the public around the capability set that that enables and the, and the trust that they should have in it. And that is where I, you know, I, I kind of, I, I, I'm actually incrementally more, you know, concerned around level three systems and, you know, how exactly a level two system is marketed and delivered, uh, and, you know, how people... how much effort people have put into those human factors. So I still believe, uh, uh, several things around this. One is people will over-trust the technology. Uh, we've seen over the last few weeks, you know, a spade of people sleeping in their Tesla. You know, I watched an episode last night of, um, Trevor Noah-

   24. LFLex Fridman21:21

       Mm-hmm.

   25. CUChris Urmson21:22

       ... talking about this. And, you know, him, you know, this is a smart guy who's... has a lot of resources at his disposal, describing a Tesla as a self-driving car, and that, why shouldn't people be sleeping in their Tesla? It's like, well, because it's not a self-driving car and it is not intended to be, and, you know, these people will almost certainly, you know, die at some point or, or hurt other people. And so we, we need to really be thoughtful about how that technology is described and, and brought to market. I also think that because of the economic issue, you know, econom- economic challenges we were just talking about, that that technology path will alt- the- these level two driver assistance systems, that technology path will diverge from the technology path that we need to be on to actually deliver truly self-driving vehicles, ones where you can get in it and sleep and have the, uh, equivalent or better safety than, you know, a, a human driver behind the wheel. Um, because the... again, the economics are very different in those two worlds. And so that leads to, you know, divergent technology.

   26. LFLex Fridman22:32

       So you, you just don't see the economics of gradually increasing from level two and doing so quickly enough to where it doesn't cause safety, uh, critical safety concerns? You, you believe that, that it needs to diverge at this point, uh, into different-

   27. CUChris Urmson22:49

       Yeah.

   28. LFLex Fridman22:49

       ... ba- basically different routes of-

   29. CUChris Urmson22:50

       And, and really that comes back to what are those L2 and L1 systems doing? And, and they are driver assistance functions where the, the, the people that are marketing that responsibly are being very clear and putting human factors in place, such that the driver is actually responsible for the vehicle and that the technology is there to support the driver. And the safety cases that are, are built around those are dependent on that driver attention, uh, and attentiveness. Uh, and at that point, you, you can kind of give up, to some degree... For economic reasons, you can give up on, say, false negatives. Uh, and so... And the, and the way to think about this is for a forward collision mitigation braking system, if it... half the times the driver missed a vehicle in front of it, uh, it hit the brakes and brought the vehicle to a stop, that would be an incredible, incredible advance in, in safety on our roads, right? That would be equivalent to seat belts. Uh, but it would mean that if that vehicle wasn't being monitored, it would hit one out of two cars. And so economically, that's a perfectly good solution for a driver assistance system. What you should do at that point, if you can get it to work 50% of the time, is drive the cost out of that so you can get it on as many vehicles as possible. But driving the cost out of it doesn't drive up-... performance on the false negative case. And so you'll continue to not have a technology that could, you know, really be available for, for a self-driven vehicle.

   30. LFLex Fridman24:25

       So, clearly, the communication, and this probably applies to all four vehicles as well, the, uh, marketing and the communication of what the technology is actually capable of, how hard it is, how easy it is, all that kind of stuff-
3. 30:00 – 44:32

   Yeah. Do you think…

   1. CUChris Urmson30:00

      thought this was not a great metric.

   2. LFLex Fridman30:01

      Yeah. Do you think it's possible to create a metric, a number, that, um, that could demonstrate safety outside of fatalities?

   3. CUChris Urmson30:12

      So, so I, I do. And, and I think that it, it won't be just one number.

   4. LFLex Fridman30:16

      Mm-hmm.

   5. CUChris Urmson30:17

      So as we are internally grappling with this, and, and at some point we'll be, we'll be able to talk pub- more publicly about it, is how do we think about human performance in, in different tasks? Say, detecting traffic lights or, um, safely making a left turn across traffic, and what do we think the failure rates are for those different capabilities for people? And then demonstrating to ourselves and then ultimately, uh, folks in regulatory role and, and then ultimately the public, that we have confidence that our system will work better than that. Uh, and so these, these individual metrics will kind of tell an- a com- a compelling story, ultimately. I do think, at the end of the day, what we care about i- in terms of safety is, uh, lives saved, uh, and injuries reduced, and then, and then ultimately, you know, kind of casualty dollars that people aren't having to pay to, to get their car fixed. And I do think that you can... You know, w- i- in aviation they look at a, a kind of an event pyramid, where, you know, a c- a crash is at the top of that and that's the worst event, obviously, and then there's injuries and, you know, near miss events and whatnot, and, and, you know, violation of operating procedures. And, and you kind of build a, a statistical model of, of, uh, the relevance of the, the low severity things to the high severity things, and I think that's something we will be able to look at as well. Uh, because, you know, an event per 85 million miles is a- you know, statistically a, a difficult thing, even at the scale of the US, um, to, to, to kind of compare directly.

   6. LFLex Fridman31:59

      And that event, the fatality that's connected to an autonomous vehicle, is significantly, at least currently, magnified-

   7. CUChris Urmson32:09

      Yeah.

   8. LFLex Fridman32:09

      ...in, uh, th- the amount of, um, attention it gets. So that speaks to public perception.

   9. CUChris Urmson32:14

      Yeah.

   10. LFLex Fridman32:15

       I think the most popular topic about autonomous vehicles, in the public, is, um, the trolley problem formulation, right?

   11. CUChris Urmson32:22

       Sure.

   12. LFLex Fridman32:23

       Which, uh, has... Let's not get into that too much, but, uh, is misguided b- in many ways. But it speaks to the fact that people are grappling with this idea of giving control over to a machine. So how do you win the, the hearts and minds of the people that, uh, autonomy is something that could be a part of their lives?

   13. CUChris Urmson32:45

       I think you let them experience it.

   14. LFLex Fridman32:47

       Oh.

   15. CUChris Urmson32:47

       Right? I, I think it's... I think, I think it's right. I think people should be skeptical. Uh, I think people should, um, ask questions. I think they should doubt. Because this is something new and different. They haven't touched it yet, and I think that's perfectly reasonable. And... But at the same time, it's clear there's an opportunity to make the roads safer. It's clear that we can improve access to mobility. It's clear that we can reduce the cost of mobility. And that once people try that and are... You know, understand that it's safe, and, and are able to use it in their daily lives, I think it's one of these things that will, will just be obvious. And, and I've seen this practically in... You know, in demonstrations that I've, you know, given, where I've had people come in and, you know, they're very skeptical and they, they get in the vehicle. And, you know, my favorite one is taking somebody out on the freeway, and we're on the 101 driving at 65 miles an hour, and after 10 minutes they, they kind of turn and ask, "Is that all it does?" (laughs) And you're like-

   16. LFLex Fridman33:51

       Yeah.

   17. CUChris Urmson33:51

       ..."It's a self-driving car. I'm not sure exactly what you thought it would do."

   18. LFLex Fridman33:53

       Yeah, it just drives. Yeah.

   19. CUChris Urmson33:53

       Right? Um, but they, you know, they, they... It becomes mundane-

   20. LFLex Fridman33:58

       Right.

   21. CUChris Urmson33:58

       ...which is, which is exactly what you want a technology like this t- to be, right? We don't really... When I turn the light switch on in here, I don't think about the complexity of, you know, the-

   22. LFLex Fridman34:10

       Yeah.

   23. CUChris Urmson34:10

       ... those electrons, you know, being pushed down a wire from wherever it was and being generated some- Like, it's just... It's like I just get annoyed if it doesn't work, right? And, and what I value is the fact that I can do other things in this space. I can, you know, see my colleagues, I can read stuff on a paper, I can, you know, uh, not be afraid of the dark.

   24. LFLex Fridman34:30

       Perf-

   25. CUChris Urmson34:30

       And, and I think that's what we want this technology to be like, is it's, it's in the background and people get to have those, those life experiences and, and do so safely.

   26. LFLex Fridman34:38

       So putting this technology in the hands of people speaks to s- c- scale of deployment, right? So what do you think... The, uh, the dreaded question about the future, because nobody can predict the future.

   27. CUChris Urmson34:52

       Yeah.

   28. LFLex Fridman34:53

       But just maybe, uh, speak poetically about when do you think we'll see a large-scale deployment of autonomous vehicles? 10,000, th- those kinds of numbers.

   29. CUChris Urmson35:06

       U- We'll see that within 10 years. I'm, I'm pretty confid-... I... We, um...

   30. LFLex Fridman35:14

       What's an impressive scale? W- what moment... Uh, so you've, you've done the DARPA Challenge with this one vehicle.

Episode duration: 44:47