Lex Fridman PodcastSertac Karaman: Robots That Fly and Robots That Drive | Lex Fridman Podcast #97
EVERY SPOKEN WORD
150 min read · 30,077 words- 0:00 – 1:44
Introduction
- LFLex Fridman
The following is a conversation with Sertac Karaman, a professor at MIT, co-founder of the autonomous vehicle company Optimus Ride, and is one of the top roboticists in the world, including robots that drive and robots that fly. To me personally, he has been a mentor, a colleague, and a friend. He's one of the smartest, most generous people I know, so it was a pleasure and honor to finally sit down with him for this recorded conversation. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, review it with five stars on Apple Podcasts, support it on Patreon, or simply connect with me on Twitter @LexFridman, spelled F-R-I-D-M-A-N. As usual, I'll do a few minutes of ads now and never any ads in the middle that can break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. When you get it, use the code LEXPODCAST. Cash App lets you send money to friends, buy Bitcoin, and invest in the stock market with as little as $1. Since Cash App allows you to send and receive money digitally, let me mention a surprising fact about physical money: it costs 2.4 cents to produce a single penny. In fact, I think it costs $85 million annually to produce them. That's a crazy little fact about physical money. So again, if you get Cash App from the App Store or Google Play and use the code LEXPODCAST, you get $10, and Cash App will also donate $10 to FIRST, an organization that is helping to advance robotics and STEM education for young people around the world. And now here's my conversation with Sertac Karaman.
- 1:44 – 6:37
Autonomous flying vs autonomous driving
- LFLex Fridman
Since you have worked extensively on both, what is the more difficult task, autonomous flying or autonomous driving?
- SKSertac Karaman
That's a good question. I think that autonomous flying, just kind of doing it for consumer drones and so on, the kinds of applications that we're looking at right now, is probably easier. And so I think that that's maybe one of the reasons why it took off, like literally a little earlier than the autonomous cars. But I think if we look ahead, I would think that, you know, the real benefits of autonomous flying, unleashing them in like transportation, logistics, and so on, I think it's a lot harder than autonomous driving. So I think my guess is that, you know, we've seen a few kinda machines fly here and there, but we really haven't yet seen any kind of, you know, machine like, like at massive scale, large scale being deployed and flown, and so on. And I think that's gonna be after we kind of resolve some of the large-scale deployments of autonomous driving.
- LFLex Fridman
So what's the hard part? What-what's your intuition behind why at scale when consumer-facing drones are tough?
- SKSertac Karaman
So I think, in general, at scale, it's tough. Like for example, when you think about it, um, we have actually deployed a lot of robots in the, let's say, the past 50 years.
- LFLex Fridman
We as academics or we business entrepreneurs?
- SKSertac Karaman
Um, I think we as humanity.
- LFLex Fridman
Humanity?
- SKSertac Karaman
Let's put it that way.
- LFLex Fridman
Okay.
- SKSertac Karaman
A lot of people working on it. (laughs)
- LFLex Fridman
We humans. (laughs) Okay. Good.
- SKSertac Karaman
So we humans deployed a lot of robots, and I think that... But when you think about it, you know, robots, um, they're autonomous. They work. Uh, they work on their own, but they are either like in isolated environments or they are in sort of, um, you know, um... They may be at scale, but they're really confined to a certain environment that they don't interact so much with humans. And so, you know, they work in, I don't know, factory floors, warehouses. They work on Mars, you know. They are fully autonomous over there. Um, but I think that the real challenge of our time is to take these vehicles and put them into places where humans are present. So now I know that there's a lot of like human-robot interaction, um, type of things that need to be done and so... And that's, that's one thing, but even just from the fundamental algorithms and systems and, and the business cases or maybe the business models, even like architecture, planning, societal issues, legal issu- There's a whole bunch of pack of things that are related to us putting robotic vehicles into human-present environments. And as humans, you know, they will not potentially be even trained to interact with them. They may not even be using the services that are provided by these vehicles. They may not even know that they're autonomous. They're just doing their thing, living in environments that are designed for humans, not for robots. And that, I think, is one of the big, biggest challenges, I think, of our time to put vehicles there. And, you know, to go back to your question, I think, uh, doing that, um, at scale, meaning, you know, you go out in a city and, and you have, you know, like thousands or tens of thousands of autonomous vehicles that are going around, it is so dense to the point where if you see one of them, you look around, you see another one.
- LFLex Fridman
Yeah.
- SKSertac Karaman
It is that dense. And that density, we've never done anything like that before. And I, I would, I would bet that that kind of density will first happen with autonomous cars, because I think, you know, we can bend the environment a little bit. We can... Especially kind of, um, making them safe is a lot easier when they're like on the ground. Um, when they're in the air, it's a little bit more complicated. Um, but I don't see that there's gonna be a big separation. I think that, you know, there will come a time that we're gonna quickly see these things unfold.
- LFLex Fridman
Do you think there will be a time where there's tens of thousands of delivery drones that fill the sky?
- SKSertac Karaman
You know, I think, I think it's possible, to be honest. Delivery drones is one thing, but, you know, you can imagine, um, for transportation, like a, like an important use case is, you know, we're in Boston. You wanna go from Boston to New York, and you wanna do it from the top of this building to the top of another building in Manhattan, and you're gonna do it in one and a half hours. And that's, that's a big opportunity, I think.
- LFLex Fridman
Personal transport. So like you and maybe a friend, like almost like an Uber.
- SKSertac Karaman
Um... Yeah. Or almost like a, like a, like an Uber. So like four people, six people, eight people. In our work in autonomous vehicles, I, I see that. So there's kind of like a bit of a need for, you know, one-person transport, but also like, like a few people. So you and I could take that trip together. We could have lunch. Um-... that, you know, I think kind of sounds crazy, maybe even sounds a bit cheesy. But I think that those kinds of things are, are some of the real opportunities. And I think, you know, um, i- i- it's not like the typical airplane and the airport would disappear very quickly, but I would think that, you know, um, many people would feel like they would spend an extra $100 on doing that and cutting that four-hour travel down to one and a half
- 6:37 – 10:27
Flying cars
- SKSertac Karaman
hours.
- LFLex Fridman
So how feasible are flying cars? It's been the dream. It's like when people imagine the future for 50 plus years-
- SKSertac Karaman
Mm-hmm.
- LFLex Fridman
... they think flying cars. It's, uh, it's like all technology, it's cheesy to think about now because it seems so far away, but, eh, overnight it can change. But just technically speaking, in your view, how feasible is it to make that happen?
- SKSertac Karaman
I'll, I'll get to that question. But just one thing is that I think, you know, sometimes we think about, um, what's gonna happen in the next 50 years. It's just really hard to guess, right? Next 50 years, I don't know. I mean, we could get... What's gonna happen in transportation in the next 50 years? We could get flying saucers. I, I could bet on that. I think there's a 50/50 chance that, you know, like you can build machines that can ionize the air around them and push it down with magnets and they would fly, like a flying saucer.
- LFLex Fridman
Yeah.
- SKSertac Karaman
That is possible.
- LFLex Fridman
(laughs)
- SKSertac Karaman
Um, and it might happen in the next 50 years, so it's a bit hard to guess, like when you think about 50 years before.
- LFLex Fridman
Yeah.
- SKSertac Karaman
But I would think that, you know, um, there, there's this, th- this kind of, uh, notion, uh, there's a certain type of airspace that we call the agile airspace and there is, there's good amount of opportunities in that airspace. So that would be the space that is kind of a little bit higher than the place where you can throw a stone, because that's a tough thing when you think about it. You know, it takes a kid and a stone to take an aircraft down and then what happens?
- LFLex Fridman
Yeah.
- SKSertac Karaman
Um, but, you know, imagine the airspace that's high enough so that you cannot throw the stone, but it is low enough that you're not interacting with the, with the very large aircraft, um, that are, you know, flying, um, several thousand feet above and that airspace is underutilized or it's actually kind of not utilized at all.
- LFLex Fridman
Yeah, that's right. What's up there?
- SKSertac Karaman
So there's, you know, there's like recreational people kind of fly every now and then, but it's very few. Like if you look up in the sky, you may not see any of them at any given di- time. Every now and then you'll see one airplane kind of utilizing that space and you'll be surprised. And the moment you're outside of an airport a little bit, like it just kind of flies off and then it goes out. And I think utilizing that airspace, um, the technical challenges there is, you know, um, building an autonomy and ensuring that that kind of autonomy is safe. Um, ultimately, I think it is going to be building in complex software, uh, complicated so that it's maybe a few orders of magnitude more complicated than what we have on aircraft today. And at the same time ensuring, just like we ensure on aircraft, ensuring that it's safe. And so that becomes like building that kind of complicated hardware and software becomes a challenge. Especially when, you know, you build that hardware, I mean, you build that software with data. And so, um, you know, it's, it's... Of course, there's some rule-based software in there that kind of do a certain set of things, but, but then, you know, there's a lot of training there to ............................
- LFLex Fridman
Do you think machine learning will be key to these kinds-
- SKSertac Karaman
Um-
- LFLex Fridman
... to, to delivering safe vehicles in the future, uh, especially flight?
- SKSertac Karaman
Um, not maybe the safe part, but I think the intelligent part. Um, I mean, there are certain things that we do it with machine learning and it's just there's like right now no other way. And, and I don't, I don't know how e- how else they could be done. And, um, you know, there's, there's always this conundrum. I mean, we could... Like could we... E- e- like we could maybe gather billions of programmers-
- LFLex Fridman
Mm-hmm.
- SKSertac Karaman
... uh, humans who program perception algorithms that detect things in the sky and whatever. Or, you know, we w- I don't know, we maybe even have robots like learning a simulation environment and transfer, and they might be learning a lot better in a simulation environment than a billion humans put their brains together and try to program. Human is pretty limited.
- 10:27 – 17:35
Role of simulation in robotics
- SKSertac Karaman
- LFLex Fridman
So what's, uh, what's the role of simulations with drones? You've, you've done quite a bit of work there. How promising... Just the very thing you said just now, h- how promising is the possibility of training and developing a, a safe flying robot in simulation and then deploying it and having that work pretty well in the real world?
- SKSertac Karaman
I think that, you know, a lot of people when they hear simulation, they will focus on training immediately. But I think one thing that you said, which was interesting is developing. I think simulation environments, uh, actually could be key and great for development, and that's not new. Uh, like for example, um, you know, there's, um... People in the automotive industry have been using dynamic simulation for like decades now. And, and it's pretty standard that, you know, you would build and you would simulate. If you wanna build an embedded con- controller, you plug that kind of embedded computer into another computer. That other computer would simulate dynamic and so on. And I think, you know, fast-forward these things, you can create pretty crazy simulation environments. Um, like for instance, um, one of the things that has happened recently, um, and that, you know, we can do now is that we can simulate cameras a lot better than we used to simulate them, we were able to simulate them before. And that's... I, I think we just hit the elbow on that kind of improvement. I would imagine that, um, with improvements in hardware especially, um, and with improvements in machine learning, I think that we will get to a point where we can simulate cameras very, very well.
- LFLex Fridman
Simulate cameras means simulate how a real camera would see the real world, therefore you can explore the limitations of that. You can train perception algorithms on that in simulation, all that kind of stuff.
- SKSertac Karaman
Exactly. So, you know-... it's, it's, it has been easier to simulate what we would call interoceptive sensors, like internal sensors. So for example, inertial sensing has been easy to simulate. It has also been easy to simulate dynamics, like, um, like physics that are governed by ordinary differential equations. I mean, li- like how a car goes around, maybe how it rolls on the road, how it interact with, w- it interacts with the road, or even an aircraft flying around, like the dynamic, the physics of that. What has been really hard has been to simulate exteroceptive sensors, sensors that kind of like look out from the vehicle, and that's a new thing that's coming. Like, laser range finders that are a little bit easier, cameras, radars are a little bit tougher. I think once we nail that down, the, the next challenge, I think, in simulation will be to simulate human behavior. That's also extremely hard.
- LFLex Fridman
Hmm.
- SKSertac Karaman
Even when you imagine, like, how a human-driven car would act around, even that is hard, but imagine trying to simulate, you know, a, a, a model of a human just doing a bunch of gestures and so on and, and, you know, it's, it's actually simulated. It's not captured like with motion capture, but it is simulated. That's, that's very hard. In fact, today, I get involved a lot with, like, sort of this kind of very high-end rendering projects, and I have, like, this test that I pass it to my friends or my mom, you know, I send, like, two photos, two, kinda, pictures and I say, "Rendered." Which one is rendered? Which one is real?" And it's pretty hard to distinguish, except I realize, except when we put humans in there. It's possible that our brains are trained in a way that we recognize humans extremely well, but we don't so much recognize the built environments because built environments sort of came after, per se, we evolved into sort of being humans, but, but humans were always there. Same thing happens, for example, you look at, like, monkeys and you can't distinguish one from another, but they sort of do. And it's very possible that they look at humans, it's kinda pretty hard to distinguish one from another, but we do. And so our eyes are pretty well-trained to look at humans and understand, if something is off, we will get it. We may not be able to pinpoint it. So in my typical friend test or mom test, what would happen is that we'd put like a human walking in a, in a, in a, in a thing, and they, they say, you know, "This is not right. Something is off in this video. I don't know what, but I, but I can tell it's the human." I can take the human and I can show you, like, inside of a building or, like, an apartment and it will look like, if we had time to render it, it would look great. And this should be no surprise. A lot of movies that people are watching, it's all computer-generated. You know, even nowadays, even you watch a drama movie and, like, there's nothing going on action-wise, but it turns out it's kinda, like, cheaper, I guess, to render the background, and so they would.
- LFLex Fridman
But, uh, how do, how do we get there? How do we get a, a human that's... would pass the mom/friend test? A simulation of a human walking? So, d- do you think that's something we can creep up to by just doing kind of a comparison learning, where you have humans annotate what's more realistic and not just by watching? Like, what- what's the path? 'Cause it seems totally mysterious how we-
- SKSertac Karaman
I think you're right.
- LFLex Fridman
... simulate human behavior. (laughs)
- SKSertac Karaman
Um, it's, it's, it's hard because a lot of the other things that I mentioned to you, including simulating cameras, right? It is, um... The, the thing there is that, you know, we know the physics. We know how it works, like, in the real world, and we can write some rules and we can do that. Like, for example, simulating cameras, there's this thing called ray tracing. I mean, you literally just kind of imagine... It's very similar to, it's not exactly the same, but it's very similar to tracing photon by photon, they're going around bouncing on things and coming to your eye. But human behavior, um, developing a dynamic, like, like, like a model of that, that is mathematical so that you can put it into a, a processor that would go through that, that's gonna be hard. And so, so what else do you got? You can collect data, right? And you can try to match the data. Or another thing that you can do is that, you know, you can show the friend test, you know, you can say this or that and this or that, and that would be labeling. Anything that requires human labeling, ultimately we're limited by the number of humans that, you know, (laughs) we have available at our disposal, and the things that they can do. You know? They have to do a lot of other things than also labeling this data. So, um, so that... Modeling human behavior part is, is I think going, we're gonna realize it's very tough. And I think that also affects, you know, um, our development of autonomous vehicles. I see them self-driving as well, like, we wanna use... So you're building self-driving, you know. At the first time, like, right after Urban Challenge, I think, everybody focused on localization, mapping and localization. You know, as SLAM algorithms came in, Google was just doing that, and so building these HD maps. Basically that's about knowing where you are. And then five years later in 2012, 2013, came the kind of quote-unquote "AI revolution," and that started telling us where everybody else is, but we're still missing what everybody else is gonna do next. And so you wanna know where you are, you wanna know what everybody else is. Hopefully you know about what you're gonna do next, and then you wanna predict what other people are going to do, and that last bit has, has been a real, um, real challenge.
- 17:35 – 24:30
Game theory and robotics
- SKSertac Karaman
- LFLex Fridman
What, what do you think is the role, your own, of your, of your, the ego vehicle, the robot you? The, the you, the robotic you, in controlling and having some control of how the future unrolls, of what's gonna happen in the future? That seems to be a little bit ignored in trying to predict the future, is how you yourself can affect that future by being either aggressive or less aggressive or signaling in some kind of way. Sort of this kinda game theoretic dance seems to be ignored for the moment, for the most part.
- SKSertac Karaman
It's, yeah, it's, it's totally ignored. I mean, it's, it's quite interesting actually, like, how we, um, um, how we interact with things versus we interact with humans.... like, so if, if you see a vehicle that's c- completely empty and it's trying to do something, all of a sudden it becomes a thing, so interacted with like you interact with this table. And so you can throw your backpack, or you can kick your, kick it, put your feet on it, and things like that. But when it's a human, there's all kinds of ways of interacting with a human. So if, you know, like, you and I are face to face, we're very civil, you know, we talk-
- LFLex Fridman
For the most part.
- SKSertac Karaman
... and we understand each other, for the most part. We'll see just (overlapping conversation) (laughs) ... as that, so you never know-
- LFLex Fridman
(laughs)
- SKSertac Karaman
... what's gonna happen. But, but the thing is that, like for example, you and I might interact through YouTube comments, and, you know, the conversation may go a totally different angle. And so I think, um, uh, people kind of abusing these autonomous vehicles is, is a real issue in some sense. And so when you're an ego vehicle, you're trying to, you know, coordinate your way, make your way, it's actually kind of harder than being a human, you know? It's like, it's you, you, you not only need to be as smart as, as, kind of, humans are, but you also, you're a thing, so, and they're gonna abuse you a little bit, so you, you need to make sure that you, you can get around a- and do something. So, um, uh, uh, I, I in general believe in that sort of gain theoretic aspects. I've actually personally have done, you know, quite a few papers both on that kind of game theory and also, like, this, this kind of understanding people's social value orientation, for example. You know? Some people are aggressive. Some people not so much. And, and, you know, like, a, a robot could understand that by just looking at how people drive. And as they kind of come and approach, you can actually understand, like, if someone is gonna be aggressive or, or not as a robot, and you can make certain decisions.
- LFLex Fridman
Well, in terms of predicting what they're going to do, the hard question is, you as a robot, should you be aggressive or not when faced with a, with an aggressive robot? Right now, it seems like aggressive is a very dangerous thing to do because, uh, it's costly from a societal perspective, how you're perceived. People are not very accepting of aggressive robots in modern society.
- SKSertac Karaman
I think that's accurate, so (laughs) th- it really is. And so I- I'm not entirely sure, like, how to, how to go about, but it, I know, I know for a fact that how these robots interact with other people in there is going to be... And, and that interaction is always gonna be there. I mean, it, you could be interacting with other vehicles, or other just people kind of, like, walking around. Um, and like I said, the moment there's, like, nobody in the seat, it's like an empty thing just rolling off the street. It, it becomes, like, no different than, like, any other thing-
- LFLex Fridman
Hmm.
- SKSertac Karaman
... that's not human, and so, so people... And maybe abuse is the wrong word, but, you know, people ma- maybe rightfully even, they, they feel like, you know, this is a human present environment, it's designed for humans to be, and, and they, they kind of, they wanna own it. Um, and then, you know, the robots, they would, they would need to understand it, and they would need to respond in a certain way. And I think that, you know, this actually opens up, like, quite a few interesting societal questions for us as we deploy, like we talk, robots at large scale. So what would happen when we try to deploy robots at large scale I think is that we can design systems in a way that they're very efficient, or we can design them that they're very sustainable. But ultimately, the sustainability-efficiency trade-offs, like, they're gonna be right in there, and we're gonna have to make some choices. Like, we're not gonna be able to just kinda put it aside. So for example, we can be very aggressive, and we can reduce transportation delays, increase capacity of transportation, or, you know, we can, we can be a lot nicer and allow other people to kind of, quote-end quote, "own the environment and, and, and live in a nice place," and then efficiency will drop. So when you think about it, I think sustainability gets attached to energy consumption or the environmental impact immediately, and those are, those are there, but, like, livability is another sustainability impact. So you create an environment that people wanna live in. And if, if, if robots are going around being aggressive, uh, you don't wanna live in that environment maybe. Um, however, you should note that if you're not being aggressive, then, you know, you're probably taking up some, some delays in transportation and, and this and that. So you're always balancing that. And I think this, this choice has always been there in transportation, but I think the more autonomy comes in, the more explicit the choice becomes.
- LFLex Fridman
Yeah, and when it becomes explicit, then we can start to optimize it, and then we get to ask the very difficult societal questions of what do we value more, efficiency or sustainability. It's kind of interesting-
- SKSertac Karaman
Yeah, I think that will happen.
- LFLex Fridman
(laughs)
- SKSertac Karaman
I, I think we're gonna have to, like... I, I think that the, the interesting thing about, like, the whole autonomous vehicles question I think is also kind of, um... I think a lot of times, you know, we have, we have focused on technology development, like, hundreds of years, and, you know, the products somehow followed, and, and then, you know, we got to make these choices and things like that. But this is, this is a good time that, you know, we even think about, you know, autonomous taxi type of deployments and the systems that would evolve from there, and you realize the business models are different. The impact on architecture is different, urban planning. You get into, like, regulations, um, and then you get into, like, these issues that you didn't think about before about, like, sustainability and ethics is, like, right in the middle of it. I mean, even testing autonomous vehicles, like, think about it, you're testing autonomous vehicles in human present environments. I mean, uh, the risk may be very small, but still, you know, it's, it's, uh, it's, um, it's, it's a, you know, strictly greater than zero risk that you're putting people into. And so then you have that innovation, you know, risk trade-off that you're, you're in that somewhere. Um, and we, we understand that pretty now that, pretty well now is that if we don't test, the, at least the, the development will be slower. I mean, uh, it doesn't mean that we're not gonna be able to develop. I, I think it's gonna be pretty hard actually. Maybe we can, maybe we don't, we don't... I don't know, but, but the thing is that those kinds of trade-offs we already are making, and as these systems become more ubiquitous, I think those trade-offs will just really hit-
- 24:30 – 29:46
Autonomous vehicle company strategies
- LFLex Fridman
So, you are one of the founders of Optimus Ride, an autonomous vehicle company. We'll talk about it, but let me, on that point, ask, maybe a good examples, keeping Optimus Ride out- out- out of this question, uh, sort of exemplars of different strategies on the spectrum of, uh, innovation and safety or caution. So, like, Waymo, Google self-driving car, Waymo represents maybe a more cautious approach, and then you have Tesla on the other side, headed by Elon Musk, that represents a more, however which adjective you want to use, aggressive, innovative, I don't know. But, uh, wh- what, what do you think about the difference between these two strategies? In your view, what's more likely? What's needed and is more likely to succeed in the short-term and- a- a- and the long-term?
- SKSertac Karaman
Definitely some sort of a balance is- is kind of the right way to go, but I- I do think that the- the thing that is the most important is actually, like, an informed public, so I- I don't- I don't mind, you know, I- I personally, like, if I were in some place, I wouldn't mind so much, like, taking a certain amount of risk. Um, some other people might. And so I think the key is for people to be informed and so that they can, ideally, they can make a choice. In some cases, that kind of choice, um, making that unanimously is- is of course very hard. Um, but I don't think it's actually that hard to inform people. Um, so I think in- in- in one case, like, for example, even the Tesla approach, um, I don't know, it's hard to judge how informed it is, but it is somewhat informed. I mean, you know, things kind of come out. I think people know what they're taking and- and things like that and so on. But I think the- the underlying, um, I do think that these two companies are a little bit kind of representing, like, a... Of course, they, you know, one of them seems a bit safer, the other one or, you know, um, whatever the adjective for that is, and the other one seems more aggressive, or whatever the adjective for that is-
- LFLex Fridman
Yeah.
- SKSertac Karaman
... but- but I think, you know, when you turn the tables, there are actually, there are two other orthogonal dimensions that these two are focusing on. On the one hand, for Waymo, I can see that, you know, they're, I mean, um, they, I- I think they a little bit see it as research as well, so they kind of, they don't... I'm not sure if they're, like, really interested in, like, an immediate, um, product. Um, you know, they- they talk about it. Um, sometimes there's some pressure to talk about it, so they- they kind of go for it, but I think, um, I think that they're thinking, um, maybe in the back of their minds, maybe they don't put it this way, but I think they- they realize that we're building, like, a new engine. It's kind of like, call it the AI engine or whatever that is, and- and, you know, an autonomous vehicles is- is a very interesting embodiment of that engine that allows you to understand where the ego vehicle is, the ego thing is, where everything else is, what everything else is gonna do, and- and how do you react, how do you actually, you know, interact with humans the right way, how do you build these systems? And I think, uh, they- they wanna know that. They wanna understand that. And so they keep going and doing that. And so on the other dimension, Tesla is doing something interesting. I mean, I think that they have a good product. People use it. The thing that, you know, like, it's not for me, um, but I- I can totally see people- people like it and- and people, I- I think they have a good product outside of automation, but I was just referring to the- the- the automation itself. I mean, you know, like, it- it kinda drives itself. You still have to be kind of, um, you still have to pay attention to it, right?
- LFLex Fridman
Mm-hmm.
- SKSertac Karaman
But, you know, um, people seem to use it, so it works for something. Um, and so people, I think people are willing to pay for it. People are willing to buy it. I think it, uh, it's- it's one of the other reasons why people buy a Tesla car. Maybe one of those reasons is Elon Musk is the CEO, and, you know, he seems like a visionary person. That's what people think, you know? He seems like a visionary person, and so it, that adds, like, 5K to the value of the car.
- LFLex Fridman
(laughs)
- SKSertac Karaman
And then maybe another 5K is the autopilot.
- LFLex Fridman
Yeah.
- SKSertac Karaman
And- and, you know, it's- it's useful. I mean, it's, um, useful in the sense that, like, people are using it, and so I- I- I can see Tesla. Sure, of course they wanna be visionary. They wanna kinda put out a certain approach, and they may actually get there, um, but I think that there's also a- a- a primary benefit of doing all these updates and rolling it out 'cause, you know, people pay for it. And it's- it's, you know, it's basic, you know, demand, supply, market, and people like it. They're happy to pay another 5K, 10K for that novelty or whatever that is. Um, they, and they use it. It's not like they get it and they try it a couple times as a novelty, but they use it a lot of the time. And so I think that's what Tesla is doing. It's actually pretty different. Like, they- they are on pretty orthogonal dimensions of what kinda things that they're building. They are using the same AI engine, so it's very possible that, you know, they're both gonna be, um, sort of one day, um, kind of using a similar, almost like an internal- internal combustion engine. That's a very bad metaphor, but-
- LFLex Fridman
(laughs)
- SKSertac Karaman
... similar internal combustion engine, and maybe one of them is building, like, a car. The other one is building a truck or something. So ultimately, the use case is very different.
- LFLex Fridman
So you,
- 29:46 – 47:08
Optimus Ride
- LFLex Fridman
like I said, are one of the founders of Optimus Ride. Let's take a step back. It's one of the success stories in autonomous vehicle space. It's a great autonomous vehicle company. Let's go from the very beginning. What does it take to start an autonomous vehicle company? How do you go from idea to deploying vehicles like you are in a few, a bunch of places, including New York?
- SKSertac Karaman
I would say that I think that, you know, what happened to us is, was- was the following. I think, um, we- we realized a lot of kind of talk in the autonomous vehicle industry back in, like, 2014 even, when we wanted to kind of get started, um, and- and I- I don't know. Like, I- I kind of, I would hear things like fully autonomous vehicles two years from now, three years from now. I kinda never bought it.... um, you know, I was a part of, um, MIT's Urban Challenge Entry. Um, it kind of, like, it has an interesting history. So, um, I did in, in, in college and, and high school, sort of a lot of mathematically oriented work, and I think I kind of, you know, at some point, uh, it, it kind of hit me I wanted to build something, and so I came to MIT's mechanical engineering program. And I now realize, I think my advisor hired me because I could do, like, really good math.
- LFLex Fridman
Mm-hmm.
- SKSertac Karaman
But I told him that, "No, no, no. I, I wanna work on that Urban Challenge car, you know. I, I want to build the autonomous car." And I think that was, that was kind of like a process where we really learn, I mean, what the challenges are and, and what kind of limitations are we up against. You know, like, having the limitations of computers or understanding human behavior. There's so many of these things, and I think it just kind of didn't. And so, so we said, "Hey, you know, like, why don't we take a more, like, a market-based approach?" So we focus on a certain kind of market and we build a system for that. What we're building is not so much of, like, an autonomous vehicle only, I would say. So we build full autonomy into the vehicles, um, but, you know, the way we kind of see it is that we think that, uh, the approach should actually involve humans operating them, not just, just not sitting in the vehicle. And I think today what we have is today we have one person operate one vehicle, no matter what that vehicle. It could be a forklift. It could be a truck. It could be a car, whatever that is. And we wanna go from that to 10 people operate 50 vehicles. How do we do that?
- LFLex Fridman
You're referring to a world of maybe perhaps teleoperation. So can, can you just say what it means (laughs) for 10... It might be confusing for people listening. What does it mean for 10 people to control 50 vehicles?
- SKSertac Karaman
That's a good point.
- LFLex Fridman
(laughs)
- SKSertac Karaman
So I think it's, um, I very deliberately didn't call it teleoperation 'cause people, what people think then is that people think, um, away from the vehicle sits a person, sees, like, maybe puts on goggles or something, VR, and drives the car, so-
- LFLex Fridman
Right.
- SKSertac Karaman
... that's not at all what we mean. What we mean, the kind of intelligence whereby humans are in control, except in certain places the vehicles can execute on their own. And so imagine, like, like a room where people can see what, what the other vehicles are doing and everything, and, you know, there will be some people who are more like, more like air traffic controllers. Call them, like-
- LFLex Fridman
Yeah.
- SKSertac Karaman
... AV controllers.
- LFLex Fridman
(laughs) Yeah.
- SKSertac Karaman
Uh, and so these AV controllers would actually see kind of like, like a whole map, and they would understand where vehicles are really confident and where they kind of, you know, need a little bit more help. And the help shouldn't be for safety. Help should be for efficiency. Vehicles should be safe no matter what. If you had zero people, they could be very safe, but they'd be going five miles an hour, and so if you want them to go around 25 miles an hour, then you need people to come in and... And for example, you know, the vehicle come to an intersection and the vehicle can say, "You know, I can wait. I can inch forward a little bit, show my intent, or I can turn left. Um, and right now it's clear I can't turn. I know that. But before you give me the go, I won't." And so that's one example. This doesn't mean necessarily we're doing that actually. I think, I think if you go down all the m- all, all, uh, that much detail that every intersection you're kind of expecting a person to press a button, then I don't think you'll get the efficiency benefits you want. You need to be able to kind of go around and be able to do these things. But, but I think you need people to be able to set high level behavior to vehicles. That's the other thing with autonomous vehicles, you know. I think a lot of people kinda think about it as follows. I mean, this happens with technology a lot. You know, you think, "All right. So I know about cars, and I heard robots. So I think how this is gonna work out is that I'm gonna buy a car, press a button, and it's gonna drive itself. And when is that gonna happen?" You know? And people kind of tend to think about it that way. But when you think about what really happens is that something comes in in a way that you didn't even expect. Um, if asked, you might have said, "I don't think I need that," or, "I don't think it should be there," and so on. And then, an- and then that, that becomes "the next big thing," quote unquote. And so I think that this kind of, um, different ways of humans operating vehicles could be really powerful. I think that sooner than, than later we might open our eyes up to a world in which you go around walking a mall and there's a bunch of security robots that are exactly operated in this way. You go into a factory or a warehouse, there's a whole bunch of robots that are operated exactly in this way. You go to a... You go to the Brooklyn Navy Yard.
- LFLex Fridman
Mm-hmm.
- SKSertac Karaman
You see a whole bunch of autonomous vehicles, Optimus Ride-
- LFLex Fridman
(laughs)
- SKSertac Karaman
... (laughs) and they're operated maybe in this way.
- LFLex Fridman
Yes.
- SKSertac Karaman
But I think people kind of don't see that. I, I, I, I sincerely think that it's, it's... There's a possibility that we may almost see, like, like a whole mushrooming of this technology in all kinds of places that we didn't expect before, and that may be the real surprise. Um, and then one day when your car actually drives itself, it may not be all that much of a surprise at all because you see it all the time. You interact with them. You take the Optimus Ride. Hopefully that's your choice-
- LFLex Fridman
(laughs)
- SKSertac Karaman
... (laughs) um, and then, you know, you, you, you hear a bunch of things. You go around. You interact with them. I don't know. Like, you have a little delivery vehicle that goes around the sidewalks and delivers you things, and then you take it. It says, "Thank you." And then you get used to that, and one day your car actually drives itself. And the regulation goes by and, and, you know, you can hit the button to sleep, and it wouldn't be a surprise at all. I think that may be the real reality.
- LFLex Fridman
So there's gonna be a bunch of applications that pop up around autonomous vehicles, some, some of which, maybe many of which we don't expect at all. So if we look at Optimus Ride, what do you think... You know, the, the viral application, th- the one that, like, really works for people i- in mobility. Wh- what do you think Optimus Ride will connect with in the, in the n- near future first?
- SKSertac Karaman
Um, I think that the first places that, that I like to target honestly is, like, these places where, um, transportation is required within an environment, like people typically call it geo-fenced. So you can imagine, like, a roughly two mile by two mile, could be bigger, could be smaller-... type of an environment. And there's a lot of these kinds of environments that are typically transportation deprived. Uh, the Brooklyn Navy Yard that, you know, we're in today, we were in a few different places, but, uh, that's, that was the one that was le- last publicized. Now, that's a good example. So there's not a lot of transportation there, um, and you wouldn't expect like, I don't know, um... I think maybe operating an Uber there ends up being sort of a little too expensive, or when you compare it with operating Uber elsewhere, um, that becomes the... elsewhere becomes the priority, and these peop- those places become totally transportation deprived. And then what happens is that, you know, people drive into these places, and to go from point A to pain- point B inside this place within that day, they use their cars.
- LFLex Fridman
Mm-hmm.
- SKSertac Karaman
And so we end up building more parking for them to, for example, take their cars and go to a lunch place. Um, and I think that one of the things that can be done is that, you know, you can put in, um, efficient, safe, sustainable transportation systems into these types of places first. And I think that, you know, you could deliver mobility in an affordable way, um, affordable, accessible, uh, you know, sustainable, um, way. But I think what also enables is that this kind of effort, money, area, land that we spend on parking, we could reclaim some of that, and that is on the order of, like, even for a small environment, like, two-mile by two-mile, it doesn't have to be smack in the middle of New York, I mean, anywhere else, you're talking tens of millions of dollars. If you're smack in the middle of New York, you're looking at billions of dollars of savings just by doing that, and that's the economic part of it, and there's a societal part, right? I mean, just look around. I mean, um, the places that we live are, like, built for cars. It w- it didn't look like this just, like, 100 years ago. Like, today, no one walks in the middle of the street. It's for cars. We... no one tells you that growing up, but you grow into that reality. And so sometimes they close the road. It happens here, you know, like the celebration, they closed the road, still people don't walk in the middle of the road-
- LFLex Fridman
Yeah. (laughs)
- SKSertac Karaman
... like, just walk in the m- and people don't. But I think it, it has so much impact, the, the car in, in the space that we have, and, and I think we talked about sustainability, livability. I mean, ultimately, these kinds of places, that parking spots, at the very least, could change into something more useful or maybe just, like, park areas, recreational. And so I think that's the first thing that, that we're targeting, and, and I think that we're getting, like, a really good response both from an economic, societal point of view, especially places that are a little bit forward-looking. And, like, for example, Brooklyn Navy Yard, they have tenants, um, there's this thing there called, like, New Lab, uh, it's kind of like an innovation center. There's a bunch of startups there, and so, you know, you get those kinds of people and, and, you know, they're, they're really interested in, um, sort of making that environment more livable, and these kinds of solutions that Optimus Ride provides almost kind of comes in and, and becomes that. And many of these places that are transportation deprived, you know, they have, um... they actually rent shuttles, and so, you know, um, you can ask anybody, the shuttle experience is, like, terrible. People hate shuttles. Um, and I can tell you why. It's because, you know, like, the m- the m- the driver is very expensive in a shuttle business, so what makes sense is to attach 20, 30 seats to a driver, and a lot of people have this misconception, they think that shuttles should be big. Sometimes we get that at Optimus Ride. We tell them, "We're gonna give you, like, four-seaters, six-seaters," and we get asked like, "How about, like, 20-seaters?" (laughs) I'm like, "You know, you don't need 20-seaters." You want to split up those seats so that they can travel faster and the transportation delays would go down. That's what you want. If you make it big, not only you will get delays in transportation, but you won't have an agile vehicle. It will take a long time to speed up, slow down, and so on. It'll... you need to climb up to the thing, so it's kind of, like, really hard to interact with.
- LFLex Fridman
And scheduling too, perhaps, when you have more smaller vehicles, it becomes closer to Uber where you can actually get a personal... I mean, just the, the logistics of getting the vehicle to you is... becomes easier. When you have a giant shuttle, there's fewer of them, and it probably goes on a route, a specific route that-
- SKSertac Karaman
Yeah, it-
- 47:08 – 53:22
Waymo, Tesla, Optimus Ride timelines
- LFLex Fridman
I think it's really exciting what Optimus Ride is doing in terms of it feels the most reachable, like it'll actually be here and have an impact.
- SKSertac Karaman
Yeah. That is the idea.
- LFLex Fridman
And if we contrast that, again, we'll go back to our old friends, Waymo and Tesla. So Waymo seems to have sort of technically similar approaches as Optimus Ride, but a different... they're not as interested at, as having impact today. Uh, is, and they have a longer term sort of investment. It's almost more of a research project still, meaning they're trying to solve, as far as I understand it, maybe you can, you can, uh, differentiate, but they seem to want to do more unrestricted movement, meaning move from A to B where A to B is all over the place. Versus Optimus Ride is really nicely geofenced and really sort of, uh, established mobility in a particular environment before you expand it. And then Tesla is, like, the complete opposite, which is, you know, the entirety of the world actually is going to be automated. Highway driving, urban driving, every kind of driving, you know, you kind of creep up to it by incrementally improving the capabilities of the autopilot system. So when you contrast all of these, and on top of that, let me throw a question that nobody likes, but is, uh, timeline. When do you think each of these approaches, loosely speaking, nobody can predict the future, will see mass deployment? So Elon Musk predicts the, the craziest approach is at the en- uh, I've heard figures like at the end of this year. Right? So (laughs) that's probably...... wildly inaccurate, but how wildly inaccurate is it?
- SKSertac Karaman
I mean, first thing to lay out, like everybody else, it's, it's really, it's really hard to guess. I mean, I don't know, I don't know where, where Tesla can look at, or Elon Musk can look at and say, "Hey, you know, it's the end of this year." I mean, I don't know what you can look at. You know, um, e- even the d- the data that, you know, you w- I mean, if you look at the data, um, e- even kind of trying to extrapolate the end state without knowing what exactly is gonna go, especially for like a machine learning approach, I mean, it's, it's just kind of very hard to predict. Um, but I do think the following does happen. I think a lot of people, you know, what they do is that, um, there's something that I, I called a couple of times time dilation in technology prediction happens. Uh, let me try to describe a little bit. There's a lot of things that are so far ahead people think they're close, and there's a lot of things that are actually close people think it's far ahead. People try s- to kind of look at a whole landscape of technology development. I- admittedly, it's chaos. Anything can happen in any order at any time, and there's a whole bunch of things in there. People take it, clamp it, and put it into the next three years. Um, and so then what happens is that there are some things that maybe can happen by the end of the year or next year and so on, and they push that into, like, few years ahead because it's just hard to explain, and then there are things that are like, we're looking at 20 years more maybe, you know? Um, hopefully in my lifetime type of things. And, and 'cause, you know, we, we, we don't know. I mean, we don't know how, how hard it is even. Like, that's a problem. We don't know, like, if some of these problems are actually AI complete. Like, we have no idea what's going on. And, and, you know, we, we take all of that, and then we clump it, and then we say, "Three years from now." Um, and then some of us are more optimistic, so they're-
- LFLex Fridman
Yeah.
- SKSertac Karaman
... shooting at the, at the end of the year, and some of us are more realistic. They say, like, five years. But, you know, we all, I think... it's just hard to know. And, and I think, um, trying to predict, like, products ahead two, three years, it's, it's hard to know in the following sense, you know? Like, we typically say, "Oh, okay. This is a technology company," but sometimes, sometimes really you're trying to build something where the technology doesn't... like, there's a technology gap, you know? Like, a- and Tesla had that with electric vehicles, you know? Like, when they first started, uh, they would look at a chart, much like a Moore's law type of chart, and they would just kind of extrapolate that out, and they'd say, "We wanna be here. What's the technology to get there? We don't know. It goes like this, so it's probably just gonna, you know-"
- LFLex Fridman
(laughs)
- SKSertac Karaman
"... keep going."
- LFLex Fridman
Yeah.
- SKSertac Karaman
Um, with, with AI that goes into the cars, we don't even have that. Like, we d- we d- we can't... I mean, what can you quantify?
- LFLex Fridman
Yeah.
- SKSertac Karaman
Like, what kind of chart are you looking at, you know? Um, but, so but, so I think when there's that technology gap, it's just kinda really hard to predict. So now I realize I talked like five minutes and avoid your question. I didn't tell you anything about that (laughs) .
- LFLex Fridman
It's- (laughs)
- SKSertac Karaman
It was very skillfully done.
- LFLex Fridman
That was very well done.
- SKSertac Karaman
Um-
- LFLex Fridman
And I don't think you... I think you've actually argued that it's not of use even... Any answer you provide now is not that useful
- SKSertac Karaman
It's gonna be very hard. There's one thing that I really believe in and, um, and, you know, this is not my idea and it's been, you know, discussed several times, but, but this, um, this, this kind of like... something like a startup, um, or, or, or a kind of an innovative company, um, including definitely maybe Way- Waymo, Tesla, and maybe even some of the other big companies that are kind of trying things. This kind of like iterated learning is, is very important. The fact that we're over there and we're trying things and so on, I, I think that's, um, that that's important. We try to understand. And, and I think that, you know, the quote-unquote "Silicon Valley" has done that with business models pretty well. And now I think we're trying to get to do it where there's a literal technology gap. I mean, before like, you know, you're trying to build... I'm not trying to, you know... I, I think these companies are building great technology to, for example, enable internet search-
- LFLex Fridman
Mm-hmm.
- SKSertac Karaman
... to do it so quickly, and that kinda didn't, didn't... wasn't there so much. But at least like it was a kind of a technology that you could predict to some degree and so on, and now we're just kinda trying to build, you know, things that it's kinda hard to quantify what kind of a metric are we looking at.
- LFLex Fridman
So
- 53:22 – 53:50
Achieving the impossible
- LFLex Fridman
psychologically as a sort of a... as a leader of graduate students and e- at Optimus Ride, a bunch of brilliant engineers, just a curiosity, psychologically do you think it's good to think that, uh, you know, whatever technology gap we're talking about can be closed by the end of the year? Or do you... you know, 'cause we don't know. So the way... (laughs) do you want to say
- 53:50 – 58:39
Iterative learning
- LFLex Fridman
w- that everything is going to improve exponentially to yourself and to others around you as a, as a leader or do you want to be more sort of, uh, maybe not cynical but I don't wanna use realistic 'cause it's hard to predict, but, uh, yeah, maybe more cynical, pessimistic about the b- ability to close that gap?
- SKSertac Karaman
Yeah. I, I think that, you know, going back, I think that iterated learning is like key, that, you know, you're out there, you're running experiments to learn. And that doesn't mean sort of like, you know, you, you... like, like your Optimus Ride where you're kinda doing something but, uh, like in, in a, in an environment, but like what Tesla is doing I think is also kind of like this, this kind of notion. And, and, you know, people can go around and say like, you know, this year or next year or the other year and so on, but, but I think the, the, the nice thing about it-
- LFLex Fridman
Got it.
- SKSertac Karaman
... is that they're out there, they're pushing this technology in. Um, I think what they should do more of I think that kind of inform the people about what kind of technology that they're providing, you know, the good and the bad and then, you know. Uh, not just sort of, you know, if it works very well but I think... And I'm not saying they're not doing bad on informing. I, I think they're, they're kind of trying. They, you know, they put up certain things, or at the very least, YouTube videos comes out on, on how the summon function works every now and then and, and, you know, people get informed.
- LFLex Fridman
Yes.
- SKSertac Karaman
And so that, that kinda cycle continues, but, um, uh, you know, I, I, I admire it. I think they're kind of go out there and they, they do great things, they do their own kind of experiment. I think we do our own. And I think we're closing some similar technology gaps, but some also, some are orthogonal as well. You know, I think like, like we talked about, you know, people being remote, like it's something or in the kind of environments that we're in or, or you think about a Tesla car, maybe, maybe you can enable it one day like there's, you know, low traffic, like you kind of just stop and go motion, you just hit the button and the, you can really s- ... Or maybe there's another, you know, lane that you can pass into. You go in that. I think they can enable these kinds of pro-... I believe it. And so I, I think that that part, that is really important and that is really key. And, and beyond that, I think, um, you know, when is it exactly gonna happen and, and, and so on. I mean, um, it's like I said, it's very hard to predict, um, and I would, I would imagine that it would be good to do some sort of like a, like a one or two-year plan when it's a little bit more predictable, that, you know, you, the technology gaps you close and, and the, and the kind of, um, sort of product that would ensue. So I, I know that from Optimus Ride or, you know, other companies that I get involved in, I mean, at some point you find yourself in a situation where you're trying to build a product and, and people are investing in that, in that, you know, building effort, and those investors that they do wanna know as they compare the investments they wanna make, they do wanna know what happens in the next one or two years. And I, I think that's good to communicate that. But I think beyond that, it becomes, it becomes a vision that we want to get to someday and saying five years, 10 years, I don't think it means anything.
- LFLex Fridman
But iterative learning is key though. Do and learn.
- SKSertac Karaman
I think that is key.
- LFLex Fridman
You know, I gotta sort of throw back right at you criticism, uh, in terms of, uh, uh, you know, like Tesla or somebody communicating, you know, how something works and so on. I got a chance to visit Optimus Ride and you guys are doing some awesome stuff, and yet the internet doesn't know about it. So you should also communicate more showing off, you know, showing off some of the awesome stuff. The stuff that works and stuff that doesn't work. I mean, it's just, uh, the stuff I saw with the tracking of different objects and pedestrians so I mean, incredible stuff going on there, which is c-... Maybe it's just the nerd in me, but, uh, I think the world would love to see that kind of stuff.
- SKSertac Karaman
Yeah. That's, that's well taken. I think, um, you know, I, I should say that it's not like, you know, we, we, we weren't able to. I think we made a decision at some point. Um, that decision did involve me quite a bit on kind of, um, uh, sort of doing this in kinda quote-unquote "stealth mode" for a bit. Um, but I think that, you know, we'll, we'll open it up quite a lot more. And, and I think that we are also at Optimus Ride kind of hitting, um, a new, new era. Um, you know, we're, we're, we're big now. We're doing a lot of interesting things. And, and I think, you know, some of the deployments that we kinda announced were some of the first bits, bits of, um, information that we kinda put out into the world. We'll also put out our technology. A lot of the things that we've been developing is really amazing and then, you know, we're, we're gonna, we're gonna start putting that out. Uh, we're especially interested in sort of like, um, being able to work with the best people. And I think, and I think it's, it's good to not just kinda show them when they come to our office for an interview, but-
- LFLex Fridman
Yeah.
- SKSertac Karaman
... just put it out there in terms of like, you know, get people excited about what we're doing.
- 58:39 – 1:03:21
Is Lidar is a crutch?
- SKSertac Karaman
- LFLex Fridman
So on the autonomous vehicle space, let me ask one last question. Uh, so Elon Musk famously said that lidar is a crutch. So, uh, (laughs) I've talked to a bunch of people about it, gotta ask you. You used that crutch quite a bit in the DARPA days. So, uh, uh, you know, and his i- his idea in general sort of, you know, more provocative and fun I think than a, a technical discussion, but the idea is that camera-based c- primarily camera-based systems is going to be what defines the future of autonomous vehicles. So what do you think of this idea? Lidar's a crutch versus primarily, uh, camera-based systems?
- SKSertac Karaman
First thing's first. I think, you know, I'm, I'm a big believer in just camera-based autonomous vehicle systems. Like I, I think that, you know, you can put in a lot of autonomy and, and you can do great things. And, and it's, it's, it's very possible that at the timescales like we said we can't predict, 20 years from now, like you may be a- able to do, do things that we're doing today only with lidar and then you may be able to do them just with cameras. And, and I think that, um, you know, you, you can just, um... I, I, I think that I will put my name on it too, like there will be a time when you can only use cameras and you'll be fine. Um, at that time though, it's very possible that, you know, you find the lidar system as another robustifier or, or it's so affordable that it's stupid not to, you know? Just kind of put it there. And I think, um, and I think we may be looking at a future like that.
- LFLex Fridman
Do you think we're over-relying on lidar right now because we understand it better, it's more reliable in many ways in term-... from a safety perspective?
- SKSertac Karaman
It's easier to build with. That's the other, that's the other thing. I think f-... to be very frank with you, I mean, um, you know, we've seen a lot of sort of autonomous vehicles companies come and go, and the approach has been, you know, you slap a lidar on a car and it's kind of easy to build with when you have a lidar, you know, you just kind of code it up and, and you hit the button and you do a demo.
- LFLex Fridman
Mm-hmm.
- SKSertac Karaman
Um, so I think there's admittedly there's a lot of people they focus on the lidar 'cause it's easier to build with. Um, that doesn't mean that, you know, without the camera, with just cameras, you can, uh, you cannot do what they're doing but it's just gonna be a lot harder. And so you need to have certain kind of expertise to, to exploit that.... what we believe in, in, you know, use, maybe seen some of it, is that, um, we, we believe in computer vision. We certainly work on computer vision at Optimus, right? Um, by a lot. Like, um... And, and we've been doing that from day one. And we also believe in sensor fusion. So, you know, we, we do... We have a relatively minimal use of LIDARs but, but we do use them. And I think, you know, in the future, I really believe that the following sequence of events may happen. First things first, number one, there may be a future in which, you know, there is, like, cars with LIDARs and everything and the cameras, but, you know, there's... In this 50-year-ahead future, they can just drive with cameras as well, especially in some isolated environments and cameras, they go and they do the thing. In the same future, it's very possible that, you know, the LIDARs are so cheap and frankly make the software maybe, um, a little less compute-intensive, uh, at the very least, or maybe less complicated so that they can be certified or, or ensure their, of their safety and things like that, that it's kind of stupid not to put the LIDAR. Like, imagine this, you either put, pay money for the LIDAR or you pay money for the compute. And if you don't put the LIDAR, it's a more expensive system because we have to put in a lot of compute. Like, this is another possibility. Um, I do think that a lot of the, um, sort of initial deployments of self-driving vehicles, I think they will involve LIDARs. And especially either low-range or short, um, either short-range or low-resolution LIDARs are actually not that hard to build in solid state. Uh, they're still scanning, but like MEMS type of scanning LIDARs and things like that, they're like, they're actually not that hard. I think they will... May- maybe kind of playing with the spectrum and the phase arrays, they're, they're a little bit harder, but, but I think, um, like, you know, putting a MEMS mirror in there that kind of scans the environment, it's not hard. The only thing is that, you know, you... Just like with a lot of the things that we do nowadays in developing technology, you hit fundamental limits of the universe. Um, the speed of light becomes a problem-
- LFLex Fridman
Yeah.
- SKSertac Karaman
... in when you're trying to scan the environment so you don't get either good resolution or you don't get range. Um, but, but, you know, it's still, it's something that you can put in there affordably.
- 1:03:21 – 1:18:06
Fast autonomous flight
- LFLex Fridman
So let me jump back to, uh, drones. You've, uh, you have a role in the Lockheed Martin AlphaPilot Innovation Challenge, where, uh, teams compete in drone racing. It's super cool, super intense, interesting application of AI. So can you tell me about the very basics of the challenge and where you fit in, what your thoughts are on this problem? And it's sort of echoes of the early DARPA challenge in the Through the Desert that we're seeing now, now with drone racing.
- SKSertac Karaman
Yeah, I mean, one interesting thing about it is that, you know, people... Drone racing exists as an e-sport. And so it's much like you're playing a game, but there's a real drone going in an environment.
- LFLex Fridman
A human being is controlling it with goggles on, so there's no... It is a robot, but there's no, uh, AI.
- SKSertac Karaman
There's no AI, yeah. Human being is controlling it. And so that's already there. And, um, and I've been interested in this problem for quite a while actually, um, from a roboticist point of view, and that's what's happening in AlphaPilot.
- LFLex Fridman
Which, which problem? Of aggressive flight?
- SKSertac Karaman
Of aggressive flight, fully autonomous aggressive flight. Um, the problem that I'm interested... I mean, you asked about AlphaPilot and I'll, I'll get there in a second, but the problem that I'm interested in, I'd love to build autonomous vehicles like, like drones that can go far faster than any human possibly can. I think we should recognize that we, as humans, have, you know, limitations in, in how fast we can process information. And those are some biological limitations. Like, we think about this AI this way too, I mean, this has been discussed a lot, and this is not sort of my idea per se, but a lot of people kinda think about human-level AI, and they think that, you know, "AI is not human level. One day it'll be human level and humans and AIs, they kind of interact." Um, versus I think that the situation really is that humans are at a certain place and AI keeps improving and at some point just crosses off, and, and, you know, it gets smarter and smarter and smarter. And so drone racing, the same issue. Humans play this game and, you know, you have to, like, react in milliseconds. And there's really... You know, you see something with your eyes and then that information just flows through your brain into your hands so that you can command it. And there's some also delays in, you know, getting information back and forth, but suppose those delays didn't exist, you just... Just the delay between your eye and your fingers is a delay that a robot doesn't have to have. Um, so we end up building, in my research group, like, systems that, you know, see things at a kilohertz, like a human eye would barely hit 100 hertz. So imagine things that see stuff in slow motion, like 10x slow motion. Um, it will be very useful. Like, we talked a lot about autonomous cars so, um, you know, we, we don't get to see it but 100 lives are lost every day just in the United States on traffic accidents, and many of them are, like, known cases, you know. Like the... Uh, you're coming through, like, uh, like, a ramp going into a highway, you hit somebody and you're off or, you know, like, you kind of get confused, you try to, like, swerve into the next lane, you go off the road and, and you crash, whatever. And, um, I think if you had enough compute in a car and a very fast camera right at the time of an accident, you could use all compute you have. Like, you could shut down the infotainment system and use that kind of computing resources, instead of rendering, you use it for the kind of artificial intelligence that goes in there, the autonomy. And you can, you can either take control of the car and bring it to a full stop, but even e- even if you can't do that, you can deliver what the human is trying to do. Human is trying to change the lane, but goes off the road, not being able to do that with motor skills and, and the eyes and, and you know, you can get in there.And I was, there's so many other things that you can enable with what I would call high throughput computing. You know, data is coming in extremely fast and in real time you have to process it. And the current CPUs, however fast you clock it, are typically not enough. You need to build those computers from the ground up so that they can ingest all that data. That I'm really interested in.
- LFLex Fridman
Just on that point, just really quick, is the, currently what's the bottleneck? You mentioned the delays in humans. Is it, is it the hardware? So you work a lot with NVIDIA hardware. Is it the hardware or is it the software?
- SKSertac Karaman
I think it's both. I think it's both. In fact, they need to be co-developed, I think, in the future. I mean, that's a little bit what NVIDIA does. Sort of like they almost, like, build the hardware and then they build the neural networks and then they build the hardware back and the neural networks back, and it goes back and forth. But it's that co-design. And I think that, you know, like, um, we tried to, way back, we tried to build a fast drone that could use a camera image to, like, track what's moving in order to find where it is in the world. This typical sort of, you know, visual inertial state estimation problems that we would solve. And you know, we just kind of realized that we're at the limit sometimes of, you know, doing simple tasks. We're at the limit of the camera frame rate because, you know-
- LFLex Fridman
Uh-huh.
- SKSertac Karaman
... if you really wanna track things, you want the camera image to be 90% kind of like, or, or some- somewhat the same from one frame to the next.
- LFLex Fridman
That's right.
- SKSertac Karaman
And why are we at the limit of the camera frame rate? It's because camera captures data, it puts it into some serial connection. It could be USB or, like, there's something called camera serial interface that we use a lot. It puts into some serial connection, and copper wires can only transmit so much data. And you hit the channel limit on copper wires and, you know, you, you hit yet another kind of, um, universal limit-
- LFLex Fridman
Mm-hmm.
- SKSertac Karaman
... that you can transfer the data. So you have to be much more intelligent on how you capture those pixels. You can take compute and put it right next to the pixels. People are building those-
- LFLex Fridman
How hard is it to do? How hard is, uh, it to, uh-
- SKSertac Karaman
Um-
- LFLex Fridman
... to, to, to get past the, the bottleneck of the copper wire?
- SKSertac Karaman
Um, yeah. You need to, you need to do a lot of parallel processing as you can imagine. The same thing happens in the GPUs, you know, like the data is transferred in parallel somehow, it gets into some parallel processing. I, I think that, you know, like, um, now we're really kind of diverted off into so many different dimensions, but-
- LFLex Fridman
Great. So it's aggressive flight. How do we make drones see many more frames a s- a second in order to enable aggressive flight? That's a super interesting problem.
- SKSertac Karaman
That's an interesting problem. So but, like, think about it. You have, you have CPUs, you clock them at, you know, several gigahertz. Um, we don't clock them faster largely because, you know, we run into some heating issues and things like that. But another thing is that three gigahertz clock light travels kind of like on the order of a few inches or an inch. That's the size of a chip. And so you pass a clock cycle, and as the clock signal is going around in the chip, you pass another one. And so trying to coordinate that, the design of the complexity of the chip becomes so hard. I mean, we have hit the fundamental limits of the universe-
- LFLex Fridman
Yeah.
- SKSertac Karaman
... in so many things that we're designing. I don't know if people realize that.
- LFLex Fridman
It's great. (laughs)
- SKSertac Karaman
It's great, but, like, we can't make transistors smaller because, like, quantum effects, the electrons start to tunnel around. We can't clock it faster. One of the reasons why is because, like, l- information doesn't travel faster in the universe.
- LFLex Fridman
Yeah.
- SKSertac Karaman
And we're limited by that. Same thing with the laser scanner. But so then it becomes clear that, um, you know, the way you organize the chip into a CPU or even a GPU, you now need to look at how to redesign that if you're gonna stick with silicon.
- LFLex Fridman
Yes.
- SKSertac Karaman
You could go do other things too. I mean, there's that too, but you really almost need to take those transistors, put them in a different way so that the information travels on those transistors i- i- in a different way, in a much more w- way that is specific to the high speed cameras coming in. And so that's one of the things that, that we talk about quite a bit.
- LFLex Fridman
So drone racing kinda really makes that-
- SKSertac Karaman
Embodies that.
Episode duration: 1:22:50
Install uListen for AI-powered chat & search across the full episode — Get Full Transcript
Transcript of episode M1-v-dXIzho
Get more out of YouTube videos.
High quality summaries for YouTube videos. Accurate transcripts to search & find moments. Powered by ChatGPT & Claude AI.
Add to Chrome