Boris Sofman: Waymo, Cozmo, Self-Driving Cars, and the Future of Robotics | Lex Fridman Podcast #241

Lex Fridman 0:00 The following is a conversation with Boris Sofman, who is the senior director of engineering and head of trucking at Waymo, the autonomous vehicle company, formerly the Google self-driving car project. Before that, Boris was the co-founder and CEO of Anki, a robotics company that created Cozmo, which, in my opinion, is one of the most incredible social robots ever built. It's a toy robot, but one with an emotional intelligence that creates a fun and engaging human-robot interaction. It was truly sad for me to see Anki shut down when it did. I had high hopes for those little robots. We talk about this story and the future of autonomous trucks, vehicles, and robotics in general. I spoke with Steve Viscelli recently on episode 237 about the human side of trucking. This episode looks more at the robotic side. This is the Lex Fridman Podcast. To support it, please check out our sponsors in the description, and now here's my conversation with Boris Sofman.

Lex Fridman 1:08 Who is your favorite robot in science fiction, books or movies?

Boris Sofman 1:12 WALL-E and R2-D2, where they were able to convey such an incredible degree of intent, emotion, and kinda character attachment without having any language whatsoever, um, and just purely through the emo- richness of em- emotional interaction, so those are fantastic. And then, uh, the Terminator series is just, like, really (laughs) in front of my eye-

Lex Fridman 1:33 (laughs) Okay.

Boris Sofman 1:34 ... pretty wide-

Lex Fridman 1:34 Jumped from WALL- (laughs)

Boris Sofman 1:35 ... wide range, right? Uh, but, uh, I kinda love this, uh, dynamic where you have this, like, incredible Terminator itself that, that Arnold played, but, uh, and then he was kind of like the inferior, like, previous generation version that was, like, totally outmatched, uh, you know, in terms of kinda specs by the new one, but, you know, still kind of, like, held his own. And so it was kind of interesting where you, you realize how many, how many levels there are on the spectrum from human to kind of potentials in AI and robotics to, uh, futures. And so, yeah, that movie really, uh, as much as it was, like, kind of a dark world in a way, was actually quite fascinating, gets the imagination going.

Lex Fridman 2:11 Well, from an engineering perspective, both the movies you mentioned, uh, WALL-E and, um, Terminator, uh, the first one is probably achievable, you know, humanoid robot, maybe not with, like, the realism in terms of skin and so on, but that humanoid form, we have that humanoid form. It seems like a compelling form. Maybe the challenge is it's super expensive to engine- to, to build, but you can imagine maybe not a machine of war-

Boris Sofman 2:36 Yeah.

Lex Fridman 2:36 ... but you can imagine Terminator-type robots walking around.

Boris Sofman 2:40 Yeah.

Lex Fridman 2:40 Uh, and then the same obviously w- with WALL-E. You've basically... So for people who don't know, you, uh, created the company Anki that created, uh, a small robot with a big personality called Cozmo that just... It does exactly what WALL-E does, which is somehow with very few basic visual tools is able to communicate a depth of emotion, and that's fascinating. Uh, but then again, the humanoid form is, uh, super compelling. So, like, C- uh, Cozmo is very distant from a humanoid form-

Boris Sofman 3:11 Yeah.

Lex Fridman 3:11 ... and then the Terminator has a humanoid form, and you can imagine both of those actually being in our society.

Boris Sofman 3:16 It's true, and it's interesting because, um, it- it was very intentional to go really far away from human form when you think about a character like Cozmo or like WALL-E, where, um, you can completely rethink, uh, the constraints you put on that character, um, what tools you leverage, and then how you actually create a personality, uh, and a level of intelligence interactivity that actually matches the constraints that you're under, whether it's, uh, mechanical or sensors or AI of the day. This is why I almost- um, was always very surprised by how much energy people put towards trying to replicate human form-

Lex Fridman 3:53 Mm-hmm.

Boris Sofman 3:53 ... in a robot because you actually take on some pretty significant, um, kinda constraints and, and downsides when you do that.

Lex Fridman 4:00 Mm-hmm.

Boris Sofman 4:00 Um, the first of which is obviously the cost, where it just- the, the articulation of a human body is just so, like, magical, um, in both the precision as well as the dimensionality, that to replicate that even in its clo- reasonably close form takes, like, a giant amount of joints and actuators and, uh, emotion and, and, you know, sensors and encoders and so forth. But then, um, you're almost, like, setting an expectation that the closer you try to get to human form, the more you expect the strengths to match, and that's not the way AI works is there's places where you're way stronger, and there's places where you're weaker. And by moving away from human form, you can actually, uh, change the rules and embrace your strengths and bypass your weaknesses.

Lex Fridman 4:40 And at the same time, the human form, like, has way too many degrees of freedom to play with. It's, it's kind of con- counterintuitive just as you're saying, but when you have fewer constraints, it's almost harder to master the, the communication of emotion. Like, you see this with cartoons, like stick figures. You can communicate quite a lot with just very minimal, like two dots for eyes and a line for, for a smile. I think it- like, you can almost communicate arbitrary levels of emotion with just two dots and a line.

Boris Sofman 5:12 Yeah.

Lex Fridman 5:13 And, like, that's enough, and if you focus on just that, you can communicate the full range, and then-

Boris Sofman 5:18 Yeah.

Lex Fridman 5:19 ... you, like, if you do that, then you can focus on the actual magic of, of, uh, human and dot-line interaction versus all the engineering mess.

Boris Sofman 5:30 That's right, like, dimensionality, voice, all these sort of things, they actually become a crutch-

Lex Fridman 5:34 Yeah.

Boris Sofman 5:34 ... where you get lost in a search space almost. Um, and so some of the best animators that we've worked with, um, they almost, like, study when they come up, uh, you know, kind of in buil- in building their expertise by forcing these, um, projects where all you have is, like, a ball that can, like, kind of jump and manipulate itself or, like, really, really, like-... aggressive constraints where you're forced to kind of ex- ex- extract the deepest level of emotion. And so, in a lot of ways, um, you know, when we tho- when we thought about Cozmo, it was like, "You're, you're right." Like, our... If we had to, like, describe it in, like, one small phrase, it was bringing a Pixar character to life in the real world. It's, uh, it's- it's what we were going for. And, um, in a lot of ways what was interesting is that with, like, WALL-E, which we studied incredibly deeply, and in fact some of our team were, you know, kind of, uh, had worked previously at, um, at Pixar and on that project. Um, they intentionally constrained WALL-E as well, even though in an animated film you could do whatever you wanted to because it forced you to, like, really saturate the smaller amount of dimensions. But, uh, you sometimes end up getting a far more beautiful output, um, because you're pushing at the extremes, um, o- of this emotional space in a way that you just wouldn't because you get lost in a surface area, uh, if you have, like, something that is just infinitely articulable.

Lex Fridman 6:49 So if we backtrack a little bit, and, uh, you thought of Cozmo in 2011, in 2013 actually, uh, designed and built it. What is Anki? What is Cozmo? I guess, who is Cozmo? (laughs)

Boris Sofman 7:01 Who is Cozmo, yeah.

Lex Fridman 7:02 And, uh, what was the vision behind this incredible little robot?

Boris Sofman 7:06 We started, uh, Anki back in, like, like, while we were still in graduate school. So myself and my two co-founders, we were PhD students, uh, in the Robotics Institute at Carnegie Mellon. Um, and so we were, uh, studying robotics, AI, machine learning, kind of diff- you know, different, uh, uh, areas. One of my co-founders was working on walking robots, uh, you know, uh, for a period of time. And so we all had a, um, a bit of a really deep, kind of a deeper passion for applications of robotics and AI, where, um, th- there's like a spectrum where there's people that get, like, really fascinated by the theory of AI and machine learning and robotics where, um, whether it gets applied in the near future or not is less of a kind of factor on them, but they love the pursuit of the, like, the challenge. And that's necessary, and there's a lot of incredible breakthroughs that happen there. We're probably closer to the other end of the spectrum where we love the technology and the, um, and all the evolution of it, but we were really driven by applications. Like, how can you really reinvent experiences and functionality and build value that wouldn't have been possible without, um, these approaches? And- and that's what drove us, and we had a, kind of some experiences through previous jobs and internships where we, like, got to see the applied side of robotics. And at that time, there was actually relatively few applications of robotics, um, that were outside of, um, you know, pure research or industrial applications, um, military applications and so forth. There were very few outside of it. So maybe, you know, iRobot was, like, one exception and then maybe there were a few others, but for the most part there weren't that many. And so we got excited about consumer applications of robotics where you could leverage w- way higher levels of intelligence, um, through software to create value and experiences that were just not possible, um, in- i- in those fields today. Um, and we saw kind of a- a- a pretty wide range of applications, um, that varied in the complexity of what it would take to actually solve those. And what we wanted to do was to commercialize this into a company, but actually do a bottoms-up approach where we could have a huge impact in a space that was ripe to have an impact at that time, and then build up off of that and move into other areas. And entertainment became the place to start-

Lex Fridman 9:09 Mm-hmm.

Boris Sofman 9:09 ... because, um, you had relatively little innovation in the toy space, uh, and entertainment space. You had these really rich, um, experiences in video games and, uh, and movies, but there was, like, this chasm in between. And so we thought that we could really reinvent that experience and there was a- a really fascinating transition technically that was happening at the time where the cost of components was plummeting because of the mobile phone industry and then the smartphone industry. And so the cost of a microcontroller, of a camera, of a motor, of memory, of microphones, cameras was dropping by orders of magnitude.

Lex Fridman 9:43 Mm-hmm.

Boris Sofman 9:43 And then on top of that with the iPhone coming out in 2000, uh, I think it was 2007 I believe, uh-

Lex Fridman 9:50 Mm-hmm. Yeah, this is right.

Boris Sofman 9:51 Um, you s- it- it started to become apparent within a couple of years that this could become a really incredible interface device and the brain with much more computation behind a physical world experience-

Lex Fridman 10:02 Mm-hmm.

Boris Sofman 10:03 ... that wouldn't have been possible previously. Um, and so, um, we really got excited about that and how we pushed all the complexity from the physical world into software by using really inexpensive components, but putting huge amounts of complexity into the AI side. And so Cozmo became our second product, and then the one that we're probably most proud of. The idea there was to create a physical character that had enough understanding and awareness of the physical world around it and the context that mattered to feel like, uh, like he was alive. Um, and, uh, to be able to have these, like, emotional kind of connections and experiences with people that you would typically only find, uh, inside of a movie. And the motivation very much was- was Pixar, like, we had an incredible, uh, respect and appreciation for what they were able to, um, build in this, like, really beautiful fashion in film. Um, but it was always like a, you know, one, it was virtual, and two, it was like a story on rails that had no interactivity to it. It was, uh, very fixed, and it obviously had a magic to it, but where you really start to hit like a different level of experience is when you're actually able to physically interact with our robot.

Lex Fridman 11:06 Yeah. And then that was your idea with Anki, like, the first product was the cars.

Boris Sofman 11:10 Yeah.

Lex Fridman 11:10 So basically you take, you take a toy, you add intelligence into it in the same way you would add intelligence into AI systems within a video game, but you're now bringing it into the physical space. So the idea is- is really brilliant, which is you're basically bringing video games to life.

Boris Sofman 11:29 Exactly. That's exactly right. We literally use that exact same phrase because in the case of DRIVE, this was a parallel of the racing genre.

Lex Fridman 11:36 Yeah.

Boris Sofman 11:37 And the goal was to effectively have a physical racing experience, but have a virtual state at all times that matches what's happening in the physical world. And then you can have a video game off of that, and you can have, uh, different characters, different traits for your, uh, the cars, um, weapons and interactions and special abilities and all these sort of things that you think of virtually, but then you can have it physically. And, um, one of the things that we were, like, really surprised by that really stood out and immediately led us to really, like, kind of accelerate the path towards, um, Cozmo is that things that feel like they're really constrained and simple in the physical world, they have an amplified impact on people, where the exact same experience virtually would not have anywhere near the impact, but seeing it physically really stood out. And so effectively we've, with- with DRIVE we were creating a video game engine for the physical world, um, and then with Cozmo we expanded that video game engine to create a character and, uh-... uh, kind of an animation and interaction engine on top of it that allowed us to start to create these much more rich experiences. And a lot of those elements were, uh, almost like a proving ground for what would human/robot interaction feel like in a domain that's much more forgiving, where you can make mistakes in a game.

Lex Fridman 12:51 Mm-hmm.

Boris Sofman 12:51 It's okay if like, uh, if, you know, c- car goes off the track or if a, if Cozmo makes a mistake. Um, and what's funny is actually we were so worried about that, um, in reality we realized very quickly that those mistakes can be endearing and if you make a mistake, as long as you realize you made a mistake and have the right emotional reaction to it, it builds even more empathy-

Lex Fridman 13:08 Yeah.

Boris Sofman 13:09 ... with the character.

Lex Fridman 13:09 Yeah, that's brilliant.

Boris Sofman 13:10 Um, which is-

Lex Fridman 13:10 Exactly. So when, uh, the, the thing you're optimizing for is fun, you have so much more freedom to fail, to explore. And, and also in the toys space, like all of this is really brilliant and I, I gotta ask you, backtrack. It seems for a roboticist to take a s- jump in, into the direction of fun is a brilliant move because one, you have the freedom to explore, to design, all those kinds of things, and you can also build cheap robots.

Boris Sofman 13:39 Yeah.

Lex Fridman 13:40 Like you don't have to... Like if, if you're not chasing perfection and like toys, it's understood that you can go cheaper.

Boris Sofman 13:47 Yeah.

Lex Fridman 13:47 Which means in robot it's still expensive but it's actually affordable by a large number of people. So it's a really brilliant space to explore.

Boris Sofman 13:55 Yeah, that's right. It's, uh... And in fact we realized pretty quickly that like perfection is actually not fun.

Lex Fridman 14:00 Yeah.

Boris Sofman 14:00 Because like in a traditional robotic, roboticist sense, the first kind of path planner... And, uh, this is the, you know, the part that I wor- worked on outta the gate, was like a lot of the kind of AI systems where you had these, you know, vehicles and then y- you know, cars racing, kind of making optimal maneuvers to try to kind of get ahead. And you realize very quickly that like that's actually not fun because you want the like chaos from mistakes-

Lex Fridman 14:22 Yeah.

Boris Sofman 14:22 ... and the... Uh, and so you start to kind of intentionally almost add noise to the system, uh, in order to kind of create more of a realism and in the exact same way the human player might start really ineffective and inefficient and then start to kind of increase their quality bar as they, um, uh, as they progress. And there is a really, really aggressive constraint that's forced on you by, uh, being a consumer product where the price point matters a ton, particularly in like kind of an entertainment where, um, you know, you, you can't make a $1,000 product unless you're gonna meet the qua- like, uh, the expectations of a $1,000 product. And so, um, in order to make this work, like your cost of goods had to be like, like, w- you know, well under $100. Uh, uh, in the case of Cozmo we got it under $50 end-to-end fully packaged and delivered.

Lex Fridman 15:04 And it was under $200, uh-

Boris Sofman 15:06 At retail, yeah.

Lex Fridman 15:06 ... the cost at retail.

Boris Sofman 15:08 Yeah.

Lex Fridman 15:08 So w- uh, uh, okay. If we sit down like at the early stages, if we go back to that, and you're sitting down and thinking about what Cozmo looks like-

Boris Sofman 15:17 Yeah.

Lex Fridman 15:17 ... from a design perspective and from a cost perspective, I imagine that was part of the conversation. Um, I... Well, first of all, what came first? Did you have a cost in mind? Is there a target you're trying to chase? Did you have a vision in mind like size? Did you have... 'Cause g- there's a lot of unique qualities to Cozmo, so for people who don't know, they should definitely check it out. There's, uh, there's a display, there's eyes on the little display.

Boris Sofman 15:41 Yeah.

Lex Fridman 15:41 And those eyes can... It's pretty, uh, low resolution eyes, right? But they, they still able to convey a lot of emotion. And there's this arm like that ou-

Boris Sofman 15:51 Lift sort of, yeah.

Lex Fridman 15:52 ... lifts stuff. But there's something about arm movement that adds even more kind of depth. It's like a... The face communicates emotion and sadness and disappointment and happiness, and then the arms kind of communicates, "I'm trying here." (laughs)

Boris Sofman 16:10 Yeah.

Lex Fridman 16:10 "I'm doing my best-"

Boris Sofman 16:12 Yeah.

Lex Fridman 16:12 "... in this complicated world."

Boris Sofman 16:14 No, no, uh, uh, exactly. So it's, um, uh, it's interesting because like, um, I think all of Cozmo's only four degrees of freedom and two of them are the two treads which is for-

Lex Fridman 16:22 Yes.

Boris Sofman 16:22 ... basic movement and so you literally have only a head that goes up and down-

Lex Fridman 16:27 Yeah.

Boris Sofman 16:27 ... a lift that goes up and down, and then your two wheels.

Lex Fridman 16:29 Yeah.

Boris Sofman 16:30 Uh, and you have sound, uh, and a screen.

Lex Fridman 16:32 Yeah.

Boris Sofman 16:32 And a low resolution screen. And with that, it's actually pretty incredible what you can c- uh, what you can come up with where, like you said, it's a, uh, it's a really interesting give and take because there's a lot of ideas far beyond that obviously as you can imagine where like you said how big is it, how much degrees of freedom, what does he look like, um, uh, what does he sound like, how does he communicate? It's, it's a formula that actually scales way beyond entertainment. This is the formula for human r- kind of robot interface more generally is you almost have this triangle between, um, the physical aspects of it, the mechanics, the industrial design, what's mass producible, the cost constraints and so forth. Uh, you have the AI side of how do you understand the world around you, interact intelligently with it, execute what you want to execute, so perceive the environment, make intelligent decisions and, uh, and m- move forward. And then you have the character side of it. Um, most, uh, companies that have done anything in human/robot interaction really, uh, miss the mark or under-invest in the character side of it.

Lex Fridman 17:29 Mm-hmm.

Boris Sofman 17:29 Um, they over-invest in the mechanical side of it, uh, you know, and then varied results on the AI side of it. And so the thinking is that you put more mechanical flexibility into it, you're gonna do better.

Lex Fridman 17:40 Mm-hmm.

Boris Sofman 17:40 Um, you don't necessarily. You actually create a much higher bar, uh, for a high ROI because now your price point goes up, your expectations go up, and if the AI can't meet it or the overall experience isn't there, you, you miss the mark.

Lex Fridman 17:52 Brilliant.

Boris Sofman 17:53 Um-

Lex Fridman 17:53 So who... Like how did you through those conversations get the cost down so much and make it, made it so simple? Like that... There's a big theme here because you come from the Mecca of robotics which is Carnegie Mellon University robotics. Like for all the people I've interacted with that come from there or just from, you know, the world experts at robotics, they don't...... they would never build something like Cozmo.

Boris Sofman 18:18 Yeah.

Lex Fridman 18:19 And so where did that come from?

Boris Sofman 18:21 So, this-

Lex Fridman 18:21 The simplicity.

Boris Sofman 18:22 It came from this, this combination of a team that we had. And it was, i- it was quite cool because, like, we... And by the way, you ask anybody that's, like, experienced in the, like, kind of, you know, the toy entertainment space, you'll never sell a product over $99. Um, that was fundamentally false and we believed it to be false. It was because the experience had to kind of, you know, meet the mark.

Lex Fridman 18:40 Mm-hmm.

Boris Sofman 18:40 And so we pushed past that amount, but there was a pressure where the higher you go, the more seasonal you become and the tougher i- it becomes. And so on the cost side, we very quickly partnered up with some previous contacts that we worked with where, just as an example, one... Our head of mechanical engineering, um, was one of the earliest heads of engineering at Logitech and has a billion units of consumer products-

Lex Fridman 19:02 (laughs)

Boris Sofman 19:02 ... in circulation that he's worked on.

Lex Fridman 19:03 Yeah.

Boris Sofman 19:03 So, like, crazy low-cost, high-volume consumer product experience. We had a really great mechanical engineering team and just a very practical mindset where we were not gonna compromise on feasibility in the market in order to chase something that would be an enabler. And we pushed a huge amount of expectations onto the software team where, yes, we're gonna use cheap, uh, noisy motors and sensors, but we're gonna fix it in the, um, on the software side. Then we found, on the design and character side, there was a faction that was more from, like, a game design background that thought that it should be very games-driven Cozmo, where you create a whole bunch of games experiences and it's all about, like, game mechanics. And then there was, um, a, a fashion which my, my co-founder and I, they're most involved in this, like, really believed in, which was character-driven. And the argument is that you will never compete with what you can do virtually from a game standpoint, but you actually, on the character side, put this into your wheelhouse and put it more towards your advantage because a physical character has a massively higher impact, uh, physically than virtually.

Lex Fridman 20:03 This is... Okay, can I just pause on that?

Boris Sofman 20:05 Yeah.

Lex Fridman 20:05 'Cause this is so brilliant. When I, uh... For people who don't know, Cozmo plays games with you.

Boris Sofman 20:10 Yep.

Lex Fridman 20:11 But there's also a depth of character. And I actually, when I was, you know, uh, playing with it, I wondered exactly what is the compelling aspect of this. Because to me, obviously, I'm, I'm biased, but to me, the character-

Boris Sofman 20:26 Yeah.

Lex Fridman 20:26 ... I gu- What I enjoyed most, honestly, or what got me to return to it, is the character.

Boris Sofman 20:32 That's right.

Lex Fridman 20:33 But that's, that's a fascinating discussion of, um... You're right. Ultimately, you cannot compete on the quality of the gaming experience.

Boris Sofman 20:42 It's too restrictive. The physical world is just too restrictive.

Lex Fridman 20:44 Yeah.

Boris Sofman 20:44 And, uh, you don't have a graphics engine. It's like all this. But on the character side, we, uh... And clearly we moved in that direction as, like, kind of the, the, the, the winning path. And, um, we partnered up with this, uh, really... We immediately, like, went towards Pixar. And Carlos Pena, he was, um, one of, like... Had been at Pixar for nine years. He'd worked on tons of the movies, including WALL-E and others. And just immediately kinda spoke the language and it just clicked on how you think about that, like, kinda magic and drive. And then he, we built out a team, uh, you know, with him as, like, a really kind of prominent kinda driver of this with different types of backgrounds and animators and character developers where, um, we put these constraints on the team, but then got them to really try to create magic despite that. And we converged on this system that was at the overlap of character and the character AI that where, if you imagine the dimensionality of emotions, happy, sad, angry, surprised, confused, uh, um, scared, like, you think of these extreme emotions, um, uh, we almost, like, kind of put this challenge to kind of populate this library of responses on how do you show the extreme, uh, response that, like, goes to the extreme spectrum on angry or-

Lex Fridman 22:01 Mm-hmm.

Boris Sofman 22:01 ... frustrated or whatever? And, and so that gave us a lot of intuition and learnings. And, um, and then we started parameterizing them where it wasn't just a fixed recording, but they were parameterized and had randomness to them where you could have infinite permutations of happy and surprised and so forth. Um, and then we had a behavioral engine that took the context from the real world and would interpret it and then create kind of probability mappings on what sort of responses you would have that actually made sense. And so if Cozmo saw you for the first time in a day, um, he'd be really surprised and happy in the same way that the first time you walk in and, like, your toddler sees you, they're so happy, but they're not gonna be that happy for the entirety of your-

Lex Fridman 22:39 Yeah.

Boris Sofman 22:39 ... next two hours. But, like, you have this, like, spike in response. Or if you leave him alone for too long, he gets bored and starts causing trouble and, like, nudging things off the table.

Lex Fridman 22:48 Yeah.

Boris Sofman 22:48 Um, or if you beat him in a game, um, the most enjoyable emotions are him getting frustrated and grumpy-

Lex Fridman 22:53 Yeah.

Boris Sofman 22:53 ... to a point where our testers and our customers would be like, "I had to let him win because I don't want him to be upset." And, uh-

Lex Fridman 23:00 (laughs)

Boris Sofman 23:00 ... and so you, you-

Lex Fridman 23:01 That's awesome.

Boris Sofman 23:01 ... start to, like, create this feedback loop where you see how powerful those emotions are. And just to give you an example, something as simple as eye contact. Um, you don't think about it in a movie, just... Like, it kind of happens, like, you know, camera angles and so forth. Um, but that's not really a prominent source of interaction. What happens when a physical character like Cozmo, when he makes eye contact with you, um, it built universal kinda connection, kids all the way through adults. Um, and it was truly universal. It was not like people stopped caring after 10, 12 years old. Um, and so, uh, we started doing experiments and we found something as simple as increasing the amount of eye contact-

Lex Fridman 23:41 Mm-hmm.

Boris Sofman 23:41 ... like, the amount of times in a minute that he'll look over for your approval to, like, kind of make eye contact, just by, I think, doubling it, we increased the playtime engagement by 40%. Like, you see these sort of, like, kind of interactions where you build that empathy. And, and so we studied pets, we studied, um, virtual characters. There's like... Uh, a lot of times, actually, dogs are, uh, one of the perfect, most perfect, uh, um, influencers behind these sort of interactions. And what we realized is that the games were not there to entertain you. The games were to create context to bring out the character.

Lex Fridman 24:11 Mm-hmm.

Boris Sofman 24:11 And if you think about the types of games that, you know, that you played, they were relatively simple, but they were always ones to create scenarios of either tension or winning or losing or surprise or whatever the case might be. And they were purely there to just, like, create context to where an emotion could feel intelligent and not random.

Lex Fridman 24:28 Mm-hmm.

Boris Sofman 24:28 Uh, and in the end, it was all about the character. Um-

Lex Fridman 24:30 So, yeah, there's so many...... elements to play with here. So you said dogs. What lessons do we draw from cats, who don't seem to give a damn about you?

Boris Sofman 24:39 (laughs)

Lex Fridman 24:40 Is that just another character? Is it-

Boris Sofman 24:42 Uh, it's-

Lex Fridman 24:42 Is it another...

Boris Sofman 24:42 It's- it's just another character. And so you- you could almost like, in the early explorations, we thought it would be really incredible if you had a diversity of characters, where you almost help encourage which direction it goes, just like in a role-playing game.

Lex Fridman 24:54 Mm-hmm.

Boris Sofman 24:54 Um, and you had, uh, like think of like the Seven Dwarves, sort of, and, uh, um, and initially we even thought that it would be amazing if, like, you know, if their, like, you know, like their characters actually helped them be, have strengths and weaknesses in some, like whatever they end up doing. Like some are scared, some are, you know, uh, arrogant, some are, uh, uh, super warm and like kind of friendly, and in the end, we focused on one because it made it very clear that, "Hey, we got to build enough depth here," because you're-

Lex Fridman 25:23 Yeah.

Boris Sofman 25:24 ... kind of trying to expand... It's almost like how long can you maintain a fiction that this character is alive, um, to where the person's explorations don't hit a boundary? Um, which happens almost immediately with- with typical toys.

Lex Fridman 25:36 Mm-hmm.

Boris Sofman 25:36 Um, and, uh, even with video games, uh, how long can we create that immersive experience to where you expand the boundary? And one of the things we realized is that you're, um, just way more forgiving when something's f- has a personality and it's physical. Um, that is the key, uh, that unlocks, uh, robotics interacting, you know, in the physical world more generally, is that- that, uh, the, when you have a, when you don't have a personality and you make a mistake as a robot, the stupid robot made- made a mistake, why is it not perfect? When you have a character and you make a mistake, you have empathy and it becomes endearing and you're way more forgiving. And that was the key that was like, I think goes far, far beyond entertainment.

Lex Fridman 26:14 It, uh, actually builds the depth of the personality, the mistakes. So let me ask the- the movie Her a question then. How... So Kosmos seems, feels like the early days of something that will obviously be prevalent throughout society at a scale that we cannot even imagine. My sense is, it seems obvious that these kinds of characters will permeate society, and that we'll be friends with them, we'll be interacting with them in different ways. In a way we, I mean, you don't think of it this way, but when you play video games, they're kind, they're often cold and impersonal, but even then, uh, you think about role-playing games, you become friends with certain characters in that game.

Boris Sofman 27:02 Yeah.

Lex Fridman 27:02 They're, they don't remember much about you, they- they- they're just telling a story. It's- it's exactly what you're saying, they exist in that virtual world. But if they acknowledge that you exist in this physical world, if the characters in the game remember that you exist, that you, like for me, like-

Boris Sofman 27:19 Yeah.

Lex Fridman 27:19 ... Lex, they understand that I'm a human being who has like hopes and dreams and so on. It seems like there's going to be a, like billions, if not trillions, of Kosmos in the world. So if we look at that future, there's still more questions to ask. How intelligent does that future Kosmos need to be to create fulfilling relationships, like friendships?

Boris Sofman 27:48 Yeah, it's- it's a g- great question. And- and part of it was the recognition that it's going to take time to get there, because it has to be a lot more intelligent, um, because what's good enough to be a magical experience for, uh, you know, an eight-year-old, um, it's a higher bar to do that, be a compan- like a pet in the home, or to help with functional interface in an office environment or in a home, or, uh, and so forth. And so, uh, and the idea was that you build on that, and you kind of get there. And as technology becomes more prevalent and less expensive and so forth, you can start to kind of work up to it. Um, but, you know, you're absolutely right. At the end of the day, um, we almost equated it to how, uh, the touchscreen created like this really novel interface to-

Lex Fridman 28:27 Mm-hmm.

Boris Sofman 28:27 ... physical kind of devices like this. This is the extension of it where you have much richer physical interaction in the real world. This is- this is the enabler for it. Um, and it shows itself in a few kind of really obvious places. So just take something as simple as a voice assistant. Um, you will never, most people will never tolerate, uh, an Alexa or a Google Home just starting a conversation, um, proactively, uh, when you weren't kind of expecting it, because it- it feels weird. It's like you were listening and like, and then now you're kind of... It- it feels intrusive. But if you had a character, um, like a cat that touches you and gets your attention, or a toddler, like you never think twice about it. And what we found really kind of immediately is that, um, these types of characters like Kosmos, they would like roam around and kind of get your attention. And we had a future version that was always on, um, kind of called Vector, uh, people were way more forgiving. Um, and so you could initiate interaction in a way that is not acceptable for, uh, for machines. And in general, um, you know, there's a lot of ways to customize it, but it makes people who are skeptical of technology much more comfortable with it. There was like, there were a couple of really, really prominent examples of this. So when we launched in Europe, and so we were in, um, uh, I think like a do- dozen countries, if I remember correctly. But like we were, we went pretty aggressively in la- uh, in launching in, um, Germany and France and, uh, and UK, and we were very worried in Europe because there's obviously like a really, a socially higher bar for privacy and sec- security, where you- you've heard about how many companies have had troubles on, uh, uh, they might have, things that might have been okay in the US but like are just not okay in Germany and France in particular. Um, and so we were worried about this, because you have, um, you know, Kosmos, who's, um, uh, you know, in- in our- in our future product vector, like where you have cameras, you have microphones, it's kind of connected and like you're playing with kids-

Lex Fridman 30:12 Mm-hmm.

Boris Sofman 30:13 ... and like, and- and these experiences, and you're like, this is like ripe to be like a nightmare if you're not careful.

Lex Fridman 30:18 Mm-hmm. Yes.

Boris Sofman 30:18 Um, and, uh, and the journalists are like notoriously like really, really tough on- on these sort of things. Um, we were shocked that we prepared so much for what we would have to encounter. We were shocked in that not once, from any journalists or customer, did we have any complaints beyond like a really casual kind of question. Uh...... and it was because the, of the character where, um, when the conversation came up, um, it was almost like, "Well, of course he has to see and hear. How else is he gonna be alive and interacting with you?" Um, and it completely disarmed, um, this, like, fear of technology that enabled this interaction to be much more fluid.

Lex Fridman 30:57 Mm-hmm.

Boris Sofman 30:57 And again, like, entertainment was a proving ground, but that is, like, a f- you know, there's, like, ingredients there that carry over to a lot of other, uh, elements down the road.

Lex Fridman 31:06 That's hilarious that we're a lot less concerned about privacy if the, if the thing has value and charisma. (laughs)

Boris Sofman 31:13 (laughs) Yeah, well it-

Lex Fridman 31:13 I mean, that's true for all of-

Boris Sofman 31:14 Well-

Lex Fridman 31:14 ... human to human interaction too.

Boris Sofman 31:16 It's an understanding of intent where, like-

Lex Fridman 31:18 Yeah.

Boris Sofman 31:18 ... "Well, if he's looking at me, he can see me."

Lex Fridman 31:19 Yeah.

Boris Sofman 31:19 "If he's not looking at me, he can't see me," right?

Lex Fridman 31:21 Yeah.

Boris Sofman 31:21 So it's almost like, uh, um, you're communicating intent and, um, with that intent, people are, like, kind of, kind of are more understanding and calmer. And it's a, it's interesting how we just, it was just the earliest kind of version of starting to experiment with this, but, um, it was an enabler and, um, and then, and then you have, like, completely different dimensions where, like, you know, kids with autism had, like, an incredible connection with Kosmo that just went beyond anything we'd ever seen. And we have, like, these just letters that we would receive from parents and we had some research projects kind of going on with some universities on s- studying this. But, um, there are, like, there's an interesting dimension there that got unlocked that just hadn't existed before, um, that has these really interesting kind of links into society and, uh, and, uh, potential building block of

Boris Sofman 32:04 future experiences.

Lex Fridman 32:05 So, if you look out into the future, do you think we will have beyond a particular game, you know, a companion, like, uh, like Her, like the movie Her or, like, a Kosmo that's kinda asks you how your day went too, right?

Boris Sofman 32:24 Yeah.

Lex Fridman 32:24 Ma- You know, like a friend.

Boris Sofman 32:26 Yeah.

Lex Fridman 32:26 Do, would you... How, how many years away from that do you think we are? What's your intuition?

Boris Sofman 32:30 Good question. So, I think the idea of a different type of character, like more closer to, like, kind of a pet style companionship-

Lex Fridman 32:37 Yes, that's-

Boris Sofman 32:37 ... will come way faster. Um, and there's a few reasons. One is, like, to, to do something like in Her, that's, like, all, effectively almost general AI, and the bar is so high that if you miss it by a bit, you hit the uncanny valley where it just becomes creepy and, like, and not, um, not, not appealing, um, because the closer you try to get to a human in form and interface and voice, the harder it becomes. Whereas, you have way more flexibility on still landing a really great experience if you embrace the idea of a character. And that's why, um, one of the other reasons why we didn't have a voice, uh, and also why, like, a lot of video game characters, uh, like, Sims for example, does not have a voice when you, uh, when you think about it. It was, uh, it wasn't just a cost savings, like, for them. It was actually, for all of these purposes, it was because when you have a voice you immediately narrow down the appeal to some particular demographic or age range or, um, kind of style or gender. Uh, if you don't have a voice, people interpret what they want to interpret, um, and an eight year old might get a very different interpretation than a 40 year old, um, but you create a dynamic range. And so you just, you can lean into these advantages much more, um, in something that doesn't resemble a human. And so that'll come faster.

Lex Fridman 33:49 Mm-hmm.

Boris Sofman 33:50 Uh, I don't know when a human-like... That's just, uh, still, like, not... just complete R&D at this point. The, the chat interfaces are getting way more interesting and, and richer, but it's still a long way to go to kind of pass the test of, you know.

Lex Fridman 34:04 Well, let me, like, let's consider, like, let me play devil's advocate. So Google is a very large company that's servicing a l- uh, it's creating a very com- compelling product that wants to provide a service to a lot of people. But let's go outside of that. Th- you said characters.

Boris Sofman 34:21 Yeah.

Lex Fridman 34:22 It feels like... A- and you also said that it requires general intelligence to be a successful participant in a relationship, which could explain why I'm single. This, this very...

Boris Sofman 34:32 (laughs)

Lex Fridman 34:32 But the... I, I honestly wanna push back on that a little bit could... because I feel like is it possible that if you're just good at playing a character-

Boris Sofman 34:41 Yeah.

Lex Fridman 34:42 ... you know, in, in a movie there's a bunch of characters. If you just understand what creates compelling characters and then you, you just are that character and you exist in the world and other people find you and they connect with you just like you do-

Boris Sofman 34:55 Yeah.

Lex Fridman 34:55 ... when you talk to somebody at a bar, "I like this character. This character's kind of shady, I don't like them," you pick the ones that you like and, you know, maybe it's somebody that's, uh, reminds you of your fa- father or mother, I don't know what it is, but the, the, the Freudian thing, but th- there's some kind of connection that happens and that's, that, that's the Kosmo you connect to.

Boris Sofman 35:14 Yeah.

Lex Fridman 35:14 That's the future Kosmo you connect... And that's... So, so I guess the statement I'm trying to make, is it possible to achieve a depth of friendship without solving general intelligence?

Boris Sofman 35:24 Yeah. I think so. And it's about intelligent kind of constraints, right?

Lex Fridman 35:27 Right.

Boris Sofman 35:27 And just, uh, you set expectations and constraints such that in the space that's left you can be successful. And so, you can do that by having a very focused domain that you can operate in. For example, you're a customer support agent for a particular product-

Lex Fridman 35:40 Right.

Boris Sofman 35:40 ... and you create intelligence and a good interface around that. Or, uh, you know, kind of in the personal companionship side, you can't be everything to, uh, ac- across the board. You, you kind of solve those constraints. And I think, uh, I think it's possible. My, my worry is I, like I... Right now, I don't see anybody that has picked up on where kind of Kosmo left off-

Lex Fridman 36:02 Yes.

Boris Sofman 36:02 ... and is pushing on it in the same way. And so, I don't know if it's a sort of thing where similar to, like, how, you know, in dot-com there were all these concepts that we considered, like, you know, that didn't work out or, like, failed or, like, were too early or whatnot and then 20 years later you have these, like, incredible successes on almost the same concept. Like, it might be that sort of thing where, like, there's another pass at it that happens in 5 years or in 10 years. But, um, it does feel like that appreciation of that, like, the, this, the three-legged stool, if you will, between, like, you know, the hardware, the AI and the character. Um, that balance, it's hard to, uh, I'm not aware of, of, uh, any pro- anywhere right now where, like, that same kind of aggressive drive with the value on the character is, uh, is happening. And so...

Lex Fridman 36:44 To me-... just a prediction, exactly as you said, something that looks awfully a lot like Cozmo, not in the actual physical form, but in the three-legged stool. Something like that, in some number of years, will be a trillion-dollar company. I don't understand, like it's obvious to me-

Boris Sofman 37:01 Yeah.

Lex Fridman 37:01 ... that like character, not just as robotic companions, but in all our computers, they'll be there. It's like, uh, Clippy was like two legs of that stool or something like that.

Boris Sofman 37:17 Yeah.

Lex Fridman 37:17 I mean that, those are all different attempts as you... And what- what's really confusing to me is they- they're born, these attempts, and they- they, everybody gets excited, and for some reason they die.

Boris Sofman 37:29 Mm-hmm.

Lex Fridman 37:29 And then nobody else tries to pick it up. And then maybe a few years later, a- a crazy guy like you comes- comes around with just enough brilliance and vision to create this thing and is born. A lot of people love it. A lot of people get excited. But maybe the timing is not right yet.

Boris Sofman 37:46 Yeah.

Lex Fridman 37:47 And then- and then when the timing is right, it just blows up.

Boris Sofman 37:51 Yeah.

Lex Fridman 37:51 It just keeps blowing up more and more until it just blows up, and I guess everything in- in-

Boris Sofman 37:56 That's right.

Lex Fridman 37:56 ... the full span of human civilization collapses eventually. (laughs)

Boris Sofman 37:59 Yeah. And that- that wouldn't surprise me at all. And like what's gonna be different in another five years or ten years or whatnot? Physical component cost will-

Lex Fridman 38:06 Competition for labor.

Boris Sofman 38:06 ... continue to come down, uh, in price. And, you know, mobile devices and computation is gonna become more and more prevalent as well as cloud as a, as a big tool, uh, to offload cost. Um, AI is gonna be a massive transformation compared to what we dealt with, uh, where, um, everything from voice understanding to, um, uh, to just, you know, kind of a broader contextual, uh, understanding and mapping of, uh, of semantics and, uh, understanding scenes and so forth. And then the character side will continue to kind of, you know, progress as well. Because that magic does exist, it just exists in different forms. Um, and you have just the brilliance of, uh, uh, the tapping and animation and then, you know, these other areas where, um, that is, that was a big unlock in, um, you know, in film obviously. Uh, and so I think, yeah, the pieces can reconnect and the building blocks are actually gonna be way more impressive than they were five years ago, so.

Lex Fridman 38:59 So in 2019, uh, Anki, the company that created Cozmo, the company that you started, had to shut down. How did you feel at that time?

Boris Sofman 39:11 Yeah. It was tough. Uh, that was a really emotional stretch and it was pr- really tough year one, like about a year ah- ahead of that was actually a pretty brutal stretch because we were, um, kind of life- life or death on many, many moments, um, just navigating these insane kind of just ups and downs and, um, barriers. And the thing that made it, like, um, like just- just sort of rewinding a tiny bit, like what, you know, what ended up being really challenging about it as a business where is, um, from a commercial standpoint and customer reception standpoint, there's a lot of things you could point to that were like, you know, pretty big successes. Sold millions of units, uh, like, you know, got to like pretty serious revenue, like ki- kind of close to 100 million annual revenue, um, uh, number one kind of product in kind of various categories. But it was pretty expensive, ended up being very seasonal where something like 85% of our volume was in Q4, um, because it was a p- you know, a present and- and it was expensive to market it and explain it and so forth. Um, and even though- though the volume was like really sizable and like the reviews were really fantastic, um, forecasting and planning for it and managing the cash operations was just brutal. Like, it was absolutely brutal. You don't think about this when you're starting a company or when you have a few million in, you know, in- in revenue because it's just your biggest costs are kind of just your headcount and operations and everything's ahead of you. But we got to a point where, um, you know, you, if you look at the entire year, you have to operate your company, pay all, you know, the people and so forth, you have to pay for the manufacturing, the marketing and everything else to do your sales in mostly November, December, and then get paid in December, January by retailers. And those swings were pretty, um, were really rough, um, and just made it like so difficult. Because the more successful you became, the more wild those swings became, um, because you'd have to like spend, you know, tens of millions of dollars on inventory, tens of millions of dollars on marketing, and tens of millions of dollars on payroll and everything else. And then-

Lex Fridman 41:11 Just the bigger dip and then you waiting for the Q4.

Boris Sofman 41:14 Wild. Yeah, and it's not a business that like is recurring kind of month to month and predictable and it's just... And then you're locking in your forecasts in July, um, you know, maybe August, uh, if you're lucky. Um, and, uh, and it's also like very hit driven and seasonal where like you don't have the sort of continued, uh, kind of slow growth like you do in some other, uh, consumer electronics industries. And so before then, like hardware kind of like went out of favor too. And so you had Fitbit and GoPro drop from 10 billion revenue to 1 billion revenue and hardware companies are getting valued at like 1X revenue oftentimes-

Lex Fridman 41:45 Mm-hmm.

Boris Sofman 41:45 ... um, which is tough, right? And so we effectively kind of got caught in the middle where we were trying to quickly evolve out of entertainment and move into some other categories, but you can't let go of that business because like that's what you're valued on, that's what you're raising money on. Um, but there was no path to pro- kind of pure profitability just there because it was, you know, such, you know, uh, specific type of price points and so forth. And so, um, we tried really hard to make that transition and, um, yeah, we had a- a financing round that fell apart at the last second and effectively there was just no path to kind of get through that and get to the next kind of like holiday season. And so we ended up, um, uh, selling some of the assets and kind of winding down the company. It was, uh, it was brutal. Like we... I was very transparent with the company, like, and the- the team while we were going through it, where actually despite how challenging that period was, very few people left. I mean, like people loved the vision, the team, the culture, the like kind of chemistry and kind of what we were doing. There was just a huge amount of pride there and that we wanted to see it through and we felt like we had a shot to kind of get through these checkpoints. Um, we ended up, uh... And I mean, by brutal, I mean like literally like days of cash, like three, four different times, uh, runway like in the year, you know, kind of before it, um, where you're like playing games of chicken on negotiating credit line-... timelines and, like, repayment, uh, terms and how to get, like, a bridge loan from an investor. It's just like-

Lex Fridman 43:10 Yeah.

Boris Sofman 43:11 ... level of stress that, like, as, as hard as things might be anywhere else, like, you'll never come w- you'll come close to that where you feel that, like, responsibility for, you know, 200 plus people, right? Um, and so we were very transparent during our fundraise on who we're talking to, the challenges, um, that we have, w- how it's going and when things were going well, when things were tough. Um, and so it wasn't a complete shock when it happened, but it was just very emotional where, like, I, you know, like, you know, when we announced it finally that, like, um, you know, we've... you know, basically we're just, like, watching kind of, like, you know, the runway and trying to kind of time it, and when we realized that, like, we didn't have any more outs, we wanted to, like, kind of wind it down, make sure that it was, like, clean and, you know, we could, like, kind of take care of people the best we could. But yeah, like, broke down crying at all, you know, hands and somebody e- else had to step in for a bit. And like, it was just very, very emotional, but the beautiful part is, like, afterwards, like, everybody stayed at the office till, like, 2:00, 3:00 in the morning just, like, drinking and hanging out and telling stories and celebrating, and it was just, like, one of the best, uh... for many people was, like, the best kind of work experience that they had, and there was a lot of pride in what we did. And there wasn't anything obviously we could point to that like, "Hey, if only we had done that different, things would have been completely different." It was just, like, the physics didn't line up, uh, and, uh, um, but the experience was pretty, uh, incredible, but it was hard. Like, it was, uh... it had this feeling that there was just, like, incredible beauty in both the technology and products and the team that, um, uh, you know, there's, there's a lot there that, like, in the, you know, right context could have been, uh, pretty incredible, but it was, um, emotional, just...

Lex Fridman 44:47 (sighs) Yeah, just thinking, I mean, just looking at this company, like you said, the product and technology, but the vision, the implementation, you got the cost down very low.

Boris Sofman 44:58 Yeah.

Lex Fridman 44:58 And the p- the compelling, the nature of the product was great. So many robotics companies failed at this, at they- o- the robot was too expensive, it didn't have the personality, it didn't really provide any value, like, a sufficient value to justify the price. So, like, you, you succeeded where basically every single other robotics company, or most of them that are, like, go in the category of social robotics, have kind of failed. And, I mean, it, it's, um, it's, it's quite tragic. I remember, uh, reading that, and I'm not sure if I talked to you before that happened or not, but I remember, you know, I'm distant from this. I remember being heartbroken reading that because, like, if, if Cozmo's not gonna succeed, what is going to succeed? 'Cause that to me was incredible. Like, the, the, the- it wo- it was an incredible idea. Cost is down the minim- the, the, the, um... it's just, like, the most minimal design in physical form that you could do. It's really compelling. The balance of games. So it's, it's a, it's a fun toy, it's a great gift for all kinds of age groups, right?

Boris Sofman 46:12 Yeah.

Lex Fridman 46:12 It's just, it's, it's compelling in every single way and it seemed like, uh, it was a, a huge success and it, it, it failing was... I don't know, there was heartbreak on many levels for me just as an external observer, is I was thinking, "How hard is it to run a business?"

Boris Sofman 46:30 (laughs)

Lex Fridman 46:30 That's, that's what I was thinking-

Boris Sofman 46:32 Oh.

Lex Fridman 46:32 ... like if this failed, this must have failed because the... it's obviously not, like, yeah, it's bu- it's business.

Boris Sofman 46:39 Yeah.

Lex Fridman 46:39 May- maybe it's some aspect of the manufacturing and so on.

Boris Sofman 46:41 Yeah.

Lex Fridman 46:41 But I'm now realizing it's also not just that, it's-

Boris Sofman 46:45 Yeah, and then, and-

Lex Fridman 46:46 ... sales, marketing, all those-

Boris Sofman 46:47 Oh, it's everything, right? Like, how do you explain something that's like a new category to people-

Lex Fridman 46:51 Yeah.

Boris Sofman 46:51 ... that, like, have all these predispositions? And so, like, uh, you know, it l- it, it had some of the hardest elements of... if you were to pick a business, it had the, some of the hardest, uh, um, customer dynamics because, like, to sell a $150 product, you got to convince both the child who want it and the parents to agree that it's valuable. So you're having, like, this dual-prong marketing challenge. You have manufacturing, you have, like, really high precision on the components that you need, you have the AI challenges. So, there were a lot of tough elements, but it was this feeling where, like, it was just really great alignment of unique strength across kind of, like, all these different areas. Just like incredible, like, you know, kind of character and animation team between this, like, Carlos, and there was, like, a character director, Day, that came on board and, like, you know, really great people there. The AI side, the, um, uh, the manufacturing, the, you know, where, um, like, never missing a launch, right? And Ashley, you know, he kind of hidden that quality. It was, um, yeah, it was, it was heartbreaking, but, uh, here's one neat thing is like we had, we had so much, like, fan mail from kind of kids with their parent- parents, like...

Lex Fridman 47:52 (laughs)

Boris Sofman 47:52 I actually, like, there was a bunch that collected in the end-

Lex Fridman 47:54 Yeah.

Boris Sofman 47:54 ... that, um, I actually saved and, like, I never... it was too emotional to open it and I still haven't opened it. Um, and so I actually have this giant envelope with like a stack this much of, like, letters from, you know, kids and families, just like every, you know, permutation- permutation you can imagine. And so planning to kind of, I don't know, maybe like a five year re- you know, five year to e- some year reunion, just inviting everybody over and we'll just, like, kind of dig into it and, um, kind of bring back some memories. But, um, you know, good impact and, uh, um-

Lex Fridman 48:22 Well, I, I think there will be companies, uh, maybe Waymo and, uh, Google will be somehow involved, that will carry this flag forward and will, uh, will make you proud. Whether you're involved or not, I think this is one of the greatest robotics companies in the history of robotics. So, you should be proud. It's still tragic to know that, you know... 'cause you read all the stories of Apple and, uh, and, uh, let's see, SpaceX and, like, companies that were just on the verge of failure several times through that story and they just... it's almost like a roll of the dice-

Boris Sofman 48:59 Yeah.

Lex Fridman 48:59 ... they succeeded.

Boris Sofman 49:01 Yeah.

Lex Fridman 49:01 And here's a roll of the dice that just happened to go...

Boris Sofman 49:04 And that's the appreciation that, like, when you really, like, talk to a lot of the, you know, founders, it's like everybody goes through those moments and sometimes it really is a matter of, like, you know, timing, a little bit of luck, like some things are just out of your control. And, um-... uh, and you'd be... You'd get a much deeper appreciation for, um, just the dimensionality of, of that challenge. But, um, the great thing is, is that, like, a lot of the team actually, like, stayed together and so, um, there were actually, uh, you know, a couple of companies that we, where we kind of kept big chunks of the team together, and we actually kind of helped align this, uh, um, you know, to, to help people out as well. Um, and one of them was Waymo where, uh, a majority of the AI and robotics team-

Lex Fridman 49:44 Mm-hmm.

Boris Sofman 49:44 ... actually had the exact background, uh, that you would look for in, like, kind of AV space. It was a space-

Lex Fridman 49:49 Mm-hmm.

Boris Sofman 49:49 ... that a lot of us, like, you know, were... You know, worked on in grad school, were always passionate about and ended up, uh... You know, maybe the timing... You know, ser- serendipitous timings from another perspective where, like, uh, um, we kind of landed in a really unique, um, circumstance that's actually been quite exciting too.

Lex Fridman 50:04 Mm-hmm. So it's interesting to ask you just your thoughts. Uh, Cozmo still lives on under Dreamlabs, I, I think. Um-

Boris Sofman 50:13 Yeah.

Lex Fridman 50:13 Is that... Are you tracking the progress there or is it too much pain? (laughs) Uh, is it-

Boris Sofman 50:19 Yeah.

Lex Fridman 50:20 ... are you... Is that something that you're excited to see where that goes?

Boris Sofman 50:24 So keeping an eye on it, of course, just out of curiosity and obviously just to kind of c- care for the product line. I think, um, it's deceptive how complex it is to manufacture and evolve that product line, um, and the amount of experiences that are required to complete the picture and be able to move that forward.

Lex Fridman 50:43 Mm-hmm.

Boris Sofman 50:43 And I think that's gonna make it pretty hard to do something really subs- substantial with it. It would be cool if, like, even the product in the way it was was able to be manufactured.

Lex Fridman 50:52 Yes.

Boris Sofman 50:52 Y- you know, again, that would

Narrator 50:53 Understand.

Lex Fridman 50:53 Which is the current goal, I suppose.

Boris Sofman 50:54 Yeah.

Lex Fridman 50:54 Okay.

Boris Sofman 50:54 Which will be neat. Um, but I... It's... I, I think, uh, it was... It's deceptive how tricky that is on, like, everything from the quality control of the details and, um... And then, like, technology changes that forces you to rick- reinvent and update certain things. Um, so, uh, I haven't been super close to it, but just kind of keeping an eye on it. Um...

Lex Fridman 51:13 Yeah. It's really interesting how, how... It's deceptively difficult, just as you're saying.

Boris Sofman 51:18 Yeah.

Lex Fridman 51:18 For example, um, uh, those same folks, uh, and I've spoken with them, they're... They par- partnered up with Rick and Morty, uh, creators to, uh, to do the Butter Robot.

Boris Sofman 51:30 Yeah.

Lex Fridman 51:30 Which I, I love the idea. I, I just recently... I've, I'd kind of half-assed watched Rick and Morty previously, but now I just watched, like, the first season. It's such a brilliant show. It... I, like... I did not understand how brilliant that show is. And obviously I think in season one is where the Butter Robot comes along for just a few minutes or whatever, but I just fell in love with the Butter Robot. The, sort of the... That particular character, just like you said, there's characters. You can create personalities, you can create... And that particular, uh, robot who's doing a particular task realizes, you know, this... Like, realizes... Asks the existential question, the s- the Myth of Sisyphus question that, uh, uh, Camus writes about. He's like, "Th- is this all there is, is he moves butter?" But, you know, (laughs) that realization-

Boris Sofman 52:22 Yeah.

Lex Fridman 52:22 ... that's a bea- that's a beautiful little realization for a robot that, "I'm... My purpose is very limited-

Boris Sofman 52:28 Limited, yeah. (laughs)

Lex Fridman 52:28 ... to this particular task." It's a bea- it's, it's humor, of course, it's darkness, it's a beautiful mix. But so they wanna release that Butter Robot, but something tells me that to do the si- same depth of personality as Cozmo had, the same richness, it would be... On the manufacturing, on the AI, on the storytelling, on the design, it's going to be very, very difficult. It could be a cool sort of, uh, toy for Rick and Morty fans-

Boris Sofman 52:58 Yeah.

Lex Fridman 52:58 ... but to create the same depth of ex- existential angst-

Boris Sofman 53:05 Yeah.

Lex Fridman 53:05 ... that the Butter Robot symbolizes is, is really... That's the brave effort you succeeded at with Cozmo, but this not easy. It's really difficult.

Boris Sofman 53:14 Not easy, and you can fail on almost any one of the kind of dimensions and, like, uh... And, yeah, it takes a... You know, a, uh, yeah, unique convergence of a lot of different skill sets to, to try to pull that off. Yeah.

Lex Fridman 53:26 On this topic, let me ask you for some advice because, uh, as I've been watching Rick and Morty, I s- I s- I told myself, "I have to build the Butter Robot just as a hobby project." And so, uh, I got a nice platform for it with treads and, and there's a camera-

Boris Sofman 53:40 (laughs)

Lex Fridman 53:40 ... that moves up and down and so on.

Boris Sofman 53:42 Yeah.

Lex Fridman 53:42 Um, I'll probably paint it, uh, but, uh, the question that I'd like to ask... There's obvious technical questions I'm fine with, communication, the personality, storytelling, all those kinds of things. Um, I think I understand the process of that. But how do you know when you got it right? So with, with Cozmo, how did you know, "This is great"? Like, or, "Mm, something is off." Like-

Boris Sofman 54:09 Yeah.

Lex Fridman 54:10 ... is this brainstorming with the team? Do you know it when you see it? Is it like love at first sight? It's like, "This is right"? Or, like, h- I guess if we think of it as an optimization space, is there an uncanny valley where you're like, "Ugh, this is not right," or, "This is right"? Or are a lot of characters right?

Boris Sofman 54:28 Yeah. We stayed away from uncanny valley just by having such a different, uh, like mapping where it didn't try to look like a dog or a human or anything like that. And so, uh, y- you avoided having, like, a weird pseudo-similarity but not quite hitting the mark. Um, but you could, like, just fall flat where just, like, a personality or a charac- you know, character emotion just didn't feel right. And so it actually mirrored very closely the kind of, the iterations that a character director at Pixar would have where you're running through it and you can virtually kind of, like, see what it'll look like. We, we created a plug-in to where... We actually used, like, like Maya, the simu- you know, the animation-

Lex Fridman 55:05 Mm-hmm.

Boris Sofman 55:05 ... tools and then we created a plug-in that perfectly ma- uh, matched it-

Lex Fridman 55:09 Nice.

Boris Sofman 55:09 ... uh, to the physical one. And so you could, like, test it out virtually and then push a button and see it physically play out. And there's, like, subtle differences and so you wanna, like, make sure that that feedback loop is super easy-

Lex Fridman 55:18 Mm-hmm.

Boris Sofman 55:19 ... to be able to test it live. Um, and then sometimes, like, a-... you would just feel it that it's right, and intuitively know, and then you'd also do ... We did user testing.

Lex Fridman 55:28 Mm-hmm.

Boris Sofman 55:29 But it was very, very often that, like, the intu- like, if we found it magical, it would scale and be magical, uh, more broadly. There were not too many cases where like, like we were pretty decent about not like getting too a- t- t- you know, geeking out or getting too attached to something that was super unique to us. Um, but trying to kind of like, you know, put a customer hat on and does it truly kind of feel magical?

Lex Fridman 55:52 Mm-hmm.

Boris Sofman 55:52 And so, in a lot of ways, we just gave a lot of, um, autonomy to the character team to really think about the, you know, character board and mood boards and storyboards, and like, what's the background of this character and how would they react? Um, and they went through a process that's actually pretty familiar, but now had to operate under these unique constraints. Um, but the moment where it felt right, um, kind of took a fairly similar journey than like a- g- as a character in an animated film, actually. It was quite cool.

Lex Fridman 56:19 Well, the- the thing that's really important to me, and I wonder if it's possible ... Well, I hope it's possible, pretty sure it's possible, is for me, even though I know how it works, to make sure there's sufficient randomness in the process.

Boris Sofman 56:33 Yeah.

Lex Fridman 56:34 Uh, probably because it would be machine learning based that I'm surprised, that I don't ... I'm surprised by certain reactions. Uh-

Boris Sofman 56:42 Yeah.

Lex Fridman 56:42 ... I'm surprised by certain communication. Maybe that's i- in the form of a question. Um, were you surprised by certain things Kosmo did, like certain interactions?

Boris Sofman 56:52 Yeah. We made it intentionally like, uh, uh, so that there would be some surprise and like, a decent amount of variability in how he'd respond in certain circumstances. And so in the end, like it's, um ... Uh, this isn't- this isn't general AI. This is a w- g- giant like spectrum and library of like parameterized kind of emotional responses and an emotional engine that would like kind of map your current state of the game, your emotions, the world, the people who are playing with you also, so forth, to what's happening. Um, but we could make it feel spontaneous by creating enough diversity, uh, and randomness, uh, but still within the bounds of what felt- felt like very realistic, um, to make that work. And then what was really neat is that we could get statistics on how much of that space we were saturating, um, and then add more animations and more diversity in the places it would get hit more often, so that you stay ahead of the, um, you know, the- the curve and maximize the, uh, the chance that it- it stays feeling alive. Um, and so, uh, but then when you like combine it, like, the permutations and kind of like the combinations of emotions stitched together sometimes surprised us because you see them in isolation, but when you actually see them and you see them live rel- you know, relative to some event that happened in the game or whatnot, like, it was kind of cool to see the combination of the two. And, um, uh, and not too different in other robotics applications where like you get- you get so used to thinking about like the modules of a system-

Lex Fridman 58:14 Mm-hmm.

Boris Sofman 58:14 ... and how things progress through a tech stack-

Lex Fridman 58:16 Mm-hmm.

Boris Sofman 58:16 ... that the real magic is when all the pieces come together and you start getting the right emergent behavior, um, in a way that's easy to lose when you just kind of go too deep into any one piece of it.

Lex Fridman 58:26 Yeah, when the system is sufficiently complex, there is something like emergent behavior, and that's where the magic is. You, as a human being, you can still appreciate the beauty of that magic-

Boris Sofman 58:33 Yeah.

Lex Fridman 58:34 ... of the fin- at the system level. First of all, thank you for humoring me on this. Uh, it's really, really, uh, fascinating. I think a lot of people love this. I- I'd love to just ... One last thing on the butter robot, I promise. (laughs)

Boris Sofman 58:45 (laughs)

Lex Fridman 58:46 In terms of, uh, speech.

Boris Sofman 58:48 Yeah.

Lex Fridman 58:50 Kosmo is able to communicate so much with just movement and face. Do you think speech is too much of a degree of freedom? Like is speech a feature or a bug of, uh, deep, uh, interaction or emotional interaction?

Boris Sofman 59:09 Yeah. For a product, it's too deep right now. It's just not real- uh, you would immediately break the fiction because the-

Lex Fridman 59:17 Yeah.

Boris Sofman 59:17 ... state of the art is just not good enough. Um, and, uh, and that's on top of just narrowing down the demographic where like the way you speak to an adult versus the way you speak to a child is very different.

Lex Fridman 59:27 Yeah.

Boris Sofman 59:27 Um, yet a dog is able to appeal to everybody. Um, and so right now, there is no speech system that is like rich enough and- and subtly reali- realistic enough to feel appropriate. Um, and so we very, very quickly kind of like moved away from it. Now, speech understanding is a different matter where understanding intent, that's a really valuable input. Um, but giving it back requires like a, you know, way, way higher bar, um, given kind of where today's, um, uh, world is. And so, uh, that realization that you can do surprisingly much with, uh, either no speech or j- or kind of tonal, like the way, you know, WALL-E, R2-D2, and kind of other characters are able to, um, it's, uh, quite powerful and it generalizes, um, across cultures and across ages really, really well. I think we're gonna be in that world for a little while where it's still very much an unsolved problem on how to like make something ... It touches on the uncanny valley thing. So if you have legs and you're a big humanoid-looking thing, you have very different expectations and a much narrower degree of what's gonna be acceptable by society-

Lex Fridman 1:00:31 Mm-hmm.

Boris Sofman 1:00:32 ... than if you're a, you know, robot like, uh, like Kosmo or WALL-E, and you can ... Or some other form where you can kind of like reinvent the- the character. Speech has that same property, where speech is so well-understood, um, in terms of expectations by humans, that you have far less flexibility on how to deviate from that and- and lean into your strengths or- and avoid weaknesses.

Lex Fridman 1:00:52 (laughs) But I wonder if there is ... Obviously, there is certain kinds of speech that, uh, activates the uncanny valley and breaks the illusion faster. So, I guess my intuition is we will solve cert- we would be able to create some speech-based personalities-

Boris Sofman 1:01:11 Yeah.

Lex Fridman 1:01:11 ... sooner than others. So for example, I could- I could think of a robot that doesn't know English and is learning English, right?

Boris Sofman 1:01:20 Yeah. Yeah.

Lex Fridman 1:01:21 Those kinds of personalities.

Boris Sofman 1:01:22 A fiction where you're like, uh, you're intentionally kind of like getting a toddler level of, uh-

Lex Fridman 1:01:26 Yeah.

Boris Sofman 1:01:27 ... speech. So that- that's exactly right. So you can have like, uh...... tie it in to the experience where, uh, it is a more limited character, or you embrace the lack of emotions as par- or the lack of, sorry, dynamic range in the-

Lex Fridman 1:01:39 Right.

Boris Sofman 1:01:39 ... speech kind of capabilities and motions as like part of the character itself. And you've seen that in like-

Lex Fridman 1:01:43 Yes.

Boris Sofman 1:01:43 ... kind of fictional characters as well.

Lex Fridman 1:01:45 Yeah.

Boris Sofman 1:01:45 Um, but, uh-

Lex Fridman 1:01:46 That's why this podcast works.

Boris Sofman 1:01:48 (laughs) Yeah, and like, and you kind of had that with like, um, I don't know, I guess like, you know, Data and some of the oth-

Lex Fridman 1:01:54 Yeah, exactly.

Boris Sofman 1:01:55 ... you know, like, um, but yeah, so you have to... And that becomes a constraint that lets you meet the bar, um-

Lex Fridman 1:02:01 See, I, I honestly think like also if you add, uh, drunk and angry, that gives you more constraints that allow you to be, uh, dumber from an NLP perspective.

Boris Sofman 1:02:16 (laughs)

Lex Fridman 1:02:16 Like, there's certain aspects... So if you modify human behavior, like so, so, so forget the sort of artificial thing where you don't know English, toddler thing. We, if you just look at the full range of humans, I think we, there's certain situations where we put up with a like lower level of intelligence in our communication. Like, if somebody's drunk, we understand the situ- that they're probably under the influ- Like, we understand that they're not going to be making any sense. Anger is another one like that. I'm sure there's a lot of other kind of situations like this.

Boris Sofman 1:02:51 Hiding. Yeah.

Lex Fridman 1:02:52 Maybe lo- Uh, yeah, again, language lost in translation, that kind of stuff that I think if you c- if you play with that, uh, what is it? The Ukrainian boy that passed the Turing test, you know, play with those ideas. I think that's really interesting and then you can create compelling characters. But you're right, that's a dangerous sort of road to walk because, uh-

Boris Sofman 1:03:12 Yeah.

Lex Fridman 1:03:12 ... you're adding degrees of freedom that can get you in trouble.

Boris Sofman 1:03:14 Yeah, and that's why like you have these, um, big pushes that, like, for most of the last decade plus, like where you'd have like full, like, hu- human replicas of robots, right? Being down to like skin and like-

Lex Fridman 1:03:26 Yeah.

Boris Sofman 1:03:26 ... kind of in some places. Um, I'm li- I'm li- my, m- my, my personal feeling is like, man, like that's not the direction that's most fruitful right now, um, uh-

Lex Fridman 1:03:37 Beautiful art, right?

Boris Sofman 1:03:38 Yeah.

Lex Fridman 1:03:38 It's not, in terms of a, uh, rich, deep, fulfilling experience, yeah, you're right.

Boris Sofman 1:03:43 Yeah, and, uh-

Lex Fridman 1:03:44 Move away from-

Boris Sofman 1:03:44 ... you're creating a minefield of potential places to feel off, uh-

Lex Fridman 1:03:49 Yeah.

Boris Sofman 1:03:49 ... uh, and then, and then you're sidestepping where like the biggest kind of functional AI challenges are to actually have, you know, kind of like really rich productivity that actually kind of justifies a, you know, kind of h- the higher price points. And that's, that's part of the challenge is like, yeah, like robots are gonna get to like thousands of dollars, tens of thousands of dollars and so forth. But you can imagine what sort of expectation of value that comes with it.

Lex Fridman 1:04:09 Mm-hmm.

Boris Sofman 1:04:10 Um, and so that's where you want to be able to invest the, the, the, the time and, uh, and depth. And so going down the full human replica route, um, creates a gigantic, uh, uh, distraction and really, really high bar that can end up sucking up so much of your resources.

Lex Fridman 1:04:30 So it's weird to say, but

Lex Fridman 1:04:33 you happen to be one of the greatest, at this point, roboticists ever, because you created, uh, this little guy. You were part, obviously, of a great team that created the, the little guy with a deep personality, and are now switching to an entirely, well, maybe not entirely, but a different fascinating, impactful robotics problem, which is autonomous driving, and more specifically the biggest version of autonomous driving, which is autonomous trucking. So you are at Waymo now. Can you give us a big picture overview? What is Waymo? What is Waymo Driver? What is Waymo One?

Boris Sofman 1:05:14 Yeah.

Lex Fridman 1:05:14 What is Waymo Via? Can you give an overview of the company and the vision behind the company?

Boris Sofman 1:05:20 For sure. Waymo, b- by the way, is just, has been eye-opening on just how incredible the, uh, the people and the talent is and how in one company you almost have to create, I don't know, 30 companies worth of like-

Lex Fridman 1:05:31 (laughs)

Boris Sofman 1:05:31 ... technology and capability to like kind of solve the full spectrum of it. So, um, yeah, so I've, I've been at Waymo since, um, 2019, so about two and a half years. So Waymo is, uh, focused on building what we call a driver, which is, uh, creating the ability to have autonomous driving across different environments, vehicle platforms, domains and use cases. Uh, you know, as you know, it got started in, uh, uh, 2009. It was a lo- uh, almost like an immediate successor to the grand challenge in urban challenges that were like incredible, uh, kind of catalysts for this whole space. Um, and so Google started this project, and then eventually Waymo spun out. And so what Waymo's doing is creating, uh, the systems, both, you know, hardware, software, infrastructure, and everything that goes into it to enable and to commercialize autonomous driving. This hits on consumer transportation and ride sharing and kind of vehicles in urban environments. Um, and as you mentioned, it hits on autonomous trucking to, uh, to transport, um, goods. So in a lot of ways, it's transporting people and transporting goods. Um, but at the end of the day, the underlying capabilities that are required to do that are surprisingly better aligned than one might expect, um, where it's the fundamentals of, um, of being able to understand the world around you, process it, make intelligent decisions, and prove that we are at a, a level of safety that enables, uh, large scale autonomy.

Lex Fridman 1:06:56 So from a branding perspective, sort of, uh, Waymo Driver is the system that's irrespective of a particular, uh, v- vehicle it's operating-

Boris Sofman 1:07:07 Yeah.

Lex Fridman 1:07:07 ... and there's... You have a set of sensors that perceive the world, can act in that world, and move this, whatever the vehicle is through the wo-

Boris Sofman 1:07:14 Pla- yeah, vehicle platform. That's right. And so in the same way that you have a driver's license and like your ability-

Lex Fridman 1:07:19 (laughs) Yeah.

Boris Sofman 1:07:19 ... to drive isn't tied to a particular make and model of a car, and of course there's special licenses for other types-

Lex Fridman 1:07:24 Yeah.

Boris Sofman 1:07:24 ... of vehicles, but the fundamentals of a, of a human driver, very, very largely carry over. And then there's uniquenesses related to a particular environment or domain or a particular, um, vehicle type that kind of add some extra additive challenges. But that's exactly right. It's the underlying systems that enable a phy- a, a physical vehicle without a human driver to, uh, very successfully accomplish the task that previously, um, wa- wasn't possible, um-

Lex Fridman 1:07:51 Mm-hmm.

Boris Sofman 1:07:51 ... without, um, you know, hu- 100% human driving.

Lex Fridman 1:07:54 Mm-hmm. And then there's Waymo One, which is the transporting people-

Boris Sofman 1:07:59 That's right.

Lex Fridman 1:07:59 ... from a brand perspective.And just in case we refer to it so people know, and then there's Waymo Via, which is the trucking component. Why Via, by the way? What does that, what does that, what, is it just like a cool sounding name that just-

Boris Sofman 1:08:13 Yeah.

Lex Fridman 1:08:13 ... uh, like, is there a, is there an interesting story there or just, it is a pretty cool sounding name.

Boris Sofman 1:08:18 It's a cool sounding name. I mean, when you think about it, it's just like, well, we're gonna transport it via this and that and that.

Lex Fridman 1:08:23 Oh, cool.

Boris Sofman 1:08:24 So it's just kind of like an allusion to, um, the mechanics of transporting something.

Lex Fridman 1:08:28 Yes, cool.

Boris Sofman 1:08:28 Um, and, uh, and it is a pretty good grouping, and the interesting thing is that even the groupings kind of blur where Waymo One is, like, human transportation, and, uh, there's a fully autonomous service in the Phoenix area that, like, every day is transporting people, and it's pretty incredible to, like, just see, you know, see that operate at reasonably large scale and just kind of happen. And then on the Via side, it doesn't even have to be, uh, like, long-haul trucking is a, like, a, a major focus of, uh, of ours. But down the road, you can stitch together the vehicle transportation as well for local delivery, um, also. And a lot of these requirements for local delivery overlap very heavily with consumer transportation, um, o- obviously, uh, you know, given that you're operating on a lot of the same roads, um, and, uh, uh, and s- navigating the same safety challenges. And so, um, yeah, and Waymo very much is a, uh, you know, multi-product company that, uh, has ambitions in both. They have different challenges, and both are tremendous opportunities. But the cool thing is, is that there's a huge amount of leverage, and this kind of core technology stack now gets pushed on by both sides.

Lex Fridman 1:09:33 Mm-hmm.

Boris Sofman 1:09:33 Um, and that adds its own unique challenges. But the success case is that, um, the challenges that you push on, um, they get leveraged across all platforms and all domains.

Lex Fridman 1:09:43 So the, the, from an engineer perspective, the teams are integrated.

Boris Sofman 1:09:47 It's a mix. So there's a huge amount of centralized kind of core teams that support all applications. And so you think of something like the hardware team that develops the lasers to compute, integrates into vehicle platforms. This is an experience that carries over across, um, you know, any application that we have, and they ebb and flow with both. Then there's, like, really unique, um, perception challenges, planning challenges, like, other, you know, types of challenges where there's a huge amount of leverage on a cortex stack, but then there's, like, dedicated teams that think of, how do you deal with a unique challenge? For example, um, an articulated trailer with varying loads that completely changes the physical dynamics of a vehicle. That doesn't exist on a car, but it becomes one of the most important, um, kind of unique new challenges on a, on a truck.

Lex Fridman 1:10:28 So what's the long-term dream of Waymo Via, uh, the autonomous trucking effort that Waymo is doing?

Boris Sofman 1:10:37 Yeah, so we're starting with developing, uh, L4 autonomy for c- um, Class 8 trucks. These are 53-foot trailers that, uh, capture, like, a big perce- uh, uh, a pretty sizable percentage of the goods transportation in the country. Um, long term, the opportunity is obviously to expand to much more diverse types of vehicles, uh, types of goods transportation, and start to really expand in both the volume and the route feasibility that's possible. And so just like we did on the car side, you start with a single route with a very specific operating kind of domain and constraints that allow you to solve the problem. But then over time, you start to really try to push, uh, against those boundaries and open up deeper feasibility across routes, across surface streets, across environmental conditions, across the type of goods that you carry, the versatility of those goods, and how little supervision is necessary to just start to scale this network. And long term, there's actually, it's a pretty incredible enabler where, um, you know, today you have already a giant shortage of truck drivers. It's, uh, over 80,000 truck driver shortage that's expected to grow to hundreds of thousands in the years ahead. You have really, really quickly increasing demand from e-commerce and just dis- distribution of, uh, where people are located. Um, you have one of the mo- uh, deepest safety challenges of, um, of any profession in the US where, um, there's a, a huge, huge, huge kind of challenge around fatigue and around kind of the, the long routes that are driven. Um, and even beyond kind of the cost and necessity of it, um, there are fundamental constraints built into our logistics network that are tied to, um, the type of human constraints and regulatory constraints that are, uh, tied to trucking today. For example, hour limits on how long a driver can be driving in a single day, um, before they're, uh, they're not allowed to drive anymore, which is a very important safety constraint.

Lex Fridman 1:12:34 Mm-hmm.

Boris Sofman 1:12:34 Um, what that does is it enforces limitations on how far jumps with a single driver could be and makes you very subject to availability of drivers, which influences where warehouses are built, which influences how goods are transported, which influences costs. And so, um, you start to have an opportunity on everything from plugging into existing fleets and brokerages and the existing logistics network, and just immediately start to have a huge opportunity to add value from a, um, you know, cost and driving fuel insurance and safety standpoint, all the way to completely reinventing the logistics network, um, across the United States and enabling something completely different than what it looks like today.

Lex Fridman 1:13:16 Yeah, I had, uh, uh, it'd be published before this. I had a great conversation with Steve Riccelli, who we talked about the manual driving, and he echoed many of the same things that you were talking about. But we talked about much of the, the fascinating human stories of truck drivers. Uh, he was also, was a truck driver for a l- for, for a bit as a, as a grad student to try to understand the depth of the problem. It's, he's a fascinating-

Boris Sofman 1:13:38 Fascinating lives. Uh, we have some drivers that have 4 million miles of lifetime driving experience.

Lex Fridman 1:13:42 Yeah.

Boris Sofman 1:13:42 It's, uh, pretty incredible. And, um, yeah, it's, uh, yeah, learning from them, like some of them were on the road for 300 days a year. It's, um, very d- unique type of lifestyle.

Lex Fridman 1:13:51 So there's fascinating stuff there. Just like you said, there's a shortage of actually, uh, people, uh, truck drivers, uh, t- taking the job counter to what is, I think is publicly believed. Um-... uh, so there's an excess of jobs and a shortage of people to take up those jobs. And just like you said, it's such a difficult problem, and these are experts at driving, at solving this particular problem, and it's fascinating to learn from them to understand, you know, how hard is this problem? And that's the question I want to ask you from a perception, from a robotics perspective, what's your sense of how difficult is a- is autonomous trucking? Maybe you can comment on which scenarios are super difficult, which are more manageable. Is there, is there a way to kind of converge into words how difficult the problem is?

Boris Sofman 1:14:40 Yeah. It's a very good question. So there's, um... And as you can expect, it's a mix. Some things become a lot, uh, uh, a lot e- easier or at least more flexible. Um, some things are harder. And so, you know, on the- on the things that are, like, uh, uh, the tail winds, the benefits,