Rohit Prasad: Amazon Alexa and Conversational AI | Lex Fridman Podcast #57

Lex Fridman 0:00 The following is a conversation with Rohit Prasad. He's the vice president and head scientist of Amazon Alexa, and one of its original creators. The Alexa team embodies some of the most challenging, incredible, impactful, and inspiring work that is done in AI today. The team has to both solve problems at the cutting edge of natural language processing, and provide a trustworthy, secure, and enjoyable experience to millions of people. This is where state of the art methods in computer science meet the challenges of real-world engineering. In many ways, Alexa and the other voice assistants are the voices of artificial intelligence to millions of people, and an introduction to AI for people who have only encountered it in science fiction. This is an important and exciting opportunity. So the work that Rohit and the Alexa team are doing is an inspiration to me, and to many researchers and engineers in the AI community. This is the Artificial Intelligence Podcast. If you enjoy it, subscribe on YouTube, give it five stars on Apple Podcast, support it on Patreon, or simply connect with me on Twitter @lexfridman, spelled F-R-I-D-M-A-N. If you leave a review on Apple Podcast especially, but also Castbox, or comment on YouTube, consider mentioning topics, people, ideas, questions, quotes, in science, tech, or philosophy that you find interesting, and I'll read them on this podcast. I won't call out names, but I love comments with kindness and thoughtfulness in them, so I thought I'd share them. Someone on YouTube highlighted a quote from the conversation with Ray Dalio, where he said that you have to appreciate all the different ways that people can be A players. This connected with me, too. Uh, on teams of engineers, it's easy to think that raw productivity is the measure of excellence, but there are others. I've worked with people who brought a smile to my face every time I got to work in the morning. Their contribution to the team is immeasurable. I recently started doing podcast ads at the end of the introduction. I'll do one or two minutes after introducing the episode, and never any ads in the middle that break the flow of the conversation. I hope that works for you and doesn't hurt the listening experience. This show is presented by Cash App, the number one finance app in the App Store. I personally use Cash App to send money to friends, but you can also use it to buy, sell, and deposit bitcoin in just seconds. Cash App also has a new investing feature. You can buy fractions of a stock, say $1 worth, no matter what the stock price is. Brokerage services are provided by Cash App Investing, a subsidiary of Square and member SIPC. I'm excited to be working with Cash App to support one of my favorite organizations called FIRST, best known for their FIRST Robotics and LEGO competitions. They educate and inspire hundreds of thousands of students in over 110 countries, and have a perfect rating on Charity Navigator, which means the donated money is used to maximum effectiveness. When you get Cash App from the App Store or Google Play and use code LEXPODCAST, you'll get $10, and Cash App will also donate $10 to FIRST, which again, is an organization that I've personally seen inspire girls and boys to dream of engineering a better world. This podcast is also supported by ZipRecruiter. Hiring great people is hard, and to me, is one of the most important elements of a successful mission-driven team. I've been fortunate to be a part of and lead several great engineering teams. The hiring I've done in the past was mostly through tools we built ourselves, but reinventing the wheel was painful. ZipRecruiter is a tool that's already available for you. It seeks to make hiring simple, fast, and smart. For example, Codable co-founder Gretchen Heubner used ZipRecruiter to find a new game artist to join her education tech company. By using ZipRecruiter's screening questions to filter candidates, Gretchen found it easier to focus on the best candidates, and finally hiring the perfect person for the role, in less than two weeks from start to finish. ZipRecruiter, the smartest way to hire. See why ZipRecruiter is effective for businesses of all sizes by signing up, as I did, for free at ziprecruiter.com/lexpod. That's ziprecruiter.com/lexpod. And now, here's my conversation with Rohit Prasad.

Lex Fridman 4:34 In the movie Her, I'm not sure if you've ever seen.

Rohit Prasad 4:37 Yeah.

Lex Fridman 4:37 A human falls in love with the voice of an AI system. Let's start at the highest philosophical level before we get to deep learning and some of the fun things. Do you think this, what the movie Her shows, is within our reach?

Rohit Prasad 4:52 I think, uh, not specifically about Her, but I think what we are seeing is a massive increase in adoption of AI assistance or AI in all parts of our social fabric. And I think it's... What I do believe is that the utility these AIs provide, some of the functionalities that sh- are shown are absolutely within reach.

Lex Fridman 5:20 So the, some of the functionalities in terms of the interactive elements, but in terms of the deep connection that's purely voice-based, do you think such a close connection is possible with voice alone?

Rohit Prasad 5:33 It's been a while since I saw Her, but I would say in terms of the, uh, in terms of interactions which are both humanlike and, in these AI assistants, you have to value what is also superhuman. We, as humans, can be in only one place. AI assistants can be in multiple places at the same time. One with you on your mobile device, one at your home, one at work. So you have to respect these superhuman capabilities too.Plus, as humans, we have certain attributes we are very good at. Very good at reasoning. AI assistance, not yet there. Uh, but in the realm of AI assistance, what they're great at is computation, memory, it's infinite and pure. These are the attributes you have to start respecting. So, I think the comparison with human-like versus the other, other aspect, which is also super human, has to be taken into consideration. So, I think we need to elevate the discussion to not just human-like.

Lex Fridman 6:28 So, there's certainly elements where you just mentioned,

Lex Fridman 6:32 Alexa is everywhere, uh, computationally speaking. So, this is a much bigger infrastructure than just the thing that sits there in the room with you. But it certainly feels, to us mere humans, that there's just another little creature there when you're interacting with it. You're not interacting with the entirety of the infrastructure, you're interacting with the device. The feeling is, okay, sure, we anthropomorphize things, but, uh, that feeling is still there. So, what do you think we, as humans, the purity of the interaction with a smart assistant, what do you think we look for in, in that interaction?

Rohit Prasad 7:10 I think in the certain interactions, I think will be very much where it does feel like a human, uh, because it has a persona of its own. And in certain ones, it wouldn't be. So, I think a simple example to think of it is if you're walking through the house and you just wanna turn on your lights on and off, and you're issuing a command, that's not very much like a human-like interaction, and that's where the AI shouldn't come back and have a conversation with you. Just, it should simply complete that command. Uh, so those, I think the blend of we have to think about this as not human-human alone.

Lex Fridman 7:43 Mm-hmm.

Rohit Prasad 7:43 It is a human-machine interaction, and certain aspects of humans are needed, and certain aspects or in situations demand it to be like a machine.

Lex Fridman 7:51 So, I told you, it's gonna be philosophical-

Rohit Prasad 7:53 (laughs) .

Lex Fridman 7:54 ... in parts. Uh, what, what's the difference between human and machine in that interaction? When we interact two humans, especially those that are friends and loved ones, versus you and a machine that you also are close with?

Rohit Prasad 8:10 I think the, uh, you have to think about the roles the AI plays, right? So, and it differs from different customer to customer, different situation to situation. Uh, especially I can speak from Alexa's perspective, it is a companion, a friend at times, an assistant, and an advisor down the line. So, I think most AIs will have this kind of attributes, and it will be very situational in nature. So, where is the boundary? I think the boundary depends on exact context in which you're interacting with the AI.

Lex Fridman 8:39 So, the depth and the richness of natural language conversation has been, by Alan Turing, been used to try to define what it means to be intelligent.

Rohit Prasad 8:50 Mm-hmm.

Lex Fridman 8:50 You know, there's a lot of criticism of that kind of test, but w- what do you think is a good test of intelligence, in your view, in the context of the Turing test, and Alexa, with the Alexa Prize, this whole realm? Do you think about this human intelligence, what it means to define it, what it means to reach that level?

Rohit Prasad 9:10 I do think the ability to converse is an, uh, sign of an ultimate intelligence. I think that is no question about it. So, if you think about all aspects of humans, there are sensors we have, and, uh, those are basically a data collection mechanism. And based on that, we make some decisions with our sensory brains, right? And from that perspective, I think that there are elements we have to talk about, how we sense the world, and then how we act based on what we sense. Those elements, clearly machines have. But then, there's the other aspects of computation that is way better. I also mentioned about memory again, in terms of being near infinite, depending on the storage capacity you have. And the retrieval can be extremely fast and pure, uh, in terms of like, there's no ambiguity of, "Who did I see when?" (laughs) , right? I mean, if you, machines can remember that quite well. So, it, again, on a philosophical level, I do subscribe to the fact that to con-, be able to converse, and as part of that, to be able to reason based on the world knowledge you've acquired and the sensory knowledge that is there, is definitely very much the essence of intelligence. But intelligence can go beyond human level intelligence, based on what machines are getting capable of.

Lex Fridman 10:29 So, what do you think, maybe stepping outside of Alexa, broadly as an AI field, what do you think is a good test of intelligence? Put it another way, outside of Alexa, because so much of Alexa is a product, is an experience for the customer. On the research side, what would impress the heck out of you if you saw? You know, what is the test where you said, "Wow, this thing is now starting to encroach into the realm of what we loosely think of as human intelligence"?

Rohit Prasad 11:00 So, well, we think of it as AGI and human intelligence-

Lex Fridman 11:03 AGI.

Rohit Prasad 11:03 ... all together, right? So, in some sense (laughs) . And I think we are quite far from that. Uh, I think, uh, an unbiased view I have is that the Alexa's ca-, intelligence capability is a great test. I think of it as there are many other proof points, like self-driving cars, game playing, like Go or chess. Let's take those two for, as an example.

Lex Fridman 11:28 Sure.

Rohit Prasad 11:28 Uh, clearly requires a lot of data-driven learning and intelligence, but it's not as hard a problem as conversing with, as an AI is with humans to accomplish certain tasks, or open domain chat, as you mentioned-

Lex Fridman 11:43 Mm-hmm.

Rohit Prasad 11:43 ... Alexa Prize.In those settings, the key difference is that the end goal is not defined, unlike game playing. You also do not know exactly what state you are in, in a particular goal completion scenario. (laughs) And so sometimes... sometimes you can if it's a simple goal, but if you're... Even certain examples like planning a weekend or... You- you can imagine how many th- things change along the way. Uh, you look for weather, you may change your mind and you, uh, you change the destination, or you want to catch a particular event and then you decide, no, I want this other event I want to go to. So these dimensions of how many different steps are possible when you're conversing as a human with a machine makes it an extremely daunting problem, and I think it is the ultimate test for intelligence.

Lex Fridman 12:32 And don't you think that natural language is enough to prove that conversation?

Rohit Prasad 12:39 Uh...

Lex Fridman 12:39 Just pure conversation?

Rohit Prasad 12:40 From a scientific standpoint, natural language, uh, is a great test. Uh, but I would go beyond... Uh, I don't wanna l- limit it to as natural language or simply understanding an intent or parsing for entities and so forth. We are really talking about dialogue.

Lex Fridman 12:55 Dialogue. Yeah.

Rohit Prasad 12:55 Uh, so- so I would say human machine dialogue is definitely one of the best tests of intelligence.

Lex Fridman 13:03 So

Lex Fridman 13:04 can you briefly speak to the Alexa Prize for people who are not familiar with it?

Rohit Prasad 13:08 Uh-huh.

Lex Fridman 13:08 And- and also just maybe where things stand, and what have you learned and what's surprising? What have you seen that's surprising-

Rohit Prasad 13:16 (laughs)

Lex Fridman 13:16 ... from this incredible competition?

Rohit Prasad 13:18 Absolutely. It's a very exciting competition. Uh, Alexa Prize is essentially a grand challenge in conversational artificial intelligence, where we threw the gauntlet to the universities, uh, who do, uh, active research in the field, to say, "Can you build what we call a social bot that can converse with you coherently and engagingly for 20 minutes?" Uh, that is an extremely hard challenge, uh, talking to someone in a- uh, who you're meeting for the first time or even if you're... (laughs) You've met them quite often, uh, to speak at 20 minutes on any topic, an evolving nature of topics is super hard. Uh, we have completed two successful years of the competition. The first was won with the University of Washington, second with University of California. We are in our third instance. We have a extremely strong team of ten cohorts and the third instance of the, uh, of the Alexa Prize is underway now. And we are seeing a constant evolution. First year was definitely a learning. It was a lot of things to be put together. We had to build, uh, a lot of infrastructure to enable these universities to be able to build magical experiences and c- uh, and do h- high-quality research.

Lex Fridman 14:31 Just a few quick questions.

Rohit Prasad 14:32 Yeah.

Lex Fridman 14:32 Sorry for the interruption. What does failure look like af- uh, in the 20-minute session? So what does it mean to fail not to reach the 20-minute mark?

Rohit Prasad 14:40 Oh, awesome question. Uh, so there are... One, first of all, uh, I forgot to mention one more detail. It's not just 20 minutes, but the quality of the conversation too that matters, and, uh, the beauty of this competition, before I answer that question on what failure means, is first that you actually converse with millions and millions of customers as these social bots. Uh, so during the judging phases, uh, there are multiple phases, uh, before we get to the finals, which is a very controlled judging in a situation where we have, uh, we bring in judges and we have interactors who interact with these social bots, that is a much more controlled setting. But to- till the point we get to the finals, the... all the judging is essentially by the customers of Alexa, and there you basically rate, uh, on a simple question how good your experience was.

Lex Fridman 15:27 Mm.

Rohit Prasad 15:28 Uh, so that's where we are not testing for a 20-minute boundary being cla- uh, crossed, because you do want it to be very much like a clear-cut winner we chosen and- and it's an absolute bar. (laughs) So did you really, uh, break that 20-minute barrier is why we have to test it in a more controlled setting with actors, essentially interactors, and see how the conversation goes. So this is why it's a subtle difference between how it's being tested in the field with real customers versus in the lab to award the prize. So on the latter one, what it means is that essentially the, uh, the j- there are three judges and two of them have to say this conversation has stalled, essentially.

Lex Fridman 16:11 Mm-hmm. Got it, and the judges are human experts that are-

Rohit Prasad 16:15 Judges are human experts.

Lex Fridman 16:16 Okay, great. So th- th- this in the third year, so what's been the evolution? How far? So the DARPA challenge in the first year n- the autonomous vehicles-

Rohit Prasad 16:25 (laughs)

Lex Fridman 16:25 ... nobody finished. In the second year, a few more finished in the desert. Uh, so how far along with- in this, I would say, much harder challenge are we?

Rohit Prasad 16:36 This challenge has come a long way, to the extent that, uh, we're definitely not close to the 20-minute barrier being with coherence and engaging, uh, conversation. I think we are still five to 10 years away in that horizon to complete that. Uh, but the progress is immense, like, uh, you... what you're finding is the accuracy and what kind of responses these social bots generate is getting better and better. Uh, what's even amazing to see that, uh, now there's humor coming in. The bots are quite, uh-

Lex Fridman 17:05 Awesome. (laughs)

Rohit Prasad 17:06 (laughs) You know, y- you're talking about ultimate signs of intelli- uh, signs of intelligence, I think humor is a very high bar-

Lex Fridman 17:11 Mm-hmm.

Rohit Prasad 17:11 ... in terms of what it takes to create humor, uh, and I don't mean just being goofy, I really mean good sense of humor-

Lex Fridman 17:19 Yeah.

Rohit Prasad 17:19 ... is also a sign of intelligence in my mind, and something very hard to do. So these social bots are now exploring not only what we think of natural language abilities, but also personality attributes and, uh, aspects of when to inject an appropriate joke, when to, uh... when you don't know the ques- uh, the domain, how you come back with something more intelligible so that you can continue the conversation if- if you and I are talking about AI and we are domain experts, we can speak to it, but if you suddenly switch a topic to that I don't know of, how do I change the conversation? So you're starting to notice these elements as well.And that's coming from... partly by (laughs) by the a- nature of the 20 minute t- challenge.

Lex Fridman 18:00 Mm-hmm.

Rohit Prasad 18:00 That people are getting quite clever on how to, uh, really converse and, m- uh, essentially mask some of the understanding defects if they exist.

Lex Fridman 18:09 So some of this, this is not Alexa the product, this is f- somewhat for fun, for research, for innovation, and so on. Uh, I have a question sort of in this modern era, there's a lot of... if you look at, uh, Twitter and Facebook and so on, there's- there's discourse, public discourse going on, and some things that are a little bit too edgy, people get blocked and so on. I- um, just outta curiosity, are people in this context pushing the limits? Is anyone using the F word? Is anyone s- uh, sort of pushing back, uh, sort of, uh, you know, arguing, I- I guess I should say, in- as part of the dialogue to really draw people in?

Rohit Prasad 18:48 First of all, let me just back up a bit-

Lex Fridman 18:50 Yeah.

Rohit Prasad 18:50 ... in terms of why we are doing this, right? So, uh, you said it's fun. Uh, I think fun is, uh, more part of the, uh, engaging part for customers. It is one of the most, uh, used skills as well in our skill store. But up- that apart, the real goal was essentially what was happening is with lot of AI research moving to industry, uh, we felt that academia has the risk of not being able to have the same resources at disposal that we have, which is lots of data, massive computing power, uh, and, uh, clear, uh, ways to test these AI advances with, uh, real customer benefits. So we brought all these three together in the Alexa Prize, that's why it's one of my favorite projects in Amazon. And, uh, with that, uh, eh- the secondary effect is, yes, it has become engaging for our customers as well. Uh, we're not there in terms of where we want to- it to be (laughs) , right? But it's a huge progress. But coming back to your question on-

Lex Fridman 19:47 Yeah.

Rohit Prasad 19:47 ... how do the con- conversations evolve, yes, there is some, uh, natural attributes of what you said in terms of argument and-

Lex Fridman 19:53 Yeah.

Rohit Prasad 19:53 ... some amount of swearing. The way we take care of that is that there is a sensitive filter we have built that's used-

Lex Fridman 19:59 Certain keywords and so on.

Rohit Prasad 20:00 It's more than keywords, a little more in terms of, uh, uh, of course there's keyword base too, but there's more in terms of conte- these words can be very contextual as you can see.

Lex Fridman 20:09 Yeah.

Rohit Prasad 20:09 And also the topic can be, uh, something that you don't want a conversation to happen, uh, because this is a communal device as well. A lot of people use these devices. So we have put a lot of guardrails for the conversation to be more useful for, uh, advancing AI and not so much of these, uh, uh, these other issues you attributed, uh, what's happening in the AI field as well.

Lex Fridman 20:33 Right. So this is actually a- a serious opportunity. I didn't use the right word, fun. I think, uh, it's an open opportunity to do some- some of the best innovation in conversational agents in- in the world. I was at-

Rohit Prasad 20:45 Absolutely.

Lex Fridman 20:46 Why just universities?

Rohit Prasad 20:49 Uh, why just universities? Because as I said, uh, I really felt that-

Lex Fridman 20:52 Young minds?

Rohit Prasad 20:53 Yeah, young minds. It's, uh, uh, also to f- uh, if you think about the other aspect of where the whole industry is moving with AI, there's a dearth of talent, uh, in- in... be- given the demands. So you do want universities to have a clear place where they can invent and research and not fall behind with that they can't motivate students. Imagine all grad students left-

Lex Fridman 21:16 Mm-hmm.

Rohit Prasad 21:16 ... to- to industry, uh, like us.

Lex Fridman 21:19 Yeah. Yeah.

Rohit Prasad 21:19 Or, uh, or faculty members, which has happened too. So this is an a- way that if you are so passionate about the field where you feel industry and academia need to work well, this is a great example and a great way for universities to participate.

Lex Fridman 21:35 So what do you think it takes to build a system that wins the Alexa Prize?

Rohit Prasad 21:39 I think you have to start focusing on aspects of reasoning, uh, that it is... there are still more lookups of what intents customers asking for and responding to those, uh, rather than, uh, really reasoning about (laughs) the elements of the, uh, of the conversation. For instance, if you have... if you're playing... if the conversation is about games and it's about a recent sports event-

Lex Fridman 22:10 Mm-hmm.

Rohit Prasad 22:11 ... there's so much context involved and you have to understand the entities that are being mentioned so that the conversation is coherent rather than you suddenly just switch to knowing some fact about, uh, a sports entity and you're just relaying that rather than understanding the true context of the game. Like, you, uh... if you just said, "I learned this fun fact about Tom Brady," rather than really say how he played the game, eh, the previous night, then the conversation is not really that intelligent. Uh, so you have to go to more reasoning elements of understanding the context of the dialogue and giving more appropriate responses, which tells you that we are still quite far. Because a lot of times it's, uh, more facts being looked up (laughs) and something that's close enough as an answer, but not really the answer. Uh, so that is where the research needs to go more, in actual true understanding and reasoning. And that's why I feel it's a great way to do it, because you have an engaged set of, uh, users working to make... help, uh, these AI advances happen in this case.

Lex Fridman 23:17 Y- right. So you mention customers, uh, uh, there quite a bit t- and there's a skill. Uh, what is the-

Rohit Prasad 23:23 Mm-hmm.

Lex Fridman 23:23 ... experience for- for the, uh, for the user that it's helping? So just to clarify-

Rohit Prasad 23:27 Yeah.

Lex Fridman 23:27 ... this isn't, as far as I understand, the Alexa... so this skill is a standalone for the Alexa Prize. I mean, it's focused on the Alexa Prize.

Rohit Prasad 23:35 Yes.

Lex Fridman 23:35 It's not you o- ordering certain things on amazon.com or tr- checking the weather or-

Rohit Prasad 23:39 Yeah.

Lex Fridman 23:39 ... playing Spotify, right? This is a s- separate skill.

Rohit Prasad 23:42 Exactly.

Lex Fridman 23:42 And so you're focused on helping not... uh, h- I don't know, how- how do people, how do customers think of it? Are they having fun? Are they helping teach the system? What's the experience like for them?

Rohit Prasad 23:53 I think it's both actually. Um, uh, and let me tell you how the, uh, how you invoke this skill. So you, all you have to say, "Alexa, let's chat."

Lex Fridman 23:59 Right.

Rohit Prasad 24:00 And then, the first time you, uh, say, "Alexa, let's chat," it comes back with a clear message that you're interacting with one of those university social bots, uh, and there's a clear... so you know exactly-

Lex Fridman 24:10 That's so awesome.

Rohit Prasad 24:10 ... how we interact, right?

Lex Fridman 24:11 Yeah.

Rohit Prasad 24:11 And that is why it's very transparent. Uh, you are being asked to help, right? And, uh, and we have lot of mechanisms where as the, uh, we are in the f- uh, first phase or feedback phase then, you send a lot of emails to our customers. And then they s- uh, they know that this, uh, the team needs a lot of interactions to improve the accuracy of the system. So, we know we have a lot of customers who really want to help these university bots, and they're conversing with that, and some are just having fun with just saying, "Alexa, let's chat." And, uh, also some adversarial behavior to see whether, how much do you understand (laughs) as a social bot. So, I think we have a good healthy mix of all three situations.

Lex Fridman 24:53 So, what is the, if we talk about solving the Alexa challenge, the Alexa Prize, uh, wh- what's the data set of really engaging pleasant conversations look like? 'Cause if we think of this as a supervised learning problem-

Rohit Prasad 25:10 Mm-hmm.

Lex Fridman 25:10 ... I don't know if it has to be, but if it does, maybe you can comment on that.

Rohit Prasad 25:14 Mm-hmm.

Lex Fridman 25:15 Do you think there needs to be a data set of, of, uh, what it means to be an engaging, successful, fulfilling conversation?

Rohit Prasad 25:22 I think that's part of the research question here. This was, I think, uh, ex- uh, we at least got the first part right, which is have a way for universities to, uh, build and test in a real world setting.

Lex Fridman 25:35 Right.

Rohit Prasad 25:35 Now you're asking in terms of the next phase of questions, which we are still, we're also asking by the way, what does success look like, uh, from an optimization function? That's what you're asking.

Lex Fridman 25:46 Yes, definitely.

Rohit Prasad 25:46 In terms of, uh, we as researchers are used to having a great corpus of annotated data and then making our op, uh, then, you know, sort of tune our algorithms on those, right? And, uh, fortunately and unfortunately, in this world of Alexa Prize, that is not the way we are going after it.

Lex Fridman 26:05 Mm-hmm.

Rohit Prasad 26:05 So you have to focus more on learning based on live feedback. Uh, that is another element that's unique, where, uh, just now to, uh, I started with giving you how you ingress and experience this, uh, capability as a customer. What happens when you're done? Uh, so they ask you a simple question, uh, "On a scale of one to five, how, uh, how likely are you to interact with this social bot again?" Uh, that is a good feedback, and customers can also leave more open-ended feedback. And I think partly that, to me, is one part of the question you're asking, which I'm saying is a mental model shift that as researchers also, you have to change your mindset that this is not a DARPA evaluation or an NSF-funded study.

Lex Fridman 26:52 It's real.

Rohit Prasad 26:52 And you have a nice corpus. This is where it's real world. You have real data.

Lex Fridman 26:58 The scale is amazing.

Rohit Prasad 26:59 It's amazing.

Lex Fridman 26:59 And that's, that's, that's a beautiful thing. And then, and then the customer, the user can, uh, quit the conversation at any time.

Rohit Prasad 27:06 Exactly. The user can... that is also a signal for how good you were at that point.

Lex Fridman 27:11 So, and then on, on a scale of one to five, one to three, do they say how likely are you, or is it just a binary?

Rohit Prasad 27:17 Uh, one to five.

Lex Fridman 27:18 One to five?

Rohit Prasad 27:20 Yeah.

Lex Fridman 27:20 Wow, okay. That's such a beautifully constructed challenge. Okay.

Lex Fridman 27:24 You said the only way to make a smart assistant really smart is to give it eyes and let it explore the world. Uh, I'm not sure, it might have been taken out of context, but can you, uh, comment on that? Can you elaborate on that idea? 'Cause I personally also find that idea super exciting from a social robotics, personal robotics perspective.

Rohit Prasad 27:46 Yeah. A lot of things do get taken out of context. My, this particular one was just as philosophical discussion we were having on terms of what does intelligence look like. And the context was, in terms of learning, I think, uh, just we said, we as humans are empowered with many different sensory abilities.

Lex Fridman 28:04 Yeah.

Rohit Prasad 28:05 Uh, I do believe that, uh, eyes are an important aspect of it in terms of, uh, if you think about how we as humans learn, it is quite complex and it's also not unimodal that you are fed a ton of text-

Lex Fridman 28:21 Mm-hmm.

Rohit Prasad 28:21 ... or audio and you just learn that way. No. You are, you learn by experience, you learn by seeing, you're taught by, uh, humans. And we are very efficient in how we learn. Uh, machines, on the contrary, are very inefficient on how they learn, especially these AIs. I think the next wave of research is going to be with less data, not just less human, uh, not, uh, just with less label data, but also with a lot of weak supervision. And where you can increase the, uh, learning rate, I don't mean less data in terms of not having a lot of data to learn from, that we are generating so much data, but it is more about from an aspect of how fast can you learn?

Lex Fridman 29:04 Mm-hmm. So, improving the, the, the quality of the data that's, uh, the quality of the data and the learning process, but-

Rohit Prasad 29:10 I think more on the learning process. I think we have to, we as humans learn with a lot of noisy data, right? (laughs)

Lex Fridman 29:15 Yeah.

Rohit Prasad 29:15 And, uh, and I think that's the part that, uh, I don't think should change. What should change is how we learn, right? So if you look at, you mentioned supervised learning, we have making transformative shifts from moving to more unsupervised, more weak supervision. Those are the key aspects of how to learn. Uh, and I think in that setting, you, I hope you agree with me that, uh, having other senses is very crucial in terms of how you learn.

Lex Fridman 29:43 So, absolutely. And from a, from a machine learning perspective, which I hope we get a chance to talk to a few aspects that are fascinating there. But to stick on the point of sort of, um, a body, you know, uh, embodiment. So Alexa has a body, has a very s- minimalistic-

Rohit Prasad 29:59 Mm-hmm.

Lex Fridman 29:59 ... beautiful interface, or there's a ring and so on. I mean, I'm not sure of all the flavors of, uh, the devices that Alexa lives on, but there's a minimalistic basic interface, uh-And nevertheless, we're humans so I have a Roomba, I have all kinds of robots all over everywhere. So, uh, what do you think the, uh, Alexa of the future looks like if it begins to shift what its body looks like? What, uh, what maybe beyond Alexa, what do you think are the different devices in the home as they start to embody their intelligence more and more, what do you think that looks like? Philosophically-

Rohit Prasad 30:39 Yeah.

Lex Fridman 30:39 ... a fu- a future, what do you think that looks like?

Rohit Prasad 30:41 I think, uh, let's look at what's happening today. You mentioned, I think, our devices as in Amazon devices.

Lex Fridman 30:46 Yeah.

Rohit Prasad 30:46 But I also wanted to point out, Alexa is already integrated on a lot of third-party devices, which also come in lots of forms and shapes. Some in robots, right? Some in, uh, microwaves. (laughs) Some in appliances of, uh, that you use in e- everyday life. So, I think it is... it's not just the shape Alexa takes in terms of form factors, but it's also where all it's available. Uh, it's getting in cars, it's getting in different appliances in homes, even toothbrushes. (laughs) Right?

Lex Fridman 31:18 Yeah.

Rohit Prasad 31:18 So, I think you have to think about it as not, uh, a physical assistant. It will be in some embodiment. As you said, we already have these nice devices, uh, but I think it's also important to think of it, uh, it is a virtual assistant. It is superhuman in the sense that it is in multiple places at the same time. Uh, so I think the, uh, the actual embodiment to, in some sense, to me, doesn't matter.

Lex Fridman 31:46 Right.

Rohit Prasad 31:47 I think you have to think of it as not as human-like, and more of what its capabilities are that derive a lot of benefit for customers.

Lex Fridman 31:58 Right.

Rohit Prasad 31:58 And how there are different ways to delight it... and, uh, delight customers in different experiences. And I think I'm a big fan of it not being ins- just human-like. It should be human-like in certain situations. Alexa, prior social bot in terms of conversation is a great way to look at it. But there are other, uh, uh, scenarios where human-like I think is underselling the abilities of this AI.

Lex Fridman 32:22 So, if I could, uh, trivialize what we're talking about. So, if you look at the way Steve Jobs thought about the interaction with the device that the- that Apple produced, there was a- a extreme focus on controlling the experience by making sure there's only these sp- Apple-produced devices. You see the voice of Alexa being... taking all kinds of forms depending on what the customers want, and that means, uh, that means it could be anywhere from the microwave, to a vacuum cleaner, to the home, and so on. The voice is the essential element-

Rohit Prasad 32:56 Correct.

Lex Fridman 32:57 ... of the interaction.

Rohit Prasad 32:57 I think voice is an essence. It's not all, but it's a key aspect. I think, uh, to your question, in terms of, uh, you should be able to recognize Alexa.

Lex Fridman 33:08 Yeah.

Rohit Prasad 33:08 And that's a huge problem, I think, in terms of... a huge scientific problem, I should say. Like, what are the traits? What makes it look like Alexa? Especially in different settings, and especially if it's primarily voice, what it is. But Alexa's not just voice either, right? I mean, we have devices with a screen. Uh, now you're seeing just other behaviors of Alexa. So, I think we are in very early stages of what that means, and this will be an important topic for the following years. Uh, but I do believe that being able to recognize and tell when it's Alexa versus it's not is going to be important from an Alexa perspective. I'm not speaking for the entire AI (laughs) -

Lex Fridman 33:45 Right.

Rohit Prasad 33:45 ... community, but from, uh... but I think attribution, and as we go into, uh, more of understanding who did what, that identity of the AI is crucial in the coming world.

Lex Fridman 33:58 I think from the broad AI community perspective, that's also a fascinating problem.

Rohit Prasad 34:02 Mm-hmm.

Lex Fridman 34:02 So basically, if I close my eyes and listen to the voice, what would it take for me to recognize that this is Alexa?

Rohit Prasad 34:08 Exactly.

Lex Fridman 34:09 Or at least the Alexa that I've come to know from my personal experience in my home th- through my interactions, that kind of thing.

Rohit Prasad 34:15 Yeah, and the Alexa here in the US is very different than Alexa in UK and Alexa in India, even though they are all speaking English, uh, or the Australian version. Uh, so again, where... so now think about when you go into a different culture, different community, but you travel there,-

Lex Fridman 34:30 Mm-hmm.

Rohit Prasad 34:30 ... what... do you recognize Alexa? I think these are super hard questions, actually.

Lex Fridman 34:34 So

Lex Fridman 34:35 w-... there's a, there's a team that works on personality.

Rohit Prasad 34:37 Yeah.

Lex Fridman 34:37 So if we talk about those different flavors of what it means, culturally speaking, India, UK, US, what does it mean to add... so, so the problem that we just stated, which is fascinating, how do we make it purely recognizable that it's Alexa, assuming that the qualities of the voice are not sufficient? Uh, it, uh, it's also the content of what is being said.

Rohit Prasad 35:01 Yeah.

Lex Fridman 35:01 W- how do we, how do we do that? How does the personality c-

Rohit Prasad 35:03 (laughs)

Lex Fridman 35:03 ... come into play? What's, uh, what, what's that research even look like? I mean, it's such a fascinating speaker

Rohit Prasad 35:08 We have some very fascinating, uh, uh, folks who, from both a UX background and human factors, are looking at these aspects and these exact questions.

Lex Fridman 35:16 Yeah.

Rohit Prasad 35:17 Uh, but I'll definitely say it's not just how it sounds. The choice of words, the tone, not just, I mean the voice identity of it, (laughs) but-

Lex Fridman 35:27 Yeah.

Rohit Prasad 35:27 ... the tone matters, the speed matters. Uh, how you speak, how you enunciate words, how, uh... what choice of words are you using? Uh, how terse are you or how, uh, lengthy in your explanations you are. All of these are factors. Uh, and y- also, you mentioned something crucial that it's... may have... you may have personalized it, uh, Alexa, to some extent-

Lex Fridman 35:49 Yeah.

Rohit Prasad 35:50 ... in your homes or in the devices you are interacting with. So, you as... your individual, how you prefer Alexa sounds can be different than how I prefer. (laughs)

Lex Fridman 36:01 Yeah.

Rohit Prasad 36:01 And we may... uh, and the amount of customizability you want to give is also a key debate we always have.

Lex Fridman 36:06 Right.

Rohit Prasad 36:07 Uh, but I do want to point out it's more than the voice actor that recorded, and it sounds like that actor. (laughs) It is more about the choices of words, the attributes of, uh, tonality, the volume in terms of how you raise your pitch and so forth,-... all of that

Rohit Prasad 36:23 matters.

Lex Fridman 36:23 This is such a fascinating problem from a product perspective.

Rohit Prasad 36:27 Uh-huh.

Lex Fridman 36:27 I could see those debates just happening inside of the Alexa team-

Rohit Prasad 36:31 (laughs)

Lex Fridman 36:31 ... of how much personalization do you do for, for the specific customer, 'cause you're taking a risk if you over personalize, uh, because you don't... I, I if you create a personality for a million people, you can test that better, you can create a-

Rohit Prasad 36:47 Yeah.

Lex Fridman 36:47 ... rich, fulfilling experience that will do well. But if you... the more you personalize it, the less you can test it, the less you can know that it's a, it's a great-

Rohit Prasad 36:55 Mm-hmm.

Lex Fridman 36:55 ... experience. So how much personalization? What's the right balance?

Rohit Prasad 36:59 I think the right balance depends on the customer. Give them the control.

Lex Fridman 37:02 Yeah.

Rohit Prasad 37:02 So I'll say, uh, I think the, uh, more control you give customers, the better it is for everyone. And, uh, I'll give you some key personalization features.

Lex Fridman 37:13 Mm-hmm.

Rohit Prasad 37:13 I think we have a feature called Remember This, which is where you can tell Alexa to remember something. Uh, there, you have an explicit sort of control in customer's hand because they have to say, "Alexa, remember X, Y, Z."

Lex Fridman 37:26 What kind of things would that be used for?

Rohit Prasad 37:28 So you can like- (laughs)

Lex Fridman 37:28 For song title or something or...

Rohit Prasad 37:29 I, I have stored my tire specs for my car-

Lex Fridman 37:32 Nice.

Rohit Prasad 37:33 ... because it's so hard to go and find and see what it is, (laughs) right? When you're having some issues. So, uh, I, I store my mileage plan, uh, numbers for all the frequent flyer ones where I'm sometimes just looking at it and it's not handy.

Lex Fridman 37:46 Mm-hmm.

Rohit Prasad 37:46 Uh, so, and, so those are my own personal choices I've made for Alexa to remember something on my behalf, right? So again, I think the choice was be explicit about how you provide that to a customer as a control. So I think these are the aspects of, uh, what you do. Like think about where we can use speaker recognition capabilities that it's if you taught Alexa that you are Lex and this person in your household is Person 2-

Lex Fridman 38:16 Yeah.

Rohit Prasad 38:16 ... then you can personalize the experiences. Again, these are very in the C- uh, in the CX, customer experience, patterns are very clear about and transparent when a personalization action is happening. And then you have other ways like you go through explicit control right now through your app that, uh, you have multiple service providers, let's say for music. Which one is your preferred one? So when you say, "Play Sting," depend on your- whether you have preferred Spotify or Amazon Music or Apple Music that the decision is made where to play it from.

Lex Fridman 38:49 So what's Alexa's backstory from her perspective?

Rohit Prasad 38:52 (laughs)

Lex Fridman 38:53 Does... is there, um... I, I remember just asking, as probably a lot of us are, just the basic questions about love and so on-

Rohit Prasad 39:01 Yeah.

Lex Fridman 39:01 ... of Alexa just to see what the answer would be, just the... Uh, I... It feels like there's a little bit of a back... Like there's a, it feels like there's a little bit of personality, but not too much. Is, is Alexa have a metaphysical presence in this human universe we live in or is it something more ambiguous? Is there a past? Is there a birth? Is there a family kind of-

Rohit Prasad 39:28 (laughs)

Lex Fridman 39:28 ... idea even for joking purposes and so on?

Rohit Prasad 39:31 I think, uh, well, it does tell you if I think you... uh, I should double-check this, but if you said, "When were you born?"

Lex Fridman 39:37 When were you born, yeah.

Rohit Prasad 39:37 I think we do respond. I need to double-check that, but I'm pretty positive about it.

Lex Fridman 39:41 I think you, you do actually-

Rohit Prasad 39:42 Yes.

Lex Fridman 39:42 ... 'cause I think I've tested that.

Rohit Prasad 39:43 So, uh-

Lex Fridman 39:44 But that's like, uh, that's like how-

Rohit Prasad 39:46 (laughs)

Lex Fridman 39:46 Like I was, I was born in Urbana-Champaign and whatever the year-

Rohit Prasad 39:50 Yeah.

Lex Fridman 39:50 ... kind of thing, yeah.

Rohit Prasad 39:51 So in terms of the metaphysical, I think it's early. Uh, does it have the historic knowledge about herself to be able to do that? Maybe. Have we crossed that boundary? Not yet, right? In terms of being... thinking of... Uh, have we thought about it? Quite a bit, uh, but I wouldn't say that we have come to a clear decision in terms of what it should look like. But, uh, you can imagine though, and I bring this back to the Alexa Prize social bot one, there you will start seeing some of that. Like you-

Lex Fridman 40:21 Yeah.

Rohit Prasad 40:21 ... these bots have their identity and in terms of that you may find, uh, you know, this is such a great research topic that some academia team may think of these problems and start solving them too.

Lex Fridman 40:35 So let me ask, uh, a question. It's kinda difficult I think. Um, but it feels, and fascinating to me 'cause I'm fascinated with psychology. It feels that the more personality you have, the more dangerous it is in terms of a customer perspec- a product, if you wanna create a product that's useful. Da- by dangerous I mean creating an experience that upsets me. (laughs) And so, uh, uh, what... how do you, how do you get that right? Because, uh, if you look at the, the relationships, may- maybe I'm just a screwed up Russian-

Rohit Prasad 41:11 (laughs)

Lex Fridman 41:11 ... but, uh, if you look at the re- human to human relationship, some of our deepest relationships have fights, have tension, have the push and pull, have a little flavor in them. Uh, do you want to have such flavor in an, an interaction with Alexa? How do you think about that?

Rohit Prasad 41:29 So there's one other common thing that you didn't say but is, we think of it as paramount-

Lex Fridman 41:34 Mm-hmm.

Rohit Prasad 41:34 ... for any deep relationship. That's trust.

Lex Fridman 41:37 Trust, yeah.

Rohit Prasad 41:38 So I think if you trust every attribute you said-

Lex Fridman 41:41 Mm-hmm.

Rohit Prasad 41:42 ... a fight, some tension-

Lex Fridman 41:44 Yeah.

Rohit Prasad 41:44 ... is all healthy. But the, what is, uh, sort of unnegotiable in this instance is trust. And I think the bar to earn customer trust for AI is very high.

Lex Fridman 41:55 Yeah.

Rohit Prasad 41:56 In some sense more than a human.

Lex Fridman 41:58 Yes.

Rohit Prasad 41:58 It's, uh, it's not just about personal information or your data, it's also about your actions on a daily basis. How trustworthy are you in terms of consistency, in terms of, uh, how accurate are you in understanding me? Like if, if you're talking to a person on the phone, if you have a problem with your, let's say, your internet or something, if the person is not understanding, you lose trust right away. You don't want to talk to that person.

Lex Fridman 42:22 Yeah.

Rohit Prasad 42:22 S- that whole example gets amplified by a factor of 10 because as... when you're a human interacting with an AI-... you have a certain expectation. Either you expect it to be very intelligent, and then you get upset, why is it behaving this way-

Lex Fridman 42:35 Mm-hmm.

Rohit Prasad 42:35 ... or you expect it to be, uh, not so intelligent, and when it surprises you, you're like, "Really?" (laughs) "You were trying to be too smart?" So, I think we grapple with these hard questions as well, but I think the key is, actions need to be trustworthy from these AIs. Not just about data protection, your personal information protection, uh, but also from how accurately it accomplishes all commands or all interactions.

Lex Fridman 43:01 Well, it's tough to hear because trust, you're absolutely right, but trust is such a high bar with AI systems-

Rohit Prasad 43:07 Yep.

Lex Fridman 43:07 ... because people, and I see this 'cause I work with autonomous vehicles, I mean-

Rohit Prasad 43:10 (laughs)

Lex Fridman 43:10 ... the bar that's placed on AI system is unreasonably high.

Rohit Prasad 43:14 Yeah, that is going to be a s- uh, I agree with you. And, uh, I think of it as, it's, uh-

Lex Fridman 43:20 A challenge? (laughs)

Rohit Prasad 43:20 It's a challenge, and it's also keeps my job. (laughs)

Lex Fridman 43:23 (laughs)

Rohit Prasad 43:24 Right? So, so from that perspective, the- I totally, I think of it at both sides, as a customer and as a researcher. I think as a researcher, yes, occasionally it will frustrate me that, why is the bar so high for these AIs? And as a customer, then I say, "Absolutely it has to be that high."

Lex Fridman 43:41 Yeah.

Rohit Prasad 43:41 Right? So I think that's the trade-off we have to balance, uh, but doesn't change the fundamentals, that trust has to be earned, and the question then becomes, is, are we holding the AIs to a different bar in accuracy and mistakes than we hold humans? That's gonna be a great societal questions for years to come,

Rohit Prasad 44:00 I think, for us.

Lex Fridman 44:01 Well, one of the questions that we grapple as a society now, that I think about a lot, that I think a lot of people in the AI think about a lot, and Alexa's taking on head on, is privacy.

Rohit Prasad 44:11 Mm-hmm.

Lex Fridman 44:12 Is, th- the reality is, us giving over data to any AI system can be used to enrich our lives in h- in, in, in profound ways.

Rohit Prasad 44:25 Mm-hmm.

Lex Fridman 44:25 So if it, may, basically any product that does anything awesome for you, would, m- the more data it has, the more awesome things it can do. And yet, uh, on the other side, people imagine the worst case possible scenario of what can you possibly do with that data. People h- it's, it go- boils down to trust, as you said before.

Rohit Prasad 44:45 Yeah.

Lex Fridman 44:45 There's a fundamental distrust of, uh, in certain groups of governments and so on. Depending on the government, depending on who's in power, depending on all these kinds of factors. And so here's Alexa in the middle of all of it-

Rohit Prasad 44:59 (laughs)

Lex Fridman 44:59 ... uh, in the home, trying to do good things for the customers, so how do you think about privacy in this context, the smart assistants in the home? How do you maintain, how do you earn trust?

Rohit Prasad 45:08 Absolutely. So, as you said, trust is the key here.

Lex Fridman 45:12 Yeah.

Rohit Prasad 45:12 So you start with trust, and then privacy is a key aspect of it. It's, has to be designed from very beginning about that. And we believe in two, uh, fundamental principles. One is transparency, and second is control. So if you, uh, by transparency I mean, uh, when we built, uh, what is now called Smart Speaker, or the first Echo, we were quite judicious about making these right trade-offs on customers' behalf that it is pretty clear when, when the audio's being sent to cloud, the light ring comes on when it has heard you say the word, wake word, and then the streaming happens, right? So and the light ring comes up. We also had, we put a physical mute button on it, just so you're, if you didn't want it to, uh, be listening even for the wake word, then you turn the, uh, power button, uh, the mute button on, and that, uh, disables the microphones. That's just the first decision on essentially transparency and control. O- then, even when we launched, we gave the control in the hands of the customers that you can go and look at any of your individual utterances that is recorded and delete them anytime, and, uh, we have cut to, true to that promise, right? So and that is super, again, a great instance of showing how you have the control. Then we made it even easier. You can say, "Alexa, delete what I said today." So that is now making it even just, (laughs) just more control in your hands-

Lex Fridman 46:37 Mm-hmm.

Rohit Prasad 46:37 ... but what's most convenient about this technology is voice. You delete it with your voice now.

Lex Fridman 46:43 Yeah.

Rohit Prasad 46:44 Uh, so these are the types of decisions we continually make. Uh, we just recently launched this feature called, uh, what we think of it as if you wanted humans not to review your tr- uh, data, uh, because sm- you've mentioned supervised learning, right?

Lex Fridman 46:59 Yeah.

Rohit Prasad 46:59 So you, in supervised learning, humans, uh, have to give some annotation. And that also is now a feature where you can, uh, essentially, if you've selected that flag, your data will not be reviewed by a human. So, these are the types of controls that we have to constantly offer with customers.

Lex Fridman 47:18 So, why do you think it bothers people so much that, so th- uh, so everything you just said, uh, is really powerful, so the control, the ability to delete, 'cause we collect, we have studies here running at MIT that collects huge amounts of data-

Rohit Prasad 47:32 Mm-hmm.

Lex Fridman 47:32 ... and people consent and so on. Uh, the b- ability to delete that data is really empowering, and m- almost nobody ever asks to delete it, but the ability to have that control is, is really powerful. But still,

Lex Fridman 47:45 you know, there's these popular anecdote, anecdotal evidence that people say, they like to tell that, uh, them and a friend were talking about something, I don't know, uh, sweaters for cats, and all of a sudden they'll have advertisements for cat sweaters-

Rohit Prasad 47:59 Mm-hmm.

Lex Fridman 47:59 ... on Amazon. There's that, that's a popular anecdote, as if something is always listening. What, can you explain that anecdote, that experience that people have? What's the psychology of that?

Rohit Prasad 48:10 Mm-hmm.

Lex Fridman 48:11 What's that experience? Uh, and can you ... You've answered it, but let me just ask, is Alexa listening? (laughs)

Rohit Prasad 48:18 No, Alexa listens only for the wake word on the device, right? Uh-

Lex Fridman 48:22 And a wake word is ...

Rohit Prasad 48:24 The words like, "Alexa," "Amazon," "Echo," and you, uh, but you only choose one at a time.

Lex Fridman 48:29 Yeah.

Rohit Prasad 48:29 So you choose one and it listens only for that on our devices.

Lex Fridman 48:33 Okay.

Rohit Prasad 48:34 So that's first. From a listening perspective, we have to be very clear that it's just a wake word. So you said, "Why is there, uh, this anxiety?" If you may.

Lex Fridman 48:41 Yeah, exactly.

Rohit Prasad 48:41 It's because there's lot of confusion what (laughs) it really listens to, right? And you, uh, and I think it's partly on us to keep educating, uh, our, our customers and the general media more in terms of like how, what really happens, and we have done, uh, a lot of it. Uh, and w- uh, our pages on information are clear, but still people have to have, uh, more... There's always a hunger for information (laughs) -

Lex Fridman 49:06 Yeah.

Rohit Prasad 49:06 ... and clarity. And we'll constantly look at how best to communicate. If you go back and read everything, yes, it states exactly that, uh, and then people could still question it. And I think that's absolutely okay to question. Uh, what we have to make sure is that we are, uh, because our fundamental philosophy is customer first, customer obsession is our leadership principle, if you put... As researchers, I put myself in the f- uh, shoes of the customer, and all decisions in Amazon are made with that in line, so. And trust has to be earned and we have to keep earning (laughs) the trust of our customers in this setting. Uh, and to your other point on like, uh, is there something showing up based on your conversations? No.

Lex Fridman 49:47 Mm-hmm.

Rohit Prasad 49:47 I think the answer is like you, uh, a lot of times when those experiences happen, you have to also be, know that, okay, it may be a winter season. (laughs) People are looking for sweaters, right?

Lex Fridman 49:56 Yeah.

Rohit Prasad 49:56 And it shows up on your amazon.com because it is popular. So-

Lex Fridman 50:00 Okay.

Rohit Prasad 50:00 ... there are many of these, uh, uh, you mentioned that, uh, personality or personalization. Turns out we are not that unique either. (laughs)

Lex Fridman 50:08 Yeah.

Rohit Prasad 50:08 Right? So those things we, we as humans start thinking, "Oh, must be because something was heard and that's why this other thing showed up," the answer is no. Probably it is just the season (laughs) for sweaters.

Lex Fridman 50:21 I'm not gonna ask you this question 'cause it's just, 'cause you're also, 'cause people have so much paranoia. But from my, let me just say from my perspective, I hope there's a day when customer can ask Alexa to listen all the time, uh, to improve the experience, to improve... Because I, I personally don't see the negative of, because if you have the control and if you have the trust, there's no reason why you shouldn't be listening all the time to the conversations to learn more about you. Because ultimately, as long as you have control and trust, every data you provide to the device that the device wants (laughs) is going to be useful.

Rohit Prasad 51:01 Mm-hmm.

Lex Fridman 51:01 And that, so that to me, I, as, as a machine learning person, I think it worries me how sensitive people are about their data relative to how, uh, empowering it could be for the devices around them, how m- enriching it could be for their own life to ex- improve the product. So I just, (laughs) it's something I think about sort of a lot, how do we make that in devices. Obviously Alexa thinks about it a, a lot as well. I don't know if you wanna comment on that-

Rohit Prasad 51:34 (laughs)

Lex Fridman 51:34 ... sort of. Okay, have you seen, let me ask it-

Rohit Prasad 51:36 So you, do-

Lex Fridman 51:36 ... in the form of a question. Okay.

Rohit Prasad 51:37 Yeah.

Lex Fridman 51:37 Have, have you seen an evolution in the way people think about their private data in the previous several years? So as we as a society get more and more comfortable to the benefits we get by sharing more data?

Rohit Prasad 51:53 First, let me answer that part and then I wanna go back-

Lex Fridman 51:56 Sure. Sure.

Rohit Prasad 51:56 ... to the other aspect you were mentioning. Um, so as a society, on a general, we are getting more comfortable as a society. Doesn't mean that everyone is. And I think we have to respect that. Uh, I don't think, uh, one size fits all is always gonna be the answer for all (laughs) , right? Uh, by definition. So I think that's something to keep in mind in these. Uh, going back to your, on what more, uh, magical experiences can be launched in these kind of AI settings. Uh, I think, again, if you give the control, we, uh, it's possible, certain parts of it. So we have a feature called follow-up mode where you, uh, if you turn it on, and Alexa, after you've spoken to it-

Lex Fridman 52:40 Mm-hmm.

Rohit Prasad 52:40 ... will open the mics again thinking you'll ask something again.

Lex Fridman 52:44 Yeah.

Rohit Prasad 52:44 Like if you're adding sh- uh, lists to your shopping-

Lex Fridman 52:47 Yeah.

Rohit Prasad 52:47 ... items, right? Or a shopping list or to-do list, you're not done. (laughs)

Lex Fridman 52:51 Right.

Rohit Prasad 52:51 You want to keep... So in that setting, it's awesome that it opens the mic for you to say, "Eggs and milk and then bread." Right?

Lex Fridman 52:57 Mm-hmm.

Rohit Prasad 52:57 So these are the kind of things which you can empower. So I, and then another feature we have, which is called Alexa Guard, I said it only listens for the wake word, all right?

Lex Fridman 53:07 Mm-hmm.

Rohit Prasad 53:07 But if you have a, let's say, you're going to say, "Alexa," uh, you leave your home and you want Alexa to listen for couple of sound events-

Lex Fridman 53:15 Mm-hmm.

Rohit Prasad 53:15 ... like smoke alarm going off or someone breaking your glass, right? So it's like just to keep, uh, your peace of mind. So you can say, "Alexa, on guard," or, uh, uh, or, "I'm away," or, uh, and then it can, uh, be listening for these sound events. And when you're home, it, you come out of that mode, right?

Lex Fridman 53:32 Mm-hmm.

Rohit Prasad 53:33 So this is another one where you again gave controls in the hands of the user or the customer, uh, to enable some experience that is u- high utility and maybe even more delightful in the certain settings like follow-up mode and so forth. And again, this general principle is the same, control in the hands of the customer.

Lex Fridman 53:52 (laughs) So I know we, we kind of started with a lot of philosophy and a lot of interesting topics-

Rohit Prasad 53:56 Yeah.

Lex Fridman 53:56 ... and we're just jumping all over the place, but really some of the fascinating things that, uh, the Alexa team and Amazon is doing is in the, the algorithm side, the data side, the technology, the deep learning and machine learning and, and so on. So can you give a brief history of Alexa-

Rohit Prasad 54:12 Uh-huh.

Lex Fridman 54:13 ... from the perspective of just innovation, the algorithms, the data of-

Rohit Prasad 54:16 Mm-hmm.

Lex Fridman 54:16 ... how, how it was born, how it came to be, how it's grown-

Rohit Prasad 54:20 Yeah.

Lex Fridman 54:20 ... where it is today?

Rohit Prasad 54:22 Yeah. It start with the... In Amazon, everything starts with the customer, and we have a process called working backwards. Alexa, and more specifically than the product Echo, there was a working backwards document essentially that reflected what it would be. Started with a very simple...... a vision statement, for instance, (laughs) that, uh, morphed into a full-fledged document along the way changed into what all it can do, uh, right?

Rohit Prasad 54:51 You can, uh... But the inspiration was the Star Trek computer. So when you think of it that way, you know, e- everything is possible, but when you launch a product, you have to start with some place. And when I joined, we... the product was already in conception, and we started working on the far-field speech recognition because that was the first thing to solve. By that, we mean that you should be able to speak to the, uh, device from a distance, and in those days, that wasn't a common practice. Uh, and even in the previous research world I was in, was considered to, uh, unsolvable problem then-

Lex Fridman 55:24 Mm-hmm.

Rohit Prasad 55:24 ... in terms of whether you can converse from a length.

Lex Fridman 55:28 Yeah.

Rohit Prasad 55:28 And here, I'm still talking about the first part of the problem where you say... get the attention of the device, as in by saying what we call the wake word, uh, which means the word Alexa has to be detected with a very high accuracy because it is a very common word. It has sound units that map with words like I like you or Alek, Alex. (laughs)

Lex Fridman 55:50 Yeah.

Rohit Prasad 55:51 Right? So it's a, uh, undoubtably hard problem to detect the right mentions of Alexa's address to the device versus I like Alexa.

Lex Fridman 56:02 So you have to pick up that signal when there's a lot of noise and all kinds of background-

Rohit Prasad 56:06 Not only noise, but a lot of conversation-

Lex Fridman 56:08 Yeah.

Rohit Prasad 56:08 ... are in the house, right?

Lex Fridman 56:09 Oh-

Rohit Prasad 56:09 You remember on the device, you're s- simply listening for the wake word, Alexa. And there's a lot of words being spoken in the house. How do you know it's Alexa and directed at Alexa? Because I could say, "I love my Alexa." "I hate my Alexa." Uh, "I want Alexa to do this." And in all these three sentences, I said, "Alexa," I didn't want it to wake up.

Lex Fridman 56:30 Yeah.

Rohit Prasad 56:31 Uh, so, um-

Lex Fridman 56:32 Can I just pause on that second?

Rohit Prasad 56:33 Yeah.

Lex Fridman 56:33 What would be your device that I should probably in the introduction of this conversation give to people in terms of with them turning off their Alexa device if they're listening to this podcast conversation out loud? Like w- what's the probability that an Alexa device will go off because we mentioned Alexa like a million times?

Rohit Prasad 56:54 (laughs) So it will... Uh, we have done lot of different things where we can, uh, figure out that there is the device, the speech is coming from a human versus, uh, over the air. Also, I mean, in terms of like, also it is... Think about ads or, uh, so we have also-

Lex Fridman 57:13 Ads.

Rohit Prasad 57:13 ... launched a technology for watermarking kind of approaches in terms of-

Lex Fridman 57:17 Mm-hmm.

Rohit Prasad 57:17 ... filtering it out. But, yes, if this kind of a podcast is happening, it's possible your device will wake up a few times.

Lex Fridman 57:23 Okay.

Rohit Prasad 57:24 Right? It's an unsolved problem-

Lex Fridman 57:25 (laughs)

Rohit Prasad 57:25 ... but it is, uh, definitely, uh, uh, something we care very much about, right?

Lex Fridman 57:31 But the idea is you wanna detect Alexa first of all.

Rohit Prasad 57:34 Meant for the device.

Lex Fridman 57:35 Meant for the device. I mean, f- first of all, just even hearing Alexa versus I like...

Rohit Prasad 57:38 Yeah.

Lex Fridman 57:39 ... something, I mean, that's a fascinating part. So that was the first really f-

Rohit Prasad 57:43 That's the first part. That-

Lex Fridman 57:43 ... build the world's best detector of Alexa.

Rohit Prasad 57:46 First, yeah, the fir- world's best wake word detector-

Lex Fridman 57:48 Yeah.

Rohit Prasad 57:48 ... or in a far-field setting, not like something where the phone is sitting on the table.

Lex Fridman 57:52 Yeah.

Rohit Prasad 57:53 This is like people have devices 40 feet away, like in my house, or 20 feet away, and you still get an answer.

Lex Fridman 58:00 Yeah.

Rohit Prasad 58:00 Uh, so that was the first part. The next is, okay, you're speaking to the device, of course, you're gonna issue many different requests. Some may be simple, some may be extremely hard, but it's a large vocabulary speech recognition problem essentially, where the audio is now not coming, uh, onto your phone or a handheld mic like this, or a close talking mic, but it's from 20 feet away, where if you're in a busy household, your son may be l- uh, listening to music, your daughter may be running around (laughs) with something and asking your mom something and so forth, right? So this is like a common household setting where the words you're speaking to Alexa-

Lex Fridman 58:39 Mm-hmm.

Rohit Prasad 58:40 ... need to be recognized with very high accuracy.

Lex Fridman 58:42 Yes.

Rohit Prasad 58:42 Right? Now, we are still just in the recognition problem. We haven't yet come to the understanding one, right?

Lex Fridman 58:48 And, and if we pause, I'm sorry, once again, wh- what year was this? Is this before neural networks began to s- start, uh, to seriously prove themselves in the audio space?

Rohit Prasad 59:00 Yeah, this is around... So, uh, I joined in 2013.

Lex Fridman 59:04 13.

Rohit Prasad 59:04 In April, right? So the early research in neural networks coming back and showing, uh, some promising results in speech recognition space had started happening, but it was very early.

Lex Fridman 59:14 Very. Yeah.

Rohit Prasad 59:15 Uh, but we just to now build on that on the very first thing we did when, uh, when I joined and we, uh, with the team, uh, and remember, it was a very sm- much of a startup environment, which is great about Amazon. And we doubled down on deep n- deep learning right away. And, uh, we, we knew we'll have to improve accuracy fast. And because of that, we worked on... And the scale of data, once you have a device like this, if it is successful, will improve big time. Like, you'll suddenly have large volumes of data to learn from to make the customer experience better. So how do you scale deep learning? So we did our f- uh, one of the first works in, in training with distributed GPUs and where the training time, uh, was, uh, you know, was linear, uh, in terms of like in the amount of data. So that was quite important work, uh, where it was algorithmic improvements as well as a lot of engineering improvements to be able to train on thousands and thousand o- of speech. Uh, and that was an important factor. So the, uh... if you ask me, like, in back in 2013 and 2014 when we, uh, launched Echo, uh, the combination of large-scale data, uh, deep learning progress, near infinite GPU s- we had available on AWS even then, was all came together for us to be able to solve the far-field speech recognition to the extent it could be useful to the customers. It's still not solved, like, I mean, it's not that we are perfect at recognizing speech, but we are great at it in terms of the settings that are, uh, in homes, right? So, and that was important even in the early stages.

Lex Fridman 1:00:51 So first of all, just even, uh, I'm trying to look back at that time, uh...If I remember correctly, uh, the, it was, it seems like the task would be pretty daunting.

Rohit Prasad 1:01:01 (laughs)

Lex Fridman 1:01:02 So like, uh, so we kinda take it for granted that it works now.

Rohit Prasad 1:01:06 Yes. So, you're right.

Lex Fridman 1:01:07 Uh, so let me, like how... First of all, you mentioned startup. I wasn't familiar how big the team was. I kinda... 'cause I know there's a lotta really smart people working on it.

Rohit Prasad 1:01:15 Yeah.

Lex Fridman 1:01:16 So now it's very, very large team. (laughs) How big was the team? How likely were you to fail in the eyes of everyone else? (laughs)

Rohit Prasad 1:01:24 (laughs) And ourselves. (laughs)

Lex Fridman 1:01:26 And, and, and yourself. So like what-

Rohit Prasad 1:01:28 I'll give you a very interesting-

Lex Fridman 1:01:29 Yeah.

Rohit Prasad 1:01:29 ... anecdote on that. When I joined the team, esp- the speech recognition team was six people. My first meeting, and we had hi- hired a few more people, it was 10 people, nine out of 10 people thought it can't be done.

Lex Fridman 1:01:47 (laughs)

Rohit Prasad 1:01:47 Right?

Lex Fridman 1:01:48 Who was the one? (laughs)

Rohit Prasad 1:01:49 (laughs) The one was me saying-

Lex Fridman 1:01:51 Okay. (laughs)

Rohit Prasad 1:01:51 And actually I should say, and one was semi-optimistic.

Lex Fridman 1:01:55 Yeah. (laughs) In the face.

Rohit Prasad 1:01:57 Uh, and the, uh, and eight were trying to convince, "Let's go to the management and say, 'Let's not work on this problem. Let's work on some other problem like, uh,"-

Lex Fridman 1:02:06 Mm-hmm.

Rohit Prasad 1:02:06 "... either telephony speech for customer service calls and so forth." But this is the kind of belief you must have, and I had experience with far-field speech recognition, and I, my eyes lit up when I saw a problem like that, saying, "Okay, we have been in speech recognition, always looking for that killer app."

Lex Fridman 1:02:23 Yeah.

Rohit Prasad 1:02:23 And this was a killer use case to bring something delightful in the hands of customers.

Lex Fridman 1:02:28 So you mentioned, uh, you, the way you kinda think of it in the product way in the future, have a press release and an FAQ, and you think backwards.

Rohit Prasad 1:02:35 That's right.

Lex Fridman 1:02:35 Did you have, did the team have the Echo in mind? Th- so this far-field-

Rohit Prasad 1:02:42 Yeah.

Lex Fridman 1:02:42 ... speech recognition, could actually putting a thing in the home that works, that's able to interact with, was that the press release? What was the...

Rohit Prasad 1:02:48 Very, very close I would say in terms of the, uh, as I said, the vision was Star Trek computer, right? So-

Lex Fridman 1:02:55 Yeah.

Rohit Prasad 1:02:55 (laughs) Or the inspiration. Uh, and from there-

Lex Fridman 1:02:57 (laughs) Whoa.

Rohit Prasad 1:02:58 ... uh, I can't divulge all the exact specifications-

Lex Fridman 1:03:00 Sure.

Rohit Prasad 1:03:01 ... but, uh, one of the first things that was magical on Alexa was music. It brought me to, back to music because my taste is still in when I was an undergrad.

Lex Fridman 1:03:13 Right.

Rohit Prasad 1:03:14 Uh, so I still listen to those songs, and I, it was too hard for me to be a music fan with a phone, right? So I, and I don't, and I hate things in my ear. (laughs) So from that perspective, it was, uh, quite, uh, hard. And, and, and music was part of the, uh, uh, at least the documents I've seen, right? So, so from that perspective, I think yes, in terms of, uh, our, uh, "How far are we from the original vision?" Uh, I can't reveal that, but it's-

Lex Fridman 1:03:42 (laughs)

Rohit Prasad 1:03:42 That's why I have ton of fun at work because every day we go in, and thinking like, "These are the new set of challenges to solve."

Lex Fridman 1:03:49 Yeah, it's a great way to do, uh, great engineering is you think of the pro- the press release. I like that idea actually.

Rohit Prasad 1:03:54 Yeah.

Lex Fridman 1:03:55 Maybe we'll talk about it a b- a bit later, but it's just a super nice way to have focus on what you're doing.

Rohit Prasad 1:03:59 I'll tell you this. You're a scientist, uh, and a lot of my scientists have adopted that.

Lex Fridman 1:04:03 Yeah.

Rohit Prasad 1:04:03 They, they have now, they love it-

Lex Fridman 1:04:06 Yeah.

Rohit Prasad 1:04:06 ... as a process because it was very, uh, as scientists we are trained to write great papers, but they're all after you've done the research or you've proven like, and your PhD dissertation proposal is something that comes closest, or a DARPA proposal or a NSF proposal is the closest that it comes to a press release. But that process is now ingrained in our scientists, which is like, uh, delightful for me to see.

Lex Fridman 1:04:29 (laughs) You write the paper first, then make it happen. (laughs)

Rohit Prasad 1:04:33 That's right. I mean in fact it's not-

Lex Fridman 1:04:34 State-of-the-art results. (laughs)

Rohit Prasad 1:04:36 Yeah, or you leave the results section open-

Lex Fridman 1:04:38 Yeah.

Rohit Prasad 1:04:38 ... where you have a thesis about here's what I expect, right?

Lex Fridman 1:04:41 Yeah.

Rohit Prasad 1:04:41 And here's what it will change.

Lex Fridman 1:04:44 Yeah.

Rohit Prasad 1:04:44 Right? So I think it is a great thing. It works for researchers as well.

Lex Fridman 1:04:48 Yeah. The, so far-field recognition-

Rohit Prasad 1:04:50 Yeah.

Lex Fridman 1:04:51 ... uh, wha- what was the big leap? What, what, what were the breakthroughs and, um, yeah, what was that journey like to today?

Rohit Prasad 1:04:58 Yeah, I think the, as you said, first there was a lot of skepticism on whether-

Lex Fridman 1:05:02 Yes.

Rohit Prasad 1:05:02 ... far-field speech recognition will ever work, uh, to be good enough, right? Uh, and what we first did was got a lot of training data in a far-field setting, and that was extremely hard to get because none of it existed. So how do you collect data in far-field setup, right?

Lex Fridman 1:05:20 With no customer base at the start.

Rohit Prasad 1:05:21 With no customer base, right?

Lex Fridman 1:05:22 Yeah.

Rohit Prasad 1:05:22 So that was first innovation. And once we had that, the next thing was, okay you ha- if, if you have the data, first of all we didn't talk about like what would magical mean in this kind of a setting? What is good enough for customers, right? That's always a-

Lex Fridman 1:05:38 Mm-hmm.

Rohit Prasad 1:05:38 Since you've never done this before, what would be magical? So y- uh, so it wasn't just a research problem. You had to put some, in terms of accuracy and customer experience features, some stakes on the ground saying, "Here's where I think should, it should get to." Uh, so you established a bar, and then how do you measure progress towards, given you have no customers right now?

Lex Fridman 1:06:01 Yeah.

Rohit Prasad 1:06:01 So from that perspective, we went, uh, so first was the data without customers. Second was doubling down on deep learning as a way to learn.

Lex Fridman 1:06:11 Mm-hmm.

Rohit Prasad 1:06:12 And I can just tell you that the combination of the two cut our error rates by a factor of five. From where we were when I started to, uh, within six months of having that data, we, at that point I, uh, I got the conviction that this would work, right? So because that was magical in terms of when it started working, and, uh-

Lex Fridman 1:06:34 That reached the magical-

Rohit Prasad 1:06:35 That's the-

Lex Fridman 1:06:36 Or came close to the magical bar.

Rohit Prasad 1:06:37 Not to the bar, right? That we felt would be where people will use it. That was critical because you, you, you really have one chance at this. If we had launched in November 2014, is when we launched, and if it was below the bar, I don't think this category exists if you don't meet the bar.

Lex Fridman 1:06:58 Yeah, and I, just having, uh, looked at voice-based interactions like in the car, uh, our earlier systems-... it's a source of huge frustration for people. In fact, we use voice-based interaction for collecting data on subjects to measure frustration.

Rohit Prasad 1:07:14 Frustration, yeah.

Lex Fridman 1:07:14 So as a training set for computer vision, for face data, to, to... so we can get a data set of frustrated people.

Rohit Prasad 1:07:20 That's right.

Lex Fridman 1:07:20 That's the best way to get frustrated people is having them interact with a voice-based system in the car. So the... so that bar, I imagine is pretty high. There-

Rohit Prasad 1:07:28 Oh, it's very high, and we talked about how also errors are perceived from AIs versus errors by humans. But we are not done with the problems that ended up... uh, we had to solve to get it to launch. So do you want the next one? (laughs)

Lex Fridman 1:07:42 Yeah, what's the next one? (laughs)

Rohit Prasad 1:07:45 So the next one was what I think of as multi-domain natural language understanding. It's very... I wouldn't say easy, but it is... uh, during those days, er, f- uh, solving it, understanding in one domain, a narrow domain, was doable. Uh, but for these multiple domains, like music, like information, other kinds of household productivity, alarms, timers, even though it wasn't as big as it is in terms of the number of skills Alexa has and the confusion space has, like, grown by three orders of magnitude-

Lex Fridman 1:08:21 Mm-hmm.

Rohit Prasad 1:08:22 ... it was still daunting even those days. And-

Lex Fridman 1:08:24 Again, no customer base yet.

Rohit Prasad 1:08:26 Again, no customer base. So now you're looking at meaning understanding and intent understanding and taking actions on behalf of customers based on their requests, and that is the next hard problem. Even if you have gotten the words recognized, how do you make sense of them? In those days, uh, there was still a lot of emphasis on rule-based systems for writing grammar patterns to understand the intent, but we had a statistical first approach even then, where for language understanding we had, uh, in even the starting days of an, an entity recognizer and an intent classifier, which was all trained statistically. Uh, in fact, uh, we had to build the deterministic matching as a follow-up to fix-

Lex Fridman 1:09:13 (laughs)

Rohit Prasad 1:09:13 ... bugs that statistical models have, right?

Lex Fridman 1:09:16 Right.

Rohit Prasad 1:09:16 So it was just a different mindset where we focused-

Lex Fridman 1:09:19 Yeah.

Rohit Prasad 1:09:19 ... on data-driven statistical understanding.

Lex Fridman 1:09:22 Wins in the end if you have a huge data set.

Rohit Prasad 1:09:24 Yes, it is contingent on that, and that's why it came back to how do you get the data? Before customers, the fact that this is why data becomes crucial to get it to the point that you have, uh, the understanding system built in, built up. And notice that for hu-... we were talking about human-machine dialogue, and even those early days, uh, even it was very much transactional, do one thing, one-shot utterances in great way, there was a lot of debate on how much should Alexa talk back in terms of-

Lex Fridman 1:09:55 Mm-hmm.

Rohit Prasad 1:09:55 ... if it misunderstood you, or you said, "Play, uh, songs by The Stones," and let's say it doesn't know, uh... you know, early days, knowledge can be sparse. Uh, who, who are The Stones, right? I, uh... (laughs) It's The Rolling Stones, right?

Lex Fridman 1:10:10 Yeah.

Rohit Prasad 1:10:10 So, uh, our, uh, uh... and you don't want the match to be Stone Temple Pilots or Rolling Stones, right?

Lex Fridman 1:10:17 Stone Temple Pilots, yeah. Yeah.

Rohit Prasad 1:10:17 So you don't know which one it is. So these kind of other signals to, uh... and now there we had great assets, right, from Amazon in terms of, uh-

Lex Fridman 1:10:27 UX. Like, what, what is it... what kind of... yeah, how do you solve that problem? Uh, is it-

Rohit Prasad 1:10:31 In terms of what we think of it as an entity resolution problem, right? So because which one is it, right?

Lex Fridman 1:10:36 Oh, I j-

Rohit Prasad 1:10:36 I mean, the, uh... even if you figured out The Stones is an entity-

Lex Fridman 1:10:40 Yeah.

Rohit Prasad 1:10:40 ... you have to resolve it to whether it's The Stones or the Stone Temple Pilots-

Lex Fridman 1:10:43 So, so-

Rohit Prasad 1:10:44 ... or some other stones.

Lex Fridman 1:10:44 ... maybe I misunderstood. Is the resolution, uh, the job of the algorithm, or is the job of UX communicating with the human to help the resolution?

Rohit Prasad 1:10:52 Well, there is, uh, both, right? It is lot... you want 90% or high 90s to be done without any further questioning or UX, right? So, uh, uh... but that it's absolutely okay, just like as humans, we ask the question, "I, I didn't understand you, Alex."

Lex Fridman 1:11:08 Yeah.

Rohit Prasad 1:11:08 Uh, it's fine for Alexa to occasionally say, "I did not understand you," all right? And, uh, and that's a important way to learn, and I'll talk about where we have come with more self-learning with these kind of feedback signals. But in those days, just solving the ability of understanding the intent and resolving to an action, where action could be play a particular artist or a particular song, was super hard. Again, the, the bar was high, as you, as we are talking about, right? So while we launched it in, uh, sort of 13 big domains, I would say, in terms of our, uh, thing... we think of it as 13 big skills we had, like music is a massive one, uh, when we launched it, and now we have 90,000 plus skills on Alexa.

Rohit Prasad 1:11:51 So-

Lex Fridman 1:11:51 So what, what are the big skills? Can you just go over them? 'Cause the only thing I use it for is music, weather, and shopping.

Rohit Prasad 1:11:58 (laughs) So, uh, we think of it as music information, uh, right? So it sh- uh, weather is a part of information, right?

Lex Fridman 1:12:05 Weather is definitely... right.

Rohit Prasad 1:12:05 So, uh, when we launched, we didn't have smart home, but within s-... uh, by smart home I mean you connect your smart devices, you control them with voice. If you haven't done it, it's worth... it will change your life.

Lex Fridman 1:12:15 Like turning on the lights and so on? Yeah.

Rohit Prasad 1:12:16 Yeah, turning on your light to, uh, anything that's connected, uh, and has a... it's just that-

Lex Fridman 1:12:21 What's your favorite smart device for you that you-

Rohit Prasad 1:12:23 My light. (laughs)

Lex Fridman 1:12:24 Light?

Rohit Prasad 1:12:24 And now you have the smart plug with... and you don't... uh, we also have this Echo Plug, uh, which is-

Lex Fridman 1:12:30 Oh yeah, you can plug in anything-

Rohit Prasad 1:12:31 ... so you can plug in anything, and now you can turn on... that one on and off, right? So-

Lex Fridman 1:12:33 I'll use this conversation motivation and get one at some point.

Rohit Prasad 1:12:35 The garage door, you can check, uh, your status of the garage door and things like... and we have gone make Alexa more and more proactive, where it even have a hunch- has hunches now that, uh... or looks-

Lex Fridman 1:12:47 Hunches?

Rohit Prasad 1:12:47 Hunches like you left your light on. Uh, uh, let's say you've gone to your bed and you left the garage light on. So, uh, so it'll help-

Lex Fridman 1:12:54 It will help you out.

Rohit Prasad 1:12:55 ... it will help you out in these settings, right? So-

Lex Fridman 1:12:56 That's smart devices.

Rohit Prasad 1:12:58 Right.

Lex Fridman 1:12:58 Information, smart devices, you said music.

Rohit Prasad 1:13:01 Yeah, so I don't remember everything we had, but alarms, timers were the big ones.

Lex Fridman 1:13:05 Yeah, those were the big ones. Yeah, yeah, to have.

Rohit Prasad 1:13:05 Like, that was...... you know, the timers were very popular fr- right away. Uh, music also, like, you could play song, artist, album, everything, uh, and so that was, like, a, a clear win in terms-

Lex Fridman 1:13:17 Yeah.

Rohit Prasad 1:13:17 ... of the customer experience.

Rohit Prasad 1:13:19 So, that's ... again, this is language understanding. Now things have evolved, right? So, where we want Alexa definitely to be more accurate, competent, trustworthy, based on how well it does these core things, but we have evolved in many different dimensions. First is what I think of her doing more conversational for high utility, not just for chat. Right? And there we, uh, at re:MARS this year, which is our AI conference, we launched what is called Alexa Conversations. Uh, that is providing, uh, the ability for developers to author multi-turn experiences on Alexa with no code, essentially, where, uh, in terms of the dialogue code. Initially it was like, uh, you know, all these IVR systems, uh, you have to fully author if the customer says this, do that, right? So, the whole dialogue flow is hand-authored.

Lex Fridman 1:14:11 Mm-hmm.

Rohit Prasad 1:14:11 And with Alexa Conversations, the way it is that you just provide a sample interaction data with your service order API, let's say your ATOM tickets that, uh, provides a service for buying movie tickets.

Lex Fridman 1:14:22 Mm-hmm.

Rohit Prasad 1:14:23 You provide a few examples of how your customers will interact with your APIs, and then the dialogue flow is automatically constructed using a recurrent neural network, uh, trained on that data. So, that simplifies the developer experience. We just launched our preview for the developers to try this capability out. And then the second part of it, which shows i- even increased utility for customers is you and I, when we interact with Alexa or any customer, as I'm coming back to our initial part of the conversation, the goal is often unclear, uh, or unknown to the AI. If I say, "Alexa, what movies are playing nearby?" Am I trying to just buy movie tickets? Am I actually even ... do you think I'm looking for just movies for curiosity, whether the-

Lex Fridman 1:15:13 Yeah.

Rohit Prasad 1:15:13 ... Avengers is still in theater or when is (laughs) maybe it's gone, and maybe it will come on my ... missed it, so I may watch it on Prime? (laughs)

Lex Fridman 1:15:20 Yeah. Right.

Rohit Prasad 1:15:21 Which happened to me.

Lex Fridman 1:15:22 Yeah. (laughs)

Rohit Prasad 1:15:22 Uh, so, uh, so from that perspective now, you're looking into, what is my goal? And let's say I now complete the movie ticket purchase. Maybe I would like to get dinner nearby.

Lex Fridman 1:15:35 Mm-hmm.

Rohit Prasad 1:15:36 Uh, so what is really the goal here? Is it night out or is it movies, as in just go watch a movie?

Lex Fridman 1:15:45 Yeah.

Rohit Prasad 1:15:45 The answer is we don't know. So, can Alexa now figuratively have the, uh, intelligence that I think this meta goal is really night out, or at least say to the customer, "When you've completed the purchase of movie tickets from ATOM Tickets or Fandango or Pick Your or any one, then the next thing is do you want to, uh, get to, uh, get a Uber to the theater?" Right?

Lex Fridman 1:16:10 Mm-hmm.

Rohit Prasad 1:16:10 Or, uh, "Do you want to book a restaurant next to it?" And, uh, and then not ask the same information over and over again, what time? (laughs)

Lex Fridman 1:16:20 Yeah.

Rohit Prasad 1:16:20 What, uh, uh, how many people in your party, right? So, uh, so this is where you shift the cognitive burden from the customer to the AI, where it's thinking the, of what is your, uh, it anticipates your goal and takes the next best action to complete it. Now, that's the machine learning problem. Uh, s- but essentially you're, uh, the way we solved this first instance, and we have, uh, got a long way to go to make it-

Lex Fridman 1:16:47 Yeah.

Rohit Prasad 1:16:47 ... scale to everything possible in the world, but at least for this situation, it is from, uh, at every instance, Alexa is making the determination whether it should stick with the experience with ATOM Tickets or offer, uh-

Lex Fridman 1:17:00 Mm-hmm.

Rohit Prasad 1:17:00 ... or you, based on what you say, whether either you have completed the interaction or you said, "No, get me an Uber now," so it will shift context into another experience or skill or another service. So, that's a dynamic decision-making. That's making Alexa, you can say, more conversational for the benefit of the customer, rather than simply complete transactions which are well thought through, you have i- uh, you as a customer has fully specified what you want to be accomplished.

Lex Fridman 1:17:29 Yeah.

Rohit Prasad 1:17:29 It's accomplishing that.

Lex Fridman 1:17:30 So, it's kinda as, uh, in, uh, we do this with pedestrians, right, intent modeling. It's, uh, predicting w- what your possible goals are-

Rohit Prasad 1:17:38 Exactly.

Lex Fridman 1:17:38 ... and what's the most likely goal and then switching that depending on the things you say. So, the ... my question is there ... it seems ... maybe it's a dumb question, but

Lex Fridman 1:17:47 it would help a lot if Alexa remembered me, what I said previously.

Rohit Prasad 1:17:53 Right. So-

Lex Fridman 1:17:54 Is, is it, is it trying to use some memory for the customer's purpose?

Rohit Prasad 1:17:58 So, it- yeah, it is using a lot of memory within that. So, right now not so much in terms of, okay, which restaurant do you prefer-

Lex Fridman 1:18:04 Right.

Rohit Prasad 1:18:05 ... right? That is a more long-term memory. But within the short-term memory, within the session, it is remembering how many people did you ... so if you said-

Lex Fridman 1:18:12 Oh, with the-

Rohit Prasad 1:18:12 ... uh, buy four tickets, now it has made an implicit assumption that, uh, you are gonna have ... you need four, at least four seats at a restaurant. Right? So, these are the kind of, uh, contexts it's preserving between these skills but within that session. But you're asking the right question in terms of for it to be more and more useful-

Lex Fridman 1:18:31 Yes.

Rohit Prasad 1:18:32 ... it has to have more long-term memory, and that's also an open question. And, uh, again, this is still early days.

Lex Fridman 1:18:37 That's right. So, so for me, I, I mean, everybody's different, but, uh, yeah, I'm definitely not representative of the general population in the sense that I do the same thing every day.

Rohit Prasad 1:18:46 (laughs)

Lex Fridman 1:18:47 (laughs) Like, I eat the same ... like, I do everything the same, the same thing. Um, wear the same thing clearly, uh, this or the black shirt. Uh, so it's frustrating when Alexa doesn't get what I'm saying-

Rohit Prasad 1:18:59 Okay.

Lex Fridman 1:18:59 ... because I have to correct her every time the exact same way. This has to do with certain songs.

Rohit Prasad 1:19:05 Yeah.

Lex Fridman 1:19:05 Like sh- she doesn't know certain weird songs I like and doesn't know-... I've complained to Spotify about this. Talked to the-

Rohit Prasad 1:19:12 (laughs)

Lex Fridman 1:19:12 ... RD, uh, head of RD at Spotify. Stairway to Heaven, I have to correct it every time.

Rohit Prasad 1:19:16 Really?

Lex Fridman 1:19:16 It do- doesn't play Led Zeppelin correctly.

Rohit Prasad 1:19:18 (laughs)

Lex Fridman 1:19:18 It plays-

Rohit Prasad 1:19:19 Good-

Lex Fridman 1:19:19 ... a cover of Led Z- of Stairway to Heaven.

Rohit Prasad 1:19:21 (laughs)

Lex Fridman 1:19:22 So, I'm-

Rohit Prasad 1:19:22 You should figure... You should send me your, uh... Next time it fails-

Lex Fridman 1:19:26 This-

Rohit Prasad 1:19:26 ... feel free to send it to me. We'll take care of it.

Lex Fridman 1:19:28 Okay. Well, you know what?

Rohit Prasad 1:19:29 Because Led Zeppelin is one of my favorite-

Lex Fridman 1:19:31 That's awesome.

Rohit Prasad 1:19:31 ... bands and it works for me, so I'm, like, shocked it doesn't work for you.

Lex Fridman 1:19:34 This is an official bug report.

Rohit Prasad 1:19:35 (laughs)

Lex Fridman 1:19:36 I'll put it, I'll put it... I'll make it public-

Rohit Prasad 1:19:37 (laughs)

Lex Fridman 1:19:37 ... and make everybody retweet it. We're gonna fix the Stairway to Heaven problem.

Rohit Prasad 1:19:41 (laughs)

Lex Fridman 1:19:41 Anyway, but the point is, uh, you know, I'm pretty boring and do the same things, but I'm sure most people do the same set of things. Uh, do you see Alexa sort of utilizing that in the future for improving the experience?

Rohit Prasad 1:19:52 Yes. And not only utilizing, it's already doing some of it. We call it mo- uh, where Alexa's becoming more self-learning.

Lex Fridman 1:19:59 Mm-hmm.

Rohit Prasad 1:19:59 So, uh, Alexa is now auto-correcting millions and millions of utterances in the US without any human supervision involved. The way it does it is, uh, let's take an example of a, a particular song didn't work for you. Uh, what do you do next? You... Either it played the wrong song and you said, "Alexa, no, that's not the song I want." Or you say, "Alexa, play, uh, that, uh, uh..." You try it again.

Lex Fridman 1:20:24 Mm-hmm.

Rohit Prasad 1:20:25 And that is a signal to Alexa that sh- she may have done